Conserving Resources

[RCull]

I need to find a way to reduce cpu and memory use. My site is based on a Vbulletin Forum and a fair amount of static reference pages.

I notice in stats the robots. Some are "Unknown". Is there a way to stop unknown robots, and are there disadvantages to stopping them?

I look at the "Hosts, Top 25" shows a Ukraine IP using a lot of resources. Too many for the rare Ukrainian user of the site. Should I just ban an IP like that? I don't want to make the site difficult for international users.

I have activated CloudFlare.

Thanks for pointing down the right roads to helping with this problem.

Bob

[AndrewGlow]

Hello Bob,

If there is some specific IP that is causing much load - then you can ban it. Blocking one IP should not cause issues for many customers.

The main problem is that Cloudflare works at www.yourdomain.com. Bots still can access the server directly via yourdomain.com link.

I suggest you creating a robots.txt file in your public_html folder and blocking all bots except of those you'd like to visit your website.

For example, to allow only bots from Google, Yahoo and MSN visiting your website, you can create a robots.txt file with following contents:

User-agent: googlebot
Disallow:

User-agent: Slurp
Disallow:

User-agent: msnbot
Disallow:

User-agent: *
Disallow: /

Also, you should be able to see in stats what exact file is usually accessed by bots. If it is something like login.php being attacked by bad bots, we can provide you with instructions to rename that file properly. Please open a support ticket for this.

[RCull]

So something like this would allow googlebot and slurp into all directories except the ones listed as "Disallow" and all other agents would not be allowed anywhere:

User-agent: googlebot
Disallow: /forums/classifieds
Disallow: /cgi-bin/
Disallow: /data/
Disallow: /download/
Disallow: /facebook/
Disallow: /generator/
Disallow: /include/
Disallow: /reference/library/images
Disallow: /temp/
Disallow: /test/
Disallow: /upload/
Disallow: /webstats/
Sitemap: http://www.teambuick.com/sitemap.xml

User-agent: Slurp
Disallow: /forums/classifieds
Disallow: /cgi-bin/
Disallow: /data/
Disallow: /download/
Disallow: /facebook/
Disallow: /generator/
Disallow: /include/
Disallow: /moderators/
Disallow: /openx/
Disallow: /reference/library/images
Disallow: /temp/
Disallow: /test/
Disallow: /upload/
Disallow: /webstats/
Sitemap: http://www.teambuick.com/sitemap.xml

User-agent: *
Disallow: /

[AndrewGlow]

Yes, this looks like a solution that should work.

[Matt]

It's important to know that "bad" bots do not always honor the instructions that you have in the robots.txt - You may end up having to ban those IPs directly.

"Good" bots like Slurp and googlebot will honor your instructions.

[RCull]

I have added this to my htaccess, but I am getting this error:
[Sat Sep 19 06:34:08 2015] [alert] [client 217.73.208.152] /home/teambuic/public_html/.htaccess: Invalid command 'User-agent:', perhaps misspelled or defined by a module not included in the server configuration

I don't think it is related to spelling.

[RCull]

It also brought the site down with a 500 error.

[RCull]

OPPS figured it out

[AndrewGlow]

Nice to hear that!

[AndrewGlow]

Just to ask, what was the problem in this exact issue?