Simple Match Fishing Forum

Sorry for the downtime yesterday. We are still having problems with bots (Googlebot I think) using silly amounts of bandwidth.
Still trying to sort it

The problem will be your http://simplematchfishing.co.uk/robots.txt read more here, http://www.robotstxt.org/robotstxt.html

Gareth

I edited that today and I checked that very site you linked Gareth
Bad bot dont obey the robots.txt file anyway but the problem is googlebot.
I have changed the crawl rate of the googlebot via Google webmaster tools to see if that helps.
There is another bot but shows as Unknown in AWstats, but I suspect that is google also.
All started when I updated to the latest version of the forum software.
Is gone from an average 10gig a month to 50gig a month bandwidth.



Cheers, Gareth

I'd check the raw access logs for the site and ban the IP Address.

G

Already have banned some Gareth. Banned whole ranges but it continues. Been checking Raw access logs but can't seem to pin point anything.

Since I updated to 2.6 my bandwidth has gone through the roof.



Upgraded from 2.4 on the 10th and you can see the increase in bandwidth.



The increase seems to be coming from a Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) Also random IP's sucking large amounts of page hits and bandwidth.

June
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 152,931+1085 1.71 GB 30 Jun 2014 - 23:59
Googlebot 59,844+60 703.65 MB 30 Jun 2014 - 23:59[/tt]

July
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 162,564+1486 15.31 GB 31 Jul 2014 - 23:59
Googlebot 53,330+64 2.99 GB 31 Jul 2014 - 23:59
Unknown robot (identified by 'robot') 30,193+211 2.12 GB 31 Jul 2014 - 23:42[/tt]

1.71 to 15.31 is a massive increase and only happened after I updated the forum

Crazy  ::)

in robots.txt - Shouldn't it just block out all?