If you are signing up to the forums, Thank you. You will need to activate your account by clicking a link in an email from the forums.
Please make sure you check your Junk/Spam folder for the email and make sure you spell your email address correctly or you won't get the email.
Thanks again. Dismiss this with the [X] >>>>>>

Site Down

News on Forum updates or problems.

Moderator: Simple

User avatar
Simple
Site Admin
Site Admin
Posts: 12128
Joined: September 11th, 2006, 9:58 pm
Personal Text: flapjack nom nom
Location: Staffordshire
Contact:

Site Down

#1

Unread post by Simple »

Sorry for the downtime yesterday. We are still having problems with bots (Googlebot I think) using silly amounts of bandwidth.
Still trying to sort it :facepalm :rofl
garethevans1986
Junior Member
Junior Member
Posts: 94
Joined: July 3rd, 2010, 11:51 pm
Location: Chester
Match Team/ Club: Willaston/MFS
Sponsor: Myself

Site Down

#2

Unread post by garethevans1986 »

User avatar
Simple
Site Admin
Site Admin
Posts: 12128
Joined: September 11th, 2006, 9:58 pm
Personal Text: flapjack nom nom
Location: Staffordshire
Contact:

Site Down

#3

Unread post by Simple »

I edited that today and I checked that very site you linked Gareth :o
Bad bot dont obey the robots.txt file anyway but the problem is googlebot.
I have changed the crawl rate of the googlebot via Google webmaster tools to see if that helps.
There is another bot but shows as Unknown in AWstats, but I suspect that is google also.
All started when I updated to the latest version of the forum software.
Is gone from an average 10gig a month to 50gig a month bandwidth. :shifty

Image

Cheers, Gareth :tiphat
garethevans1986
Junior Member
Junior Member
Posts: 94
Joined: July 3rd, 2010, 11:51 pm
Location: Chester
Match Team/ Club: Willaston/MFS
Sponsor: Myself

Site Down

#4

Unread post by garethevans1986 »

I'd check the raw access logs for the site and ban the IP Address.

G
User avatar
Simple
Site Admin
Site Admin
Posts: 12128
Joined: September 11th, 2006, 9:58 pm
Personal Text: flapjack nom nom
Location: Staffordshire
Contact:

Site Down

#5

Unread post by Simple »

Already have banned some Gareth. Banned whole ranges but it continues. Been checking Raw access logs but can't seem to pin point anything.

Since I updated to 2.6 my bandwidth has gone through the roof.

Image

Upgraded from 2.4 on the 10th and you can see the increase in bandwidth.

Image

The increase seems to be coming from a Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) Also random IP's sucking large amounts of page hits and bandwidth.

June
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 152,931+1085 1.71 GB 30 Jun 2014 - 23:59
Googlebot 59,844+60 703.65 MB 30 Jun 2014 - 23:59[/tt]

July
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 162,564+1486 15.31 GB 31 Jul 2014 - 23:59
Googlebot 53,330+64 2.99 GB 31 Jul 2014 - 23:59
Unknown robot (identified by 'robot') 30,193+211 2.12 GB 31 Jul 2014 - 23:42[/tt]

1.71 to 15.31 is a massive increase and only happened after I updated the forum

Crazy  ::) :shifty
Josh1337
Forum Stalker
Forum Stalker
Posts: 686
Joined: April 14th, 2011, 6:08 pm

Site Down

#6

Unread post by Josh1337 »

Code: Select all

User-agent: *
	Disallow: / 

in robots.txt - Shouldn't it just block out all?
Post Reply