Still trying to sort it
If you are signing up to the forums, Thank you. You will need to activate your account by clicking a link in an email from the forums.
Please make sure you check your Junk/Spam folder for the email and make sure you spell your email address correctly or you won't get the email.
Thanks again. Dismiss this with the [X] >>>>>>
Please make sure you check your Junk/Spam folder for the email and make sure you spell your email address correctly or you won't get the email.
Thanks again. Dismiss this with the [X] >>>>>>
Site Down
Moderator: Simple
-
garethevans1986
- Junior Member
- Posts: 94
- Joined: July 3rd, 2010, 11:51 pm
- Location: Chester
- Match Team/ Club: Willaston/MFS
- Sponsor: Myself
Site Down
The problem will be your http://simplematchfishing.co.uk/robots.txt read more here, http://www.robotstxt.org/robotstxt.html
Gareth
Gareth
-
Simple
- Site Admin
- Posts: 12128
- Joined: September 11th, 2006, 9:58 pm
- Personal Text: flapjack nom nom
- Location: Staffordshire
- Contact:
Site Down
I edited that today and I checked that very site you linked Gareth 
Bad bot dont obey the robots.txt file anyway but the problem is googlebot.
I have changed the crawl rate of the googlebot via Google webmaster tools to see if that helps.
There is another bot but shows as Unknown in AWstats, but I suspect that is google also.
All started when I updated to the latest version of the forum software.
Is gone from an average 10gig a month to 50gig a month bandwidth.
Cheers, Gareth
Bad bot dont obey the robots.txt file anyway but the problem is googlebot.
I have changed the crawl rate of the googlebot via Google webmaster tools to see if that helps.
There is another bot but shows as Unknown in AWstats, but I suspect that is google also.
All started when I updated to the latest version of the forum software.
Is gone from an average 10gig a month to 50gig a month bandwidth.
Cheers, Gareth
-
garethevans1986
- Junior Member
- Posts: 94
- Joined: July 3rd, 2010, 11:51 pm
- Location: Chester
- Match Team/ Club: Willaston/MFS
- Sponsor: Myself
Site Down
I'd check the raw access logs for the site and ban the IP Address.
G
G
-
Simple
- Site Admin
- Posts: 12128
- Joined: September 11th, 2006, 9:58 pm
- Personal Text: flapjack nom nom
- Location: Staffordshire
- Contact:
Site Down
Already have banned some Gareth. Banned whole ranges but it continues. Been checking Raw access logs but can't seem to pin point anything.
Since I updated to 2.6 my bandwidth has gone through the roof.
Upgraded from 2.4 on the 10th and you can see the increase in bandwidth.
The increase seems to be coming from a Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) Also random IP's sucking large amounts of page hits and bandwidth.
June
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 152,931+1085 1.71 GB 30 Jun 2014 - 23:59
Googlebot 59,844+60 703.65 MB 30 Jun 2014 - 23:59[/tt]
July
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 162,564+1486 15.31 GB 31 Jul 2014 - 23:59
Googlebot 53,330+64 2.99 GB 31 Jul 2014 - 23:59
Unknown robot (identified by 'robot') 30,193+211 2.12 GB 31 Jul 2014 - 23:42[/tt]
1.71 to 15.31 is a massive increase and only happened after I updated the forum
Crazy ::)
Since I updated to 2.6 my bandwidth has gone through the roof.
Upgraded from 2.4 on the 10th and you can see the increase in bandwidth.
The increase seems to be coming from a Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) Also random IP's sucking large amounts of page hits and bandwidth.
June
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 152,931+1085 1.71 GB 30 Jun 2014 - 23:59
Googlebot 59,844+60 703.65 MB 30 Jun 2014 - 23:59[/tt]
July
[tt]Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-) 162,564+1486 15.31 GB 31 Jul 2014 - 23:59
Googlebot 53,330+64 2.99 GB 31 Jul 2014 - 23:59
Unknown robot (identified by 'robot') 30,193+211 2.12 GB 31 Jul 2014 - 23:42[/tt]
1.71 to 15.31 is a massive increase and only happened after I updated the forum
Crazy ::)
Site Down
Code: Select all
User-agent: *
Disallow: / in robots.txt - Shouldn't it just block out all?