Exceeded Bandwidth

[Nicola]

It's happened again. We increased our bandwidth to 75GB a couple of months ago. 'Real' people use a max 13GB per month. The rest is bots/spiders etc. Yesterday one bot used 54GB. Every time it happens I ban the offending IP addresses/bots whatever. I don't know what else to do to be honest. If anyone has any ideas please let me know. The webhosts have allowed us 48hrs and then we will probably be offline again until the start of the month. Thankfully that's not far away.

[Macavity]

All I can think of is to switch off the registration process towards the end of each month, but I'll ask around. Is there less bot activity if more of the forums are 'hidden'?

best

mac

[Nicola]

I'm not sure about the hidden forums. The bots crawl the whole PG site which is quite large. I had considered stopping registrations. Thanks for having a think about it!

[Macavity]

I had this response on another forum:

I don't know... but one would think so, given that access is denied to human browsers, the same should apply to bots and spiders, unless they can copy or read passwords but in that case one would expect to see some evidence on the 'hidden' boards.

I went and checked, and yes, posts in The Commons from this year are averaging roughly 5000 views while only one post in the Founders Forum - from Jan 2018 - has more than 100 views, the rest are lower, realistically so.

The 'founders' group on this site have hidden forums, the viewing hits are much lower on these hidden forums. I presume the same is true of the moderators' forum on PG?

Personally, I think the 'Poetry Discussion' forum would benefit from being hidden since contemorary poetry could be quoted and discussed without copyright concerns.

hope that helps some

mac

[Macavity]

Just a heads up Nicola to say the site was down from Tuesday last week until Nov. 1st and is still showing some instability (sometimes requiring a refreshing page action to access the forum).

best

mac

[Nicola]

Thanks for the heads up. Yet another bot crawler through the site. The web hosts have banned some more IP addresses. We had increased our bandwidth etc which should be plenty. I just need to keep my eyes on it..I've been slacking..