Slow forum access times....anyone but me?

[Wiz57]

They already are @greengeek ! I read some of the comments at Palemoon it seems they are undergoing
very similar bog downs as we are. I've noticed that the numbers of "guests" here, at Palemoon Forum, at MSFN Forum and a few other
places are astronomical! I mean, we sometimes have maybe 10 registered members online...but we'll have hundreds of "guests".
Same at Palemoon and MSFN when you look at "Who's Online". Yesterday afternoon I looked at Palemoon, there were around 40 members,
and over 400 guests! Crazy stuff!

[rockedge]

This is in our logs and Who's online page(s):

Meta-ExternalAgent

The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.

The specific UA string that you will see in your log files will be similar to one of the following:

Code: Select all

    meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
    meta-externalagent/1.1

One of the AI bots we see a lot of. We allow some of these bots through if behaved.

[williwaw]

We allow some of these bots through if behaved.

are you whitelisting or blacklisting?

I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?

https://news.ycombinator.com/item?id=42861047

[greengeek]

https://news.ycombinator.com/item?id=42861047

Interesting stuff there.

This sounded simple to differentiate between robots and crawlers:
(not that I know the difference...)

ExtraPage.png (22.71 KiB) Viewed 1535 times

[N97]

Did anyone notice that the latest round of AI scrape EXACTLY coincided with the Help Vampires ?

[ozsouth]

Even the emergency site, Puppylinux Alt is getting an increasing number of hits & although I declined search engine listings, Google & Yahoo find it.

[Clarity]

Hell @greengeek.
The ONLY reason someone(s) would venture into a home use of any AI is to avoid the cost of the models. Thus, they can use their home services for the advances that their home AI version offers. In some/many cases, though this would be adequate, it certainly does not compare with the hardware performance that is in the web offerings. ... and for some/many at a small cost without the user maintenance.

Yes, @ozsouth this phenomenon has been ever-present for many years by all of the commercial and gov't crawlers in the world.

[mikewalsh]

@rockedge :-

Looks like the bots have found a way round your protections, mate. In the last half-hour, the forum has slowed to a crawl; 30 seconds per page, along with several instances of "429 - Too many requests per page...."

Anybody else seeing this?

Mike.

[backi]

@mikewalsh wrote:

Looks like the bots have found a way round your protections, mate. In the last half-hour, the forum has slowed to a crawl; 30 seconds per page, along with several instances of "429 - Too many requests per page...."

Just a few Minutes ago it did so.......but now......everything seems back to "normal".....here in Germany.

[rockedge]

Yesterday for a few minutes the forum was really slow and I did think "oh no" but it cleared up.

I'll go check who is banging away today.........

[mikewalsh]

Well, all appears 'serene' again. The tech's apparently once again doing what it's supposed to be doing (or so it seems). We'll keep an eye on it, as always...

Onwards & upwards!

Mike.

[rockedge]

I checked the logs and metrics and it appears the "tech" is doing it's job. We'll keep a weather eye out on the horizon.

@mikewalsh see how I worked a phrase from the great age of sail into the conversation?

[mikewalsh]

@mikewalsh see how I worked a phrase from the great age of sail into the conversation?

^^^ +1!! Lololol!

Mike.

[fredx181]

Forum very slow again around this time.
Got also sometimes the "Too many requests" error.
edit: about 10 minutes later it's ok again

[rockedge]

@fredx181
I am also seeing the slow down. I checked the metrics and see nothing really unusual yet.

We'll keep an eye out....I logged in remotely via SSH to look at htop on the server but couldn't get a clean prompt because of too many processes.

Suddenly the site is a bit faster but some Bot or crawler is hitting us hard somewhere.....

[fredx181]

@fredx181
I am also seeing the slow down. I checked the metrics and see nothing really unusual yet.

We'll keep an eye out....I logged in remotely via SSH to look at htop on the server but couldn't get a clean prompt because of too many processes.

Suddenly the site is a bit faster but some Bot or crawler is hitting us hard somewhere.....

Thanks, yes, fine again now, well, if the slowness will be only for a very short period everytime, it's not a problem IMO.

[bigpup]

Reduce the Google bot crawl rate
https://developers.google.com/search/do ... crawl-rate

Bing bot Crawl control
https://www.bing.com/webmasters/help/?topicid=55a30303

[rockedge]

@bigpup

It's not those Bots that are hammering us. The ones really effecting the sites are scrapping crawlers that do not adhere to many rules. I find and deny via IP on the server level when I identify the IP's.

Though there might be a suspicious vector just discovered. It is being investigated.

[greengeek]

Has been verrrry slow here in Auckland NZ this morning.
We had the most massive thunderstorm last night so could be aggravated by some level of water induced flakery ...

[fredx181]

Strange IMHO that there were 1558 users online today

2025-04-19_20-18-33.png (18.14 KiB) Viewed 417 times

[Wiz57]

Strange IMHO that there were 1558 users online today
2025-04-19_20-18-33.png

That's what I've been sort of investigating lately! The enormous number of "guests" seem to be primarily AI bots that DO NOT follow any robots.txt directives. Many are from Alibaba, some from Yandex, some from AWS, Facebook...
They tie up the server, download content. I've been searching for methods of slowing them down, perusing the phpBB forums, searching online. From there I send rockedge any ideas I come across.
The situation looks to be only going to get worse because some of these AI bots have developed the ability to write their own UA strings! By mimicking actual human users, they scrape our entire database, downloading files, etc. When the number of guests gets really big, the forum slows down noticeably. If anyone has any ideas, please share!
Another thing I'd like to ask...How many members are utilizing a VPN to access the forum?
Reason to ask is a lot of these bots have IP Addresses assigned to RIPE NNC...and I've noticed some of our regulars also trace to RIPE! Just a curiosity.

[IdfbAn]

Many are from Alibaba, some from Yandex, some from AWS, Facebook...

From a quick Google search, I've found this article about AI companies ignoring robots.txt, and some other articles specifically mention OpenAI and Anthropic (Claude). Neither of these are the companies you mentioned though, so this is interesting.

[rockedge]

I have access to various metrics and logs from the server side. I can identify patterns of what IP's have hundreds of page requests at or near the same time.

Then with IP deny rules I filter them out totally, not even letting them on the server. Along with a complex set of firewall rules we are holding with only a thousand or so guests at a time.

If one identifies such an IP please report it and I'll add to the deny. I did this accidentally to a Google Bot IP and within 2 days the forum was no longer appearing as much and fading on he Google Search indexing. I allowed the Bot and all was good again. So see an IP doing hundreds or more of page requests in very short periods of time, let me know and we will deny all access to that IP block.

[mikewalsh]

I could have it all wrong.....but I don't think most webmasters have an issue with the "resident" search-engine bots. I suspect most forums/communities with any kind of web 'presence' probably have a few kicking around underfoot. They are, after all, indirectly responsible for your SEO ratings. They're also, usually, well-behaved, and play by the accepted rules.

It's a bit like cellar spiders. It's a symbiotic arrangement; you're giving them somewhere to overwinter, and in turn they take care of the other insects for you.

A mate of mine has been letting a small group of peacock butterflies overwinter in a corner of his spare bedroom for many years. They're very pretty creatures, and they're an endangered species in the UK. He doesn't use the spare room....and they take care of aphids around his gladioli in the late spring/early summer. It's an arrangement that suits them both.

As for these AI scraper bots, huh; they're indiscriminately raping the internet......taking everything they can grab, wherever they find it, without a damn for the consequences (because it's not their problem, the carnage they leave behind them). A bit like locusts, in a way....

Mike.

[Wiz57]

I have access to various metrics and logs from the server side. I can identify patterns of what IP's have hundreds of page requests at or near the same time.

Then with IP deny rules I filter them out totally, not even letting them on the server. Along with a complex set of firewall rules we are holding with only a thousand or so guests at a time.

If one identifies such an IP please report it and I'll add to the deny. I did this accidentally to a Google Bot IP and within 2 days the forum was no longer appearing as much and fading on he Google Search indexing. I allowed the Bot and all was good again. So see an IP doing hundreds or more of page requests in very short periods of time, let me know and we will deny all access to that IP block.

It seems to be working! The number of "guests" is WAY down today (Sunday)...great work!
Ya know, I find it rather odd that Alibaba has a San Mateo, CA address??!! Sneaky!
Yeah, been checking the "whois" again!

[DOSUser123]

this happens to me on every site on every protocol
but my router is shit anyways sooooooo

[N97]

Guests must login.... name pswd IP. Blocks denied.

15:10 EDT... 1249 + me online

[Wiz57]

Just a quick "Shout out" to @rockedge for the work he has been doing in thinning the AI scraper bots! The past couple of days I've noticed forum response times have improved dramatically! Plus the number of "guests" has markedly decreased...EXCELLENT WORK MY FRIEND! The entire Forum community owe you a hardy "Thank you" in my not so humble opinion. It's tedious, repetitious and oft times goes unnoticed EXCEPT when it isn't done! So again, thank you, thank you, thank you!

[Wiz57]

Forum getting hammered again, 5/1/2025. I notice multitudes of UAs that show Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36, so perhaps blocking these UAs? Might slow them for a while, until the AI Bots create a new UA!!

[peebee]

EXTREMELY slow today - must be a massive attack.......... almost unusable