Slow forum access times....anyone but me?

[Wiz57]

They already are @greengeek ! I read some of the comments at Palemoon it seems they are undergoing
very similar bog downs as we are. I've noticed that the numbers of "guests" here, at Palemoon Forum, at MSFN Forum and a few other
places are astronomical! I mean, we sometimes have maybe 10 registered members online...but we'll have hundreds of "guests".
Same at Palemoon and MSFN when you look at "Who's Online". Yesterday afternoon I looked at Palemoon, there were around 40 members,
and over 400 guests! Crazy stuff!

[rockedge]

This is in our logs and Who's online page(s):

Meta-ExternalAgent

The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.

The specific UA string that you will see in your log files will be similar to one of the following:

Code: Select all

    meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
    meta-externalagent/1.1

One of the AI bots we see a lot of. We allow some of these bots through if behaved.

[williwaw]

We allow some of these bots through if behaved.

are you whitelisting or blacklisting?

I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?

https://news.ycombinator.com/item?id=42861047

[greengeek]

https://news.ycombinator.com/item?id=42861047

Interesting stuff there.

This sounded simple to differentiate between robots and crawlers:
(not that I know the difference...)

ExtraPage.png (22.71 KiB) Viewed 762 times

[N97]

Did anyone notice that the latest round of AI scrape EXACTLY coincided with the Help Vampires ?

[ozsouth]

Even the emergency site, Puppylinux Alt is getting an increasing number of hits & although I declined search engine listings, Google & Yahoo find it.

[Clarity]

Hell @greengeek.
The ONLY reason someone(s) would venture into a home use of any AI is to avoid the cost of the models. Thus, they can use their home services for the advances that their home AI version offers. In some/many cases, though this would be adequate, it certainly does not compare with the hardware performance that is in the web offerings. ... and for some/many at a small cost without the user maintenance.

Yes, @ozsouth this phenomenon has been ever-present for many years by all of the commercial and gov't crawlers in the world.

[mikewalsh]

@rockedge :-

Looks like the bots have found a way round your protections, mate. In the last half-hour, the forum has slowed to a crawl; 30 seconds per page, along with several instances of "429 - Too many requests per page...."

Anybody else seeing this?

Mike.

[backi]

@mikewalsh wrote:

Looks like the bots have found a way round your protections, mate. In the last half-hour, the forum has slowed to a crawl; 30 seconds per page, along with several instances of "429 - Too many requests per page...."

Just a few Minutes ago it did so.......but now......everything seems back to "normal".....here in Germany.

[rockedge]

Yesterday for a few minutes the forum was really slow and I did think "oh no" but it cleared up.

I'll go check who is banging away today.........

[mikewalsh]

Well, all appears 'serene' again. The tech's apparently once again doing what it's supposed to be doing (or so it seems). We'll keep an eye on it, as always...

Onwards & upwards!

Mike.

[rockedge]

I checked the logs and metrics and it appears the "tech" is doing it's job. We'll keep a weather eye out on the horizon.

@mikewalsh see how I worked a phrase from the great age of sail into the conversation?

[mikewalsh]

@mikewalsh see how I worked a phrase from the great age of sail into the conversation?

^^^ +1!! Lololol!

Mike.