Slow forum access times....anyone but me?

[Wiz57]

They already are @greengeek ! I read some of the comments at Palemoon it seems they are undergoing
very similar bog downs as we are. I've noticed that the numbers of "guests" here, at Palemoon Forum, at MSFN Forum and a few other
places are astronomical! I mean, we sometimes have maybe 10 registered members online...but we'll have hundreds of "guests".
Same at Palemoon and MSFN when you look at "Who's Online". Yesterday afternoon I looked at Palemoon, there were around 40 members,
and over 400 guests! Crazy stuff!

[rockedge]

This is in our logs and Who's online page(s):

Meta-ExternalAgent

The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.

The specific UA string that you will see in your log files will be similar to one of the following:

Code: Select all

    meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
    meta-externalagent/1.1

One of the AI bots we see a lot of. We allow some of these bots through if behaved.

[williwaw]

We allow some of these bots through if behaved.

are you whitelisting or blacklisting?

I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?

https://news.ycombinator.com/item?id=42861047

[greengeek]

https://news.ycombinator.com/item?id=42861047

Interesting stuff there.

This sounded simple to differentiate between robots and crawlers:
(not that I know the difference...)

ExtraPage.png (22.71 KiB) Viewed 626 times

[N97]

Did anyone notice that the latest round of AI scrape EXACTLY coincided with the Help Vampires ?

[ozsouth]

Even the emergency site, Puppylinux Alt is getting an increasing number of hits & although I declined search engine listings, Google & Yahoo find it.

[Clarity]

Hell @greengeek.
The ONLY reason someone(s) would venture into a home use of any AI is to avoid the cost of the models. Thus, they can use their home services for the advances that their home AI version offers. In some/many cases, though this would be adequate, it certainly does not compare with the hardware performance that is in the web offerings. ... and for some/many at a small cost without the user maintenance.

Yes, @ozsouth this phenomenon has been ever-present for many years by all of the commercial and gov't crawlers in the world.