@rockedge :-
What is it you have there, Erik.....a VPS (Virtual Private Server)? With a certain amount of "virtual" hardware allocated to it, then a Puppy running on top of that? Or KLV? (I'm going by your references to "X" number of cores, then another one later about 'deep into swap territory').
Looking at those graphs, it's not hard to see why everything was jammed up solid..... 
(*wheeew...*)
Mike. 
Puppy "stuff" ~ MORE Puppy "stuff" ~ ....and MORE! 
_______________________________________________________
@mikewalsh It's a shared server but because I was nice and promised the tech who fixes it will get a mention and a big woofwoof on the Puppy Linux Forum and they granted me SSH access which is usually reserved for VPS customers. So now I can open a SSH session with PuTTY and talk to the server in a terminal on the command line. Bluehost was pretty sure I know what I am doing I guess.
I opened htop for example once I had the processes killed. Right now 24 cores.
In retrospect I now think that the original DDoS attack also came through murga-linux.com links but iFastnet suspended the account quickly once the server bogged down. Bluehost just let it run which allowed us to save it or they didn't notice the server slowdown.I might still have some logs that can be analyzed from the iFastnet system to see if my theory is correct.
Coming through the murga-linu links is a sneaky move or spiders gone really aggressive perhaps. Because the URL's are rewritten to access the topics on the oldforum, the error logs show a problem in the oldforum with oldforum URL's. So it is not at first apparent where the actual requests where coming from. Only after analysis of the access logs from across all of our sites including the calls to the murga-linux domain was it clear where the requests came from and matching those results with the error logs.
Now to see if we can track down the origin of the page requests.
I have tracked down the culprit to a spider called The Knowledge AI which can not negotiate https but hammered at the http murga-linux with one page request per second. Which flooded the RAM and swap with processes. The spider only looked for incomplete URL's.
This spider isn't really looking for pages and is not smart enough to do anything other than http.
It is suspected that it scans for another actor.
@rockedge :-
Is this the thing you're referring to?
https://www.webmasterworld.com/search_e ... 896765.htm
Just did a search, and this was the only relevant article that really came up. 'Hurricane Electric - he.net?' (Don't ask me what it IS, though!)
Mike. 
Puppy "stuff" ~ MORE Puppy "stuff" ~ ....and MORE!
_______________________________________________________
@mikewalsh Yes that is the one. There is very little known about it other than the spider can't handle https. Not much is written about it either other than the few things found in forums for web masters and admins.
After researching the information found, the conclusion for me is this spider is possibly looking for specific code that is another stage in gaining control of servers.
Last night a turned the rewrite engine on to allow murga-linux links to work again and so I could monitor the incoming traffic specifically coming through this domain. At first all was normal but after an hour the requests from the spider began to mount and that request is the same incomplete URL so no actual topic or post is being probed. Before long I could not longer login to the server via SSH. No more available memory, cache's overflowing so I turned off the engine and the mounting problems instantly disappeared.
I am leaving our rewrite system in place for a future time but for now the engine is turned off. Maybe we can find out more about the The Knowledge AI.
There is a robot.txt in place with rules to disallow a search through the domain name by the spider but it's open on how effective this will be.
@rockedge :-
The only thing I've been able to dig up so far is this:-
https://udger.com/resources/ua-list/bot ... AI#id48187
.....which in itself doesn't tell us very much. (It takes a while to load this link, so.....be patient!)
I suspect this is going to be something where you're just going to have to "play it by ear" as you go. Nobody seems to know much about this thing at all; where it comes from, who wrote it, what its motives are, etc. Some are of the opinion that there are certain bots out there that are programmed to generate new bots under certain circumstances, so it's honestly anybody's guess.
Scary to think we've reached the stage where software is autonomously generating other software with no human oversight of any kind....
Mike.
Puppy "stuff" ~ MORE Puppy "stuff" ~ ....and MORE!
_______________________________________________________
Have you tried an .htaccess file?
If you know the User-Agent string and are using Apache (?), you can use this format:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} AdsBot-Google [OR]
RewriteCond %{HTTP_USER_AGENT} msnbot [OR]
RewriteCond %{HTTP_USER_AGENT} AltaVista [OR]
RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule . - [F,L]
This takes a list of conditions (RewriteCond) and applies a rule to them. The F stands for Forbidden and the L means it's the last rule in the set.
Other info Here
Never ascribe to malice what can adequately be explained as stupidity. I forget who first said that. It might apply here.
, it's far beyond my eh.. hat