Forum very slow today [SOLVED]

[rockedge]

Please report how it's working now.........they tell me it's fixed. But I am still seeing a big lag

[pemasu]

It is not fixed. There is still lag, about 10 seconds.

[rockedge]

Today Nawaz Ahmed from Bluehost tech support helped us out tracking down the problems that are slowing the sites and causing errors when pages are rendering.

Several steps were taken to improve the situation and we now await to see how the fixes migrate.

So a shout out to Nawaz for sticking to the problem and finding possible solutions!

[fredx181]

Still lagging a bit, but I can live with that (it's good test for being patient )
Just to report, I see also "Ajax error" sometimes e.g. when clicking on 'Notifications' and select from the list.
edit: and the sql error message comes up frequently too.

[tosim]

Still very slow here @1014 MST, USA

[Flash]

>10 second lag here.

[Grey]

I have a good friend who does pretty much the same thing I do for a company that pays him $100,000+ per year. And he calls me for advice. Ahhhhh the world works in mysterious ways

Such a familiar situation. My friend also calls me quite often and I ask every time: "how have you not been fired yet?!"
Maybe that's why everything is so slow. An employee of the provider's company simply cannot contact his friend.

[keniv]

Seems to have got slower over the day. Just logged in and got another sql error.

Ken.

[williwaw]

I have looked at many many possible reasons from our end. Nothing jumps out and no configuration has changed. DNS resolves fine. Strange is that even a very simple single index.html page is throwing Internal Server Error's.

frustrating. Has anything been updated recently on your end recently that can be reverted as a test?

Has anything been updated recently on their end that might not play well with your setup?
(I know you have to take on faith their answer to this question.)

[bugnaw333]

This is just like Windows now--slows over time...

[bigpup]

No improvement for me.

Still takes 8 to 10 seconds to work.

[bugnaw333]

16 seconds here.

[bigpup]

In Pale Moon browser with the Developer Tools -> Debugger turned on.

Every time I select something in the forum i get this response before it shows what I selected.

Notice the second entry about stylesheet having a 404 status.

Maybe a clue
.

Screenshot(4).jpg (102.03 KiB) Viewed 994 times

[dogcat]

just thinking out loud here

this feels like an overloaded memory issue, you have probably already checked to make sure they didn't upgrade or change mysql version & check the phpbb version against mysql versions for compatibility and checked phpbb plugin compatibility. Very odd having two separate phpbb and sql databases having similar issues when they were both fine, no such thing as a coincidence

[Mercedes350se]

'zilla browsers are as slow as cold molasses - palemoon seems normal.

Which version of Palemoon please? (And 32bit or 64bit?) Cheers!

Version 31.1.0 64 bit

BUT this morning it is exactly the same - slow as cold molasses.

[wiak]

https://firstrib.rockedge.org remains fast, but it is admittedly a very small forum, but I think same software but local rockedge host?

[rockedge]

@wiak
rockedge.org is using iFastnet. Which also was the original home on another account of this forum. We got kicked off that because of a DDoS attack on our site.

I tend to also think it's a memory and cache problem. But still nothing was changed on our end at all and the sites went to running well to really badly in a matter of minutes and has now continued to give us grief.

I still have the escalated ticket to 2nd tier admin's going which I will check on soon. I see problems with the mention BBCode responding in the editor, which indicates an AJAX problem meaning something is amiss on the server end.

[AntonioPt]

Mysql connection dont close who ever is in control need to check that script

[rockedge]

There are hundreds of background processes being generated calling viewtopic.php. This is bogging everything down. The Bluehost admin's killed those processes but they regenerated immediately again!

I am trying to take the oldforum offline for a moment to find out what can cause this to happen. I fear a vulnerability is being exploited by someone who approached us about these and wanted money to point this out. Which since we are not a company or really an organization there is no money I explained and I have not heard anything since. Just putting things out there to brainstorm a fix

I have contacted phpBB on their forum for some help. Bluehost says it's not their system and the servers are OK so we're on our own.

UPDATE: diving deep into logs and configurations I noticed all the majority of network traffic is coming through murga-linux.com. The massive load of page requests are coming through murga-linux.com where a URL rewrite corrects the domain name and sends the request to the correct URL for oldforum.puppylinux.com.

To test my theory I have disabled the URL rewrite and isolated the page requests coming through those old links to kill the processes caused by the excessive requests via murga-linux.com. It is sort of like a DDoS attack or caused by a bad or broken Bot.

Does anyone notice any improvement to the page response and load speeds? Does it work?

I kept repeating to IT support that the logs indicate to me that rapid page requests to the oldforum via a domain name only used for backward compatibility seems to be the cause of the hundreds of processes clogging our server.

I have now successfully logged in via SSH and can run commands again. Before I couldn't do anything at all. Too many processes running to kill the processes running. So now I can look at htop and see how our 26 CPU cores are doing.....which looks OKAY now.

So I think I found the problem and my short term fix seems to be working.

I understand now that the IT admin's don't really know systems and how to troubleshooting one. They kept trying to get me to "optimize" the site. I kept saying that makes no sense. Something is requesting pages or not? They never really told me BUT they did turn on SSH access for me which isn't usually part of this package. That comes from asking the right questions to get the clues. Finally the webalizer and AWStats told the story in red about all of the requests coming through murga-linux.com and I turned off the URL rewrite engine for now which sends the page to an error page

Keep your fingers crossed.

Screenshot_2022-11-09_00-35-25.png (287.93 KiB) Viewed 873 times

[rcrsn51]

That's better now. Thanks.

Of course, any link on a page at oldforum.puppylinux.com that points to murga-linux is now broken, but you can still reach the content by manually fixing the URL in your browser address bar.

Is it finally time to move any essential content out of the old forum and retire it?

[greengeek]

All good here now. Thanks!

[wiak]

Fine for me now too. Sounds hopeful you fixed it rockedge. Surprised Bluehost IT couldn't trace the issue themselves - would have thought it was obvious at their end, but maybe not.

[williams2]

Seems OK at Wed Nov 09, 2022 2:15 am

I think the denial of service attack causes the max_connections_per_hour value to exceed the maximum limit
which automatically sets max_connections_per_hour to a value of 1
which is very slow

Other things like SEF (Search Engine Friendly) rewrites URLs
which accesses the database more times
increases the max_connections_per_hour value
which causes the max_connections_per_hour to be set to 1.for an hour,
and then resets the value to the default value.
So the database works properly for a while,
then slows down a lot for the rest of the hour
then at the end of the hour resets itself.

So it works for a while
then works very slowly
then works for a while
then works very slowly
etc etc

So it should work ok as long as the connections_per_hour value does not exceed the max value.
Which is what the techs mean by optimizing.
They just read from a script anyway.

[user1234]

@rockedge-

The forums seem to be alright for me now. No errors, no delays, its just alright now. Thanks !

By the way, old forums are also working for me alright. What's the case? I thought you closed them down?

[keniv]

Just logged on this morning and the forum seems to be back to normal for me here in the UK. Glad you appear to have found the source of the problem.

Ken.

[fredx181]

Thanks @rockedge ! Fast for hours already.

[mikewalsh]

@rockedge :-

Mm-hm. Seems you've cracked it, Erik. Everything appears back to normal now...

Well done, mate. Took a while, but it sounds like this may all simply be part of the learning process connected with being a "Webmaster". Could have been a beginner's mistake, which has miraculously lasted this long, then got triggered by a specific combination of events? Who knows?

Cheers, anyway! First-class detective work...thanks!

Mike.

[rockedge]

Surprised Bluehost IT couldn't trace the issue themselves - would have thought it was obvious at their end, but maybe not.

All the clues were there but it took 3 conversations to get each piece of the puzzle. I knew that some thing was causing page requests. But the extent of the processes that were going and then immediately regenerated after being killed and then looking at logs the answer began to appear.

Once I analyzed all of the logged data with Webalizer and AWStats it was clear that while every other puppylinux.com request was very balanced and no strains at all there was Gigabytes of bandwidth coming in requests for pages, which calls viewtopic.php (parses all the data to make the page you see) with the referrer URL being murga-linux.com.

So I because now I have a real terminal via SSH I could run htop and watch the processes. But the incoming requests had flooded the RAM to the point we were deep into the swap partition with use and no commands worked and errors of no more resources were all I could see.

But the analytics were pointing to something like a DDoS attack was occurring and seemed to be coming in via the murga-linux.com domain.

Had the Bluehost tech in the first conversation told me that there was heavy traffic I would have known instantly what to do to stop it. I kept asking for them to look at the incoming requests and what the referrer's were. I understood what was happening to the MySQL server but had to eliminate the possibility of a memory leak from phpBB.

All the tech's kept telling me was to optimize but first the flood of page requests needed to be halted I told them. Then they said we are server admin's and don't do code and you are on your own and good luck.

So a combination of looking at the November access data that is put into graphs and charts I saw were the page requests were coming from, and the tech's informing me that there were hundreds processes that I saw then requesting viewtopic.php, and Bluehost granting me SSH access to the server. Finally it tool seconds to turn off the Apache URL rewrite engine for murga-linux.com and the problem stopped immediately!

I turned it off, spun my chair and on another computer clicked on the forum and it was already back to normal speeds and the many processes shutdown and disappeared.

Now what to do with murga-linux.com links? I want to trace back to find the source of the requests gone wild. Our system of rewriting murga-linux.com worked perfectly for 2+ years until now.......now something or someone kept sending those page requests through that domain name.

This I find odd so perhaps it isn't a human behind it. Don't know yet. Maybe some Bot got stuck on the murga-linux.com domain. At some point I will try to turn on the rewrite engine to see if the flood is actually continuing but deflected off into cyberspace when turned off.

@mikewalsh It's a matter of putting the clues together from bits and pieces of info from each different support tech checking from the server side and me pouring over the logs in detail. 15+ years now running public websites. 20+ running local servers like Apache and Hiawatha and some NGINX kept me on the pitch and in the game. Really the Bluehost people could have said "we see lots of traffic coming from XX and requesting pages so and so from this referrer" and this could have been tried on Saturday.

Anyway we will hold on to the murga-linux.com and try to ease it back into shape and perhaps a more complex rewrite could help out.

[rcrsn51]

Is it possible to do a wholesale search-and-replace of the old forum's data that changes "murga-linux.com/puppy" to "oldforum.puppylinux.com"?

[rockedge]

Is it possible to do a wholesale search-and-replace of the old forum's data that changes "murga-linux.com/puppy" to "oldforum.puppylinux.com"?

Yes I believe it is.

Here is a graph and chart that shows the story.

Notice the daily average of 817697 page requests per day for 9 days in November. Compare that to September's 14893 hits which is around plus or minus a few thousand, what the levels of traffic usually are that come through murga-linux.com links. (October the entire account was moved to a new data center / server farm which reset the statics that month)

Screenshot(46).jpg (17.68 KiB) Viewed 892 times

Screenshot(47).jpg (109.67 KiB) Viewed 892 times