How Puppy Linux Saved the Murga Forum

[rockedge]

Once again a Puppy Linux stepped in to do some critical work restoring the murga forum (oldforum).

The sudden loss of the sites and having to move to a new host meant recovering the databases intact and usable and is the most important part of the forum systems. The murga forum's database is large and unwieldy and since it is a conversion from a much older version of software, it can be slightly unstable at times.

Usually it's not that drastic of a procedure to create a new MySQL database and import a SQL dump file to populate the database with all of the correct tables and the data in those tables. But doing this on host provider shared systems and remotely when these SQL files are several gigabytes in size and the SQL dump may not have the exact formatting. The biggest hurdle is the dreaded "mysql server is gone" error or the variety of service unavailable pages and a stopped/crashed import.

The problem is iFastnet shutdown our forums and did the database dumps but not AFTER clearing the caches and deleting the huge search indexes , really inflating the SQL dump sizes.
So a Puppy Linux Bionic64 comes into play.

I have one running Zoneminder merrily chunking away streaming cameras and doing motion detection and other "stuff", on a decent Hiawatha, mariaDB, PHP7.2 setup. Also I do some phpBB forum work with this web server since Bionic64 is loaded with tools and utilities that come into play when working on these types of projects also very useful in a situation like this one, in a big way.

So I take the SQL dump that iFastnet made and go to load into the new host server, but too many problems and failure after failure. And these operations take lots of time when large amounts of data need to be transferred around, and @Clarity SAMBA is suddenly again very important. Some uploads take hours.

Using the command line mysql console I imported the murga database into the Bionic64 MySQL server. Takes about 2 hours all said and done. To load something like this on a remote machine on a host provider is practically impossible or very tricky to accomplish. There are some open source solutions like Bigdump.php which is a little program that loads the data in chunks with a small pause in between into a database. Made for exactly this purpose. Also on hand a freeware SQL splitter which splits very large SQL into smaller files worked well but did not import cleanly. Reason is the format used when the SQL dump is formed. To make it work what is needed is a conversion. Loading the entire database working on it, then exporting another SQL file with the right format.

The point of loading into the Bionic64 servers was to reduce the size of the database by truncating the search index tables manually with phpMyAdmin and breaking the long monolithic lines into individual commands delimited by ";" semi-colons. To do this I had to load the entire SQL file, search through the database tables and truncate caches and search index info. Then perform a mysqldump from the command line, because phpMyAdmin can not do these large exports either. BUT this step is the key.....in the config file my.cfg the most important line must be added:

Code: Select all

extended-insert=FALSE

Then proceed with the export:

Code: Select all

mysqldump -u mysql -p large_database > large_database.sql

Once this is done all the finished product is transferred by upload from Bionic64 to the web site server root and bigdump.php is used to do a staggered import.
Long story short...it worked.

Also Bionic64 easily handled the 30+ crashes I caused trying out different methods to fix all of this as the frustration levels grew, and bounced right back. This Bionic64 running the mariaDB (installed with pkg-cli and the PPM) worked very well under the tremendous loads.

And somebody wanted to mess with this Kennel?

[wizard]

Thanks Rock for the insight into the process.

wizard

[mikewalsh]

Thanks Rock for the insight into the process.

wizard

I echo wizard. And shake my head in the process at the sheer number of veteran Linux users, "stuck in a groove", who remain convinced that Puppy is little more than a toy, or curiosity at best.

I'll bet some mainstream distros would have struggled with this... Yes, the command-line is indeed powerful. but I take the view that it should be reserved for 'serious' stuff, as opposed to forcing itself on ordinary users simply to access their systems and get anything done at all.....and you have all these 'veterans' over at LQ.org - one of the oldest Linux forums still extant - who are utterly convinced that the only way to run Linux is to follow ALL the rules, only do what you're told, when you're told to do it, and to never deviate from "recommended" procedure by so much as a millimetre.

Talk about voluntarily locking yourself into a straitjacket! I always understood that one of Linux's greatest strengths was the ability to be able to 'tailor' your system to make it do exactly what YOU want it to do."Sudo", anyone..? (Mustn't do this, can't do that, not allowed to do 'x', 'y' or 'z'.... *Jeez* )

---------------------------------------------------------

Cheers, Rock! Very interesting insight.

Mike.

[rockedge]

@mikewalsh If I had to "sudo" I would have ended up tossing the entire machine out of the second story windows.

Puppy Linux was very resilient and it's small footprint in RAM allows for the comfortable work at these heavy loads. Some of these crashes would have trashed most full installed OS's. Bionic64 just bounced back like Puppy chasing a ball in the park on a warm sunny day in the park. And as the stress level rises and one starts throwing everything at the problem the last thing one wants to worry about is, can this system handle it? Yes it does and with a backed up save folder I have no worries as I try some crazy out of the box ideas to accomplish the task on hand.

There isn't a manual for this stuff. It's experience and the willingness to take risks with sometimes unorthodox methods. I used a combo of lots of GUI and some command line.

So if one is going down the "sometimes unorthodox methods" path you'll need a good loyal companion willing to go the mile.....that's a trusty Puppy called in this case, Bionic64-9.5

P.S. I am so glad Barry included Hiawatha in the original models, I barely use any other web server at any level of engagement since I was introduced to it.

[bigpup]

The one that did the saving was:

ROCKEDGE!

Plus knowing how to use Puppy Linux and what it can do

Thank you rockedge, for your great support of Puppy Linux!!!

[bigpup]

I am not saying you should or should not do this. Just something to think about.

The old Murga forum does have a lot of useful information and older programs, coded for Puppy Linux.

However, some of the really old stuff, is not good anymore, for Puppy Linux, as it now is.
Some of the really old information, on how to do things, and how Puppy operates, is bad info.
If the old program is anything put a simple script file.
Good chance, it will not run in today's Puppy Linux.

So, there is some very old stuff in the Murga forum, that could be deleted, and never be missed.

[mikewalsh]

@bigpup :-

Granted, it's something that is long overdue for doing. But that is one major task you're proposing there, y'know? It's going to require reading through every single post, line by line, and deciding what needs to stay & what needs to go.

It could take YEARS.....

Mike.

[user1234]

Didn't understand a single word . But great work , rockedge.

Would try breaking down the terms and understanding it tomorrow.

[Flash]

@bigpup :-

Granted, it's something that is long overdue for doing. But that is one major task you're proposing there, y'know? It's going to require reading through every single post, line by line, and deciding what needs to stay & what needs to go.

It could take YEARS.....

Mike.

Centuries is more like it.

I considered curating the Murga forum, but quickly realized that it would be an impossible task for one person. Many if not most of the topics are braided and have lots of irrelevant posts. (Like this one. ) Each topic would have to be read and re-read until I was sure I understood it before attempting to change anything. And the forum software is not made for rearranging on that scale. A better idea might be to read each topic over until you think you've got as much out of it as there is to get, then summarize it into the first post of the topic, splitting the braided topics into separate topics.

You can see why I gave up on the idea.

[p310don]

I'd disagree about curating and removing stuff from the old Murga forum.

There are people out there (masochists) who like to run really old hardware with really old operating systems. To remove something, could ruin it for them.

No doubt false information is not good for current Puppy work. But, this forum is the current forum, and that's fairly clear. The murga forum is pretty clearly labelled as "OLDFORUM" "READ ONLY" etc. It is a fantastic reference and well worth keeping.

Thanks again to @rockedge for keeping it alive (although I actually miss it's old phpBB style!)

[redquine]

There are people out there (masochists) who like to run really old hardware with really old operating systems.

... and (peculiar?) people who have peculiar problems requiring peculiar answers! I sometimes find the clues in the old stuff, such as commands I'd forgotten about or even complete scripts (even if they need a few tweaks to run on the latest Pups).

No doubt false information is not good for current Puppy work.

Not usually - but occasionally, getting it wrong is the first step in getting it right. Or maybe that's just me.

It is a fantastic reference and well worth keeping.

100% agreed, and I'm 100% grateful @rockedge

[rockedge]

I sometimes find the clues in the old stuff......

exactly why I put in the effort to save the murga forum. I look at it like the loss of the Library of Alexandria and I've always wondered since I was a kid, what happened to all of the knowledge it took to build Machu Picchu or any of the other wonders that still exist as ruins of the far past?

I just recently found information buried in the oldforum that saved the day. Plus there are the names and words written by some that have died along the way, that were very enthusiastic about Puppy Linux but mostly about the community around it.

I remember from some time ago a daughter of a member came on line and left a post to tell us her father had passed away but it was his wish that she leave a final note for the Puppy Linux community to reaffirm how important to him the people around the forum were to him.

This stuck with me as well as the desire to preserve so many tricks and tips and insights that would not be replaced.

[mikewalsh]

I remember from some time ago a daughter of a member came on line and left a post to tell us her father had passed away but it was his wish that she leave a final note for the Puppy Linux community to reaffirm how important to him the people around the forum were to him.

This stuck with me as well as the desire to preserve so many tricks and tips and insights that would not be replaced.

.....and I guess there's many - including myself, despite that he was a bit before my time! - who still remember "nooby" with affection, even now. The community did mean a lot to him, that was obvious from much of what he posted.

Some of the software from the early days still works in modern Pups today. Who remembers "ScreenGrab"?

Mike.

[p310don]

It was nooby's sister. He was also on the Facebook group. Also Eric

[rockedge]

Yes !! It was the sister now that I you remind me.

Somehow it's important to preserve the many many hours of time invested by members of the community into projects and the knowledge and experiences gained and documented.

The Ghosts in the Machine.

[courtage]

If I had to "sudo" I would have ended up tossing the entire machine out of the second story windows.

This single thing has kept me with Puppy since 2006.

"ooooh running root is soooooo dangerous ..."

It's *MY* computer, thanks very much, and I'll run whatever commands I like & if I trash my filesystem, that's down to me.

[BarryK]

I remember from some time ago a daughter of a member came on line and left a post to tell us her father had passed away but it was his wish that she leave a final note for the Puppy Linux community to reaffirm how important to him the people around the forum were to him.

It must be about 10 years ago, I received a written letter from a woman here in Australia. Her husband's health declined rapidly and he died, and she just wanted to let me know how much he enjoyed Puppy Linux and the community.

I can't remember his name. Yeah, that old forum has to be kept.

[ndujoe2]

Hooray for Puppylinux, the little computing powerhouse that works.

I use Bionicpup64 all the time