new initrd in testing

[wiak]

So made a big push forward today. Got back to updating WDL initrd init script(s). Very tricky... no wonder I suspended work half way through to play chess... Whilst keeping all the previous kernel options, I've added several new 'w_changes' modes. Have modified the logic of that bit of the code significantly to make the implementation cleaner and easier, but now my brain hurts and I haven't tested any of it as yet - in fact, the changes are so numerous and my initial accuracy so low I doubt the new initrd will work but at least most of the heavy lifting is done and it becomes a matter of me finding my likely syntax errors (and hopefully not any or many overall flaws in my logic/thinking). I'm not in a hurry though. I don't intend making any new releases soon at all, but instead beavering away with a view to releasing reasonably well-tested new versions by the end of the year... But... if I get it working faster than I expect I might well put up a version for dev testing @rockedge - will be a while though.

My previously roughly-described rclone to google drive scripts (encrypted and unencrypted backups) are working really well so my backup procedures are suddenly excellent even when I'm hacking away without using git. So big efficiency boost in my working practices there. I am missing chess though and still have that one game to win to reach my all-time best rating. I'm tempted, but need a week of good sleep first... hahaha

[wiak]

New initrd seems to be working... All the chess playing must have done my brain some good.

There are lot of new 'modes' almost all of which I will probably never use, but put them in for that unexpected need whilst I'm on top of the design.

I have added a considerable number of extra lines to the init script so it is a bit longer than I'd like being now a total of 470 lines of code including tons of comments. However, that's far far less than most other distro initrd/init scripts and can do things most can only dream of (current mainline Puppy init script being 1,370 lines of code, for example)...

The recent additions have resulted in a lot of duplicated code so I can or could no doubt trim it down quite a bit through a few functions, but sometimes that is more trouble than it is worth and doesn't improve efficiency at all so have to look at it and see if I want to do that (I find shell script painful when it comes to minimising close duplication - indirection is so easy in the likes of C, but not busybox ash...).

Like I say, most of the new operating modes are to provide crazy flexibility so many esoteric dreamed file arrangements can be accommodated with ease. For example, here are three grub.cfg stanzas I'm been using in testing a couple of the many available 'modes':

1.
menuentry "WDL_arch64" {
insmod ext2
search --no-floppy --fs-uuid --set 78683fe5-0323-4030-9ddd-39464a8fbf80
echo "Loading vmlinuz"
linux /WDL_arch64/vmlinuz-linux w_bootfrom=UUID=78683fe5-0323-4030-9ddd-39464a8fbf80=/WDL_arch64
echo "Loading initrd.gz"
initrd /WDL_arch64/initrd.gz
}

2.
menuentry "WDL_arch64" {
insmod ext2
search --no-floppy --fs-uuid --set 78683fe5-0323-4030-9ddd-39464a8fbf80
echo "Loading vmlinuz"
linux /WDL_arch64/vmlinuz-linux w_bootfrom=UUID=78683fe5-0323-4030-9ddd-39464a8fbf80=/WDL_arch64 w_changes=RAM1
echo "Loading initrd.gz"
initrd /WDL_arch64/initrd.gz
}

3.
menuentry "WDL_arch64" {
insmod ext2
search --no-floppy --fs-uuid --set 78683fe5-0323-4030-9ddd-39464a8fbf80
echo "Loading vmlinuz"
linux /WDL_arch64/vmlinuz-linux w_bootfrom=UUID=78683fe5-0323-4030-9ddd-39464a8fbf80=/WDL_arch64 w_changes=RAM2
echo "Loading initrd.gz"
initrd /WDL_arch64/initrd.gz
}

4.
menuentry "WDL_arch64" {
insmod ext2
search --no-floppy --fs-uuid --set 78683fe5-0323-4030-9ddd-39464a8fbf80
echo "Loading vmlinuz"
linux /WDL_arch64/vmlinuz-linux w_bootfrom=UUID=78683fe5-0323-4030-9ddd-39464a8fbf80=/WDL_arch64 w_changes=UUID=b9b2cdea-9f5c-4234-ad99-9d9c44f8fa69=/WDL_arch w_changes1=RAM2
echo "Loading initrd.gz"
initrd /WDL_arch64/initrd.gz
}

Number 1 has no w_changes= kernel line. That is the old standard where upper_changes folder gets directly read/write to on the bootfrom media.

Number 2 uses w_changes=RAM1. That mode copies the media upper_changes folder to RAM and that becomes the read/write upper overlay. I then use a special rsync script to copy back any changes (assuming I decide I want to keep the session). DIsadvantage is that whole media upper_changes takes up RAM so only useful for small upper_changes use if you don't have much RAM. In practice, once my system is built the way I want, I either remaster and hence thereafter have tiny upper_changes folder or I rename old upper_changes as NNupper_changes, which makes it a read only overlay for next boot.

Number 3 uses w_changes=RAM2. That's similar to RAM1 mode except the media upper_changes doesn't get copied up into RAM, instead it is simply mounted read-only as second-to-top overlay. That mode is pretty much the same as say DebianDog uses for changes=EXIT. Again a special rsync program is used to merge back the changes as and when desired.

Number 4 is as unusual as the grub.cfg stanza looks. Here the w_changes=specifies a different device/partition from the one booted from (hence different UUID). The additional w_changes1=RAM2 then instructs WeeDog to read-only mount the upper_changes on that device onto RAM (i.e. tmpfs). Number 4 could be used, for example, for putting upper_changes onto a usb flash stick (whilst main boot files are kept on main hard drive) and being able to use any of above RAM modes can per the usual fashion be used to protect the usb stick from too many write cycles.

All in all, with WDL init you can use sfs addon modules, or uncompressed folders as the addon modules, and you can store them anywhere on the system (they don't need to be on the bootfrom folder or partition) and similarly for the upper_changes folder - you can arrange for it to be anywhere, and independently use RAM for it if that is desired. Super flexible... Me? I'm likely to stick with either RAM1 or RAM2 upper_changes. There is also RAM0 mode which is a 'fresh' mode that uses RAM for upper_changes but ignores any previous upper_changes. That's particularly useful mode too since previous upper_changes can always still be used simply by putting a two-digit number on the upper_changes folder name (or turning it into a numbered sfs) - i.e. incremental rollback system).

Don't yet know when public release of this new work will be - too busy at the moment and more to do on overall system, and don't want to encourage the taking-without-acknowledging cherry-picker(s) further.

@rockedge I'll push early version for WDL_Void dev testing soon to you though.

wiak

[wiak]

Don't want to alarm you @rockedge, but whenever I test some new addition to WeeDog initrd/init and it works, part of my brain exclaims almost disbelievingly "oh my goodness, that works!!!". Then a 'better' part of my brain says "em, what about blah, blah, blah - haven't you forgotten that problem that needs a solution?", and then I look at the code again and find I already thought of it months ago and already coded it. Then I feel a great sense of relief, because the thought of coding it frightened the life out of me in terms of complexity, so obviously I must have been overdosed on caffeine months ago that I managed to get these tricky issues already coded (though couldn't even remember doing them already).

So yesterday was a bit like that - I wrote great big complex chunk of code (after weeks of putting off completing it, because it hurt my brain - not many lines of code admittedly - but scary) - one part of that brain nevertheless whispered to me "no chance is all this going to work since too many things need to work in sync with too many other bits", but then it did work, and then I indeed got the thought "oh no, I forgot something" and then I checked, and the extra bit was already there. Miracles really. Main thing that crosses my mind is: "oh my goodness, my partner relies on this system for all their business needs!!!". The miracle then is that after months and months of using it (the previous initrd/init dev version) - all is well with the business - WDL working always solid as a rock. I will never understand how (yet I come to trust it).

And worst thought is that maybe the very next simple improvement I try to make to it (particularly that all important, tricky, initrd/init) will break it completely!!! Anyway, it's truly reaching the stage that I have pretty much nothing I wish to add to that all-important initrd/init (and other priorities in WDL development now anyway) - it pretty much provides all the core function/facilities I need for it. Plenty of other stuff needs to be done of course, but good to know the WDL initrd is providing a solid foundation for the rest of the system(s) that rely on it. So, let's hope, I have not broken what worked so well for so long before - but... for the sake of increasing confidence I will add: all is looking good!!!

[wiak]

Generally, once cleaned up and refined somewhat I consider the WDL init/w_init pretty much complete for the foreseeable future. Except for one thing - size - I want to cut it down in size. A couple of months ago I read and followed discussion around an interesting post by forum member [metion]gumanzoy[/gumanzoy] where he suggested to fredx181 of DebianDogs using "modules copy method from linux-live/Slax" for assembling any required boot modules in initrd: https://github.com/Tomas-M/linux-live/b ... mfs_create

viewtopic.php?p=24488#p24488

Apparently the technique Tomas M employs helps create much smaller initrd files (the size and compression of boot modules being the major size issue of the initrd). I don't know when I'll get round to it, but I plan on studying that Slax method for use in WeeDogLinux too - the current module loading is a simple while load load, which is the only bit of code I didn't reinvent (its just the same method as used in Fred's previous Porteus boot initrd.

Of course, for a given known system, the smallest initrd you can make is one that just uses the exact modules needed to find the boot partition to allow switch_root to the real root filesystem stored there, but for a general boot any machine scenario it seems Slax has that one better worked out. Time will tell and that is not my current priority (depends how simple it is to implement whether I do it quickly and also on me taking the time to read up on the method more carefully).

At some stage soon, I will of course have to merge the new init/winit code into new build_weedog_initrd script, but that is a reasonably simple job that can be completed very quickly - the latest dev build_weedog_initrd script already caters for init and w_init, but I've since major upgraded w_init in particularly so have to merge these changes in. I expect I'll do that very soon and prior to any possible Tomas M related module selection improvements.