Puppy Linux Discussion Forum

Under development.

I'm currently in the process of further expanding the flexibility of the already flexible WDL initrd/init (whilst keeping its init size but a fraction of most others around the place).

Previous versions of WDL initrd/init catered for 100 overlayfs layers, which allows for tons of addon sfs modules I know. However, since kernel 3.1 the loop device limit is no longer really limited (they are now allocated dynamically as required - apparently up to theoretical 2^20). No doubt overlayfs itself has its performance limits, in terms of layers used, but as technology moves on I believe it is good practice to design for no limits or at least very high limits since most apparent 'limits' generally prove to be artificial in the end (as processing capabilities improve).

Fortunately a one-line code change allows WDL initrd/init to accommodate 1,000, or 10,000 or whatever sfs addon module layer numbers, so in newest code I'm simply using a base variable to specify the range required. Main reason is not that I expect to see hundreds of layers used in the near future (though you never know - tinycorelinux uses plenty of loop mounts as part of its design) but rather to allow for easier numeric range allocation (the larger the range is, the easier it is to allocate a range for save-changes rollback sfs module numbers for example without danger of running out of overlayfs layer numeric filename space). So in my next build I'll be testing with firstrib_rootfs at position 100firstrib_rootfs (rather than 01firstrib_rootfs) and with upper_changes_rollback sfs numbers starting at a somewhat arbitrary value of 200. That spacing makes it easy to add as many modules as perhaps reasonably wanted either below or above 100firstrib_rootfs (and hence not-overwriting or overwriting files as desired) whilst leaving tons of numeric range for upper_ranges rollback numbers (anything say in range 200 to 1000). (Any such base number choices are however entirely arbitrary and any WDL system builder can re-allocate as they wish). i.e. I'll be using NNN instead of NN (but with one line code change could use overall range of NNNN or more if ever that level of granularity proves useful in the future...).

That ability to scale is a major reason why I believe the numeric (NNN...) sfs module filename convention implemented in the WDL unique _addlayers algorithm is far superior to the likes of a limited adrv, bdrv, cdrv and so on alphabetic ordering convention (though of course it is possible to artificially assign/hack some kind of numeric alias, as a less than perfect fudge, to such generally difficult-to-scale non-ordered or simple alphabetical ordered layers types of system filesystem layer design).

Anyway, expanding the range scale (to any range desired) is a trivial change to the efficient _addlayers function in WDL init, but to complement that I'm expanding the w_changes mode capabilities to improve save changes functionality with a particular intention to make rollback handling easier to automate. The NNNupper_changes rollback files will also, for convenience, include date in their filename (though that won't be done in the init, but rather in separate rollback management utility). Having said that, I'm not personally so interested in the final implementation of the userland rollback utilities (could be commandline or GUI, for example, and lots of ways of doing that), but want to make sure the WDL_initrd/init provides sufficient facility to easily allow such schemes. User/developers will of course be able to write their own userland utilities to manage rollback and so on in whatever way they find most useful (but whatever method used does rely of course on whatever flexibility/functionality the core WDL_initrd/init layering scheme provides - which, fortunately, is quite a lot... and particularly with the easy text-based plug-in extension/modification capability it also provides).

Currently I'm coding additional w_changes options/modes: EXIT0, EXIT1 (already completed some while back) and EXIT2. I'll explain what these are all for later, but suffice to say they are concerned with management of what gets saved (e.g. out of RAM temporary store), and when, (including providing for easier management of changes rollback date-assigned snapshots). Sounds complicated, but actually only simple additions to existing initrd/init code though I am re-writing the logic of the existing w_changes code section involved since I can simplify it somewhat. Clearly will have a lot of testing to do thereafter prior to any public release so this is just an update in case anyone wonders if development is currently active or not - it very much is.

EDIT: In practice I'm planning to use "memory jogging" option names for EXIT0, EXIT1, EXIT2, and so on, to help indicate what they are used for (saves having to keep referring to documentation for more detailed reminders...).

Main thing I remain concerned with is not to break what already works so efficiently and well... there certainly won't be any significant bloat - the init only needs to provide all the core functionality, which is quite basic/fundamental really - external utility apps complete the job. Most important, for me, is that WDL initrd/init remains distro agnostic, which means basically decoupled from the main root filesystem used with it - that's why WDL can be used with most any other distro's root filesystem to provide WDL modular multi-layer addon capabilities.

wiak

What is WDL ?

https://weedoglinux.rockedge.org/viewto ... ?f=41&t=28
please read ...

peace

So this part of the development is taking a while. Despite its tiny size, WDL init was designed for flexibility, which includes not only tons of module layers, but also that some sfs modules (and/or uncompressed directories or a mixture of both) can be in boot directory, but others can be on any other directory on the system, or some or all can be embedded within the initrd itself (or any mixture of these possibilites). Furthermore, all or only some of these addon sfs modules (and/or uncompressed dirs!!!) can be copied into tmpfs RAM storage prior to use, if so desired, so the overall logic of adding new save persistence modes has to correctly cater for all that existing flexibility, so I'm taking it slowly to try and make sure I get it as correct and efficient as possible... Simple when only a few sfs modules involved (such as Puppy type arrangement) and/or these are all stored in one location and either all copied to RAM or none copied to RAM, but WDL doesn't have such configuration/usage restrictions at all...

1. In newest dev initrd/init, simplest w_changes mode is: w_changes=media (or simply w_changes="", which is default anyway if not included on grub kernel line), which results in direct writes to the boot media (read/write top layer) upper_changes folder.
2. Next simplest is w_changes=RAM0, which puts an empty (read/write top layer) upper_changes folder onto tmpfs, but ignores the boot media upper_changes folder altogether.
3. The next one, w_changes=RAM1, I've been using in practice for a month or two now. That one is a very simple way of combining media upper_changes with RAM upper_changes; all that happens is that at boot time the whole of boot media upper_changes folder gets copied as is into RAM byte by byte, so that is RAM expensive (it uses RAM so only intended for cases where media upper_changes folder is kept small such as when using alongside regular NNupper_changes.sfs rollback snapshots). Despite that actual RAM usage, the method does have the advantage that there is no merging needed at end of session - the whole RAM stored (read/write top layer) upper_changes can simply be rsync'd (basically the delta of changes) back to the permanent copy stored on the boot media. Currently I just use the following simple (rough and basic) script for that mode if I want to save changes at the end of a session (otherwise everything is just done in RAM and no writes to my hard disk). Really, it is just a single rsync commandline (but with a few choice removals first):

#!/bin/sh
R="/mnt/layers/RAM"
bootstore="/mnt/sda2/WDL_arch64"
echo -e "\e[33mSaving upper_changes from RAM to bootstore... Please wait...\e[0m"
rm -rf ${R}/root/.cache/chromium ${R}/var/cache/pacman ${R}/var/log/journal
rsync -av --delete $R/upper_changes $bootstore
echo -e "\e[33mSaving upper_changes from RAM to bootstore... Please wait...\e[0m"
sync;sync

To be honest, I just spent a couple of minutes thinking about that script and putting it together, so I haven't really considered what I prefer not to include in that rsync delta copy. For more complicated fine-tuning rsync can accept a filelist detailing exactly what is to be removed, but for the moment that crude rm -rf hatchet job is good enough for my tests. So the above code may well have imperfections, but so far has worked fine for me (with the tiny external disk upper_changes folders I use). Depending what I'm doing, I often choose to not save anything, but if I want to save the session I just run that simple script (previously I just manually typed a suitable rsync command...).

As I say, I use this "w_changes=RAM1" simple RAM-greedy method in conjunction with rollback upper_changes; currently I have a 50upper_changes folder, that holds all the main changes I previously added to the original installed system, which via WDL init is automatically loaded read-only into next to top layer (top layer being that purposively kept small RAM/upper_changes folder). One useful thing about this simplistic mode is that once the media upper_changes (e.g. from hard drive) is copied up to RAM/upper_changes, I can do whatever I like with the hard drive version (since it is not actively being used during a session anyway) - hence, once rsync is done 'back' to it , I can then simply rename the folder 51upper_changes_date (via a script or manually) as the snapshot rollback file ready for the next boot. Another part of WDL init flexibility is that it doesn't matter if I leave that as an uncompressed directory, though generally I'd use mksquashfs to convert it into an sfs since takes up less disk space...

RAM-saving mode:

Of course, it is not necessary to copy whole media upper_changes up to tmpfs RAM. Instead the media upper_changes can itself simply be mounted read-only in the second from topmost layer (i.e. not copied into RAM at all, simply mounted ready for use), and the top layer can then be a read/write empty upper_changes folder in RAM. Then at end of session (or whenever required) that top layer can get merged into the on-disk upper_changes version:

That's mode w_changes=RAM2. Again you can use rsync in the save-back utility script, but it's a little bit more complicated, than for w_changes=RAM1 case, in that care has to be taken to handle whiteout file types (such overlayfs whiteout types being used to mask underlying files/directories being deleted). But as I say, I'm taking things slowly since the overall logic of adding new w_changes modes to an init as flexible as that provided by WDL needs careful thought (I find I need to have had a good sleep prior to such init mode additions, and frankly sometimes wonder how on earth I managed to write WDL init - I must have been on form at the time...) - that's more important overall than the final merge back script, which isn't too tricky anyway. I myself actually currently prefer using rollback sfs and a very small (or zero-sized) main upper_changes, so RAM0 and RAM1 are the modes I tend to personally use anyway (but that may change - when I used DebianDogs I used to use changes=EXIT mode, which I think is pretty much the same as w_changes=RAM2 mode).

To be continued...

I read it with interest. I do not plan(yet) to use WDL and don’t know yet why I need 100(+) overlayfs layers. But I follow with interest.

Grey wrote: ↑Mon May 17, 2021 3:09 pm

...and don’t know yet why I need 100(+) overlayfs layers. But I follow with interest.

For most people (including myself) there is unlikely to ever be a 'need' for 100(+) overlayfs layers. However, does no harm since no-one is forced to use them all; the order of layers is what is important sometimes and the huge numeric range makes it easier to allocate ranges for specific purposes with almost no danger of likely extra numeric layer 'space' turning out to be needed. But I wouldn't be brave enough to say 100+ layers would 'never' be needed - it is possible someone, for example, decides to design a new distro that works similar to say tinycorelinux. That is, one that uses dozens, and even hundreds of single app sfs files to assemble the whole system pristine on each boot occasion. Tinycorelinux does that via symlink sfs techniques, but WDL could arrange the same via layers. I'm sure other possibilities/hybrids can and will arise (such as could arise from rollback/snapshots assuming overlayfs/underlying-system became efficient enough to tolerate the possibility of thousands of layers). In fact I might well be able to build a WDL_tinycorelinux using WDL init and tinycorelinux tcz sfs files... (the kernel used would need overlayfs support of course, which actual tinycorelinux doesn't need).

Main thing is, no harm to have such huge range of layers possibility, and in research it is good to aim for no limit designs rather than produce a design with what seemed like reasonable limits...

It was once thought that no-one would likely ever need as much as 1GB RAM, or even 100 MB hard disk space...

wiak

wiak wrote: ↑Mon May 17, 2021 9:33 pm

It was once thought that no-one would likely ever need as much as 1GB RAM, or even 100 MB hard disk space...

Those were good times. "640 kB ought to be enough for anybody".
Supposedly said by Bill Gates in 1981. Although Bill Gates himself claims that he did not say anything like that
Ironically, this is now the size of the floppy disk image(.trd) for ZX Spectrum emulators - in ROX properties 640K(655360 bytes)