Numbered upper_changes script logic

[geo_c]

While writing a reply to this topic: viewtopic.php?p=90710#p90710 I started thinking about how a script could easily create a new read-only numbered upper_changes to use on the next boot.

If it's easy enough to copy the current upper_changes to a numbered layer while running the system, a script would have to probe and perhaps ask the user about naming and numbering it. Once copied it wouldn't be mounted until next boot, so shouldn't be a problem sitting in the system directory until then.

But what about deleting (or renaming) the currently used upper_changes before rebooting. Seems to me that would have to be implemented in the shutdown procedure, or the boot script. Is it possible in the w_init to look for a marker file (or other log) indicating that the upper_changes directory had been backed-up/renamed for the next boot, and upon finding that log entry, rename or delete the working un-numbered upper_changes before loading the new read-only layer and creating a fresh upper_changes?

[wiak]

While writing a reply to this topic: viewtopic.php?p=90710#p90710 I started thinking about how a script could easily create a new read-only numbered upper_changes to use on the next boot.

If it's easy enough to copy the current upper_changes to a numbered layer while running the system, a script would have to probe and perhaps ask the user about naming and numbering it. Once copied it wouldn't be mounted until next boot, so shouldn't be a problem sitting in the system directory until then.

But what about deleting (or renaming) the currently used upper_changes before rebooting. Seems to me that would have to be implemented in the shutdown procedure, or the boot script. Is it possible in the w_init to look for a marker file (or other log) indicating that the upper_changes directory had been backed-up/renamed for the next boot, and upon finding that log entry, rename or delete the working un-numbered upper_changes before loading the new read-only layer and creating a fresh upper_changes?

I'd have to experiment with all above some day. I have a feeling I have renamed an in-use upper_changes directory to become a numbered (top-layer) upper_changes and then immediately reboot without issues, but I may be wrong. Worth simply trying that whilst in RAM2 save on demand mode; assuming nothing from that session needing saved worst that could happen would be system freeze... Certainly, if you wanted to save your current session changes (assuming you were currently in RAM2 save on demand mode) you'd need to do that prior to the renumbering followed by rebooting.

[geo_c]

I'd have to experiment with all above some day. I have a feeling I have renamed an in-use upper_changes directory to become a numbered (top-layer) upper_changes and then immediately reboot without issues, but I may be wrong. Worth simply trying that whilst in RAM2 save on demand mode; assuming nothing from that session needing saved worst that could happen would be system freeze... Certainly, if you wanted to save your current session changes (assuming you were currently in RAM2 save on demand mode) you'd need to do that prior to the renumbering followed by rebooting.

After giving all this careful thought, I've come to the conclusion that I'm simply doing things the hard way, rather than running upper_changes in ram, w_changes=ram2, I've been writing to disk on demand. So while running from disk I did try renaming upper_changes once, and the system crashed as I expected. But I think there's absolutely no reason that while running in ram that the changes couldn't be saved and renamed and then immediately shut down without issue.

Since I've installed some very large applications, libraries and usr/share/docs over the last 6 months, all my upper_changes total about 10GB in KLV-airedale, so I don't think I can be running them in ram.

At this point I'll be running a new install in ram and restructuring and experimenting.

[fredx181]

....
Since I've installed some very large applications, libraries and usr/share/docs over the last 6 months, all my upper_changes total about 10GB in KLV-airedale, so I don't think I can be running them in ram.

At this point I'll be running a new install in ram and restructuring and experimenting.

AFAIK when booting with w_changes1=RAM2 , the changes running in RAM are JUST the recent changes you make (located in /mnt/layers/RAM/upper_changes/ ) , the already earlier made changes in your upper_changes save-folder (if loaded at boot with w_changes=....) are NOT running in RAM (edit: goes for your numbered upper_changes folders too, not running in RAM (unless w_copy2ram is specified)).

[wiak]

Since I've installed some very large applications, libraries and usr/share/docs over the last 6 months, all my upper_changes total about 10GB in KLV-airedale, so I don't think I can be running them in ram.

You seem to be misunderstanding here. w_changes=RAM2 is a mode that uses an empty upper_changes in RAM; any changes made to that during that session can then be saved back on demand to the on-disk-media upper_changes so becomes current again on next boot. When using w_changes=RAM2 mode the on-disk-media upper_changes gets mounted (not copied) to the top read-only overlayfs layer (i.e. one layer beneath that 'empty' in RAM read-writable upper_changes).

i.e. w_changes RAM2 mode is not about running 'from' RAM but rather about using RAM for a temporary session-based RAM upper_changes that does not use up any RAM at all initially.

FirstRib does provide a facility for actually running everything from RAM (like Puppy Linux tends to do), but personally I pretty much myself never use that facility since, especially on older low-powered systems that don't have much RAM in the first case why oh why would you want to fill up your precious low resource RAM prior to doing anything?????

If you insist on actually loading all the sfs files and so on into RAM before using them you should add the following option to the grub kernel line: w_copy2ram. As I say, I don't generally advise that (also takes time to load in the sfs files and if uncompressed layers even worse...). But I do recommend using save-on-demand w_changes=RAM2 mode which only 'mount' the on-disk-media upper_changes, but doesn't physically copy it into RAM prior to use... And save on demand w_changes=RAM2 mode means exactly that, you can save back any session changes at shutdown via the save2flash utility offered then (or at any time in between by running save2flash utility manually), but saving any session is thus optional, which is why the mechanism is so powerful.

Like I said, I haven't had time to try it again, but I have a memory that even though on-disk-media upper_changes is actively mounted when using w_changes=RAM2 mode I have a recollection it was nevertheless possible to rename the on-disk-media upper_changes (with a number for rollback use) without the system crashing. Best to just try it. I would but I'm in Zorin full install with no time really to experiment for now.

[geo_c]

Like I said, I haven't had time to try it again, but I have a memory that even though on-disk-media upper_changes is actively mounted when using w_changes=RAM2 mode I have a recollection it was nevertheless possible to rename the on-disk-media upper_changes (with a number for rollback use) without the system crashing. Best to just try it.

Thanks @wiak and @fredx181

That clears it up completely. I really was confused and thinking ram2 meant loading the whole system, which is of course impractical in most cases.

So I should be able to change the boot parameter to w_changes=RAM2 without any further modification. And just store the most recent changes in ram.

All this discussion has got me thinking a lot about how versatile the KL setup really is. And I'm curious to rebuild a pristine system and structure it more logically, then maybe compress my configured changes to sfs and have it all a little more "immutable" as they say.

So if I did something like boot into a clean, new upper_changes with w_changes=RAM2, and then install a large application, clean the cache, save to disk, and rename to XXupper_changes, and reboot, then that resulting XXupper_changes would in essence just contain the newly installed application which could be loaded and unloaded, provided I didn't install or update any libraries needed by other applications or the system.

If I wanted to do the same with another application, one that might share libraries with the previously installed application, I could boot without the previous application loaded, and then install the new one, which would also contain the dependencies, and thereby having a kind of portable sfs that could be mounted or not mounted at boot by changing the two digit prefix.

[wiak]

So if I did something like boot into a clean, new upper_changes with w_changes=RAM2, and then install a large application, clean the cache, save to disk, and rename to XXupper_changes, and reboot, then that resulting XXupper_changes would in essence just contain the newly installed application which could be loaded and unloaded, provided I didn't install or update any libraries needed by other applications or the system.

Yes, indeed. In fact if new app installed during one such boot session, you can hide its installation from apt if you wish, prior to making the numbered sfs file (or uncompressed directory) of the session, in the manner I describe in KL howto here: viewtopic.php?p=84173#p84173

[williwaw]

So if I did something like boot into a clean, new upper_changes with w_changes=RAM2, and then install a large application, clean the cache, save to disk, and rename to XXupper_changes, and reboot, then that resulting XXupper_changes would in essence just contain the newly installed application which could be loaded and unloaded, provided I didn't install or update any libraries needed by other applications or the system.

Yes, indeed. In fact if new app installed during one such boot session, you can hide its installation from apt if you wish, prior to making the numbered sfs file (or uncompressed directory) of the session, in the manner I describe in KL howto here: viewtopic.php?p=84173#p84173

can an app, whether stored as a sfs or not, be loaded during a session?
or is a reboot required?

[wiak]

So if I did something like boot into a clean, new upper_changes with w_changes=RAM2, and then install a large application, clean the cache, save to disk, and rename to XXupper_changes, and reboot, then that resulting XXupper_changes would in essence just contain the newly installed application which could be loaded and unloaded, provided I didn't install or update any libraries needed by other applications or the system.

Yes, indeed. In fact if new app installed during one such boot session, you can hide its installation from apt if you wish, prior to making the numbered sfs file (or uncompressed directory) of the session, in the manner I describe in KL howto here: viewtopic.php?p=84173#p84173

can an app, whether stored as a sfs or not, be loaded during a session?
or is a reboot required?

Despite FR initrd using overlayfs, an app can actually be loaded during a session if you include the special sym-link-method based sfs-load utility system script fredx181 of the DebianDogs put together (considered experimental since more testing required according to fredx181, but certainly worked). The method itself isn't actually experimental - in fact it is pretty much the mechanism that has always been used by Tiny Core Linux to load its tcz (sfs) files into the running TCL system (i.e. doesn't actually need aufs or overlayfs to work). The 'worry' I think fredx181 had was that he felt more time was needed to test all symlinks used get cleared up correctly whenever the sfs is unloaded; I haven't been using sfs-load at all so I can't comment further about it, except to say it does work and no doubt can be made to work reliably (no reboot required). Having said that, using numbered overlayfs layer naming instead, followed by a reboot, always works and works well. Certainly, aufs supports sfs-load-on-the-fly inherently, but personally, if I wanted the admittedly useful facility, I'd prefer to use the well-established symlink mechanism since aufs requires a patch and recompile that is not supported by Linux kernel team and thus potentially dangerous. Symlink approach, on the other hand, doesn't involve any kernel core-level patch that may or may not later remain supported by any dev team.

An alternative are big apps installed completely into the likes of /opt directory. These can simply be mounted to a mount point under /opt and run from there without any sfs-load facility per se being required (you can mount an sfs designed that way of course). Slimjet browser (chromium-based) can be installed in that way for example (alternatively, as you no doubt know, mikewalsh/fredx181 produces a portable version you just need to copy somewhere on your system; doesn't even need specially mounted then).

I can't remember where that symlink sfs-load utility is discussed in KL threads; hopefully one of the others habitually using KL distros can point you to it. Just remembered that Sofiya has included that (but not tested) in KLA-OT2baseCE:
viewtopic.php?p=92141#p92141
Bluetooth(blueman), Touchpad Setup (libinput), USBimager (for burning ISO), Gpick (Color picker), Changing wallpaper(Changing the desktop wallpaper), GtkHash (Checksum Calculator), Create SFS(packaging in SFS), SFS-Load- loading sfs on the fly (not tested)
but I can't remember where the main thread posts discussing that sfs-load symlink-based mechanism is.

[rockedge]

Despite FR initrd using overlayfs, an app can actually be loaded during a session if you include the special sym-link-method based sfs-load utility system script fredx181 of the DebianDogs put together (considered experimental since more testing required according to fredx181, but certainly worked). The method itself isn't actually experimental - in fact it is pretty much the mechanism that has always been used by Tiny Core Linux to load its tcz (sfs) files into the running TCL system (i.e. doesn't actually need aufs or overlayfs to work). The 'worry' I think fredx181 had was that he felt more time was needed to test all symlinks used get cleared up correctly whenever the sfs is unloaded

@fredx181's SFS-Load mechanism is installed in KLV-Airedale because I use it on occasion and I have yet to have any problems when using it.
I have tested the orphaned symlink clean up by forgetting to unload a SFS that is mounted using the loader and rebooting or shutting down. Not had any issues to this point by an early.

That brings us to perhaps including the mechanism in KLV-Spectr for further testing and potential use analysis.

[fredx181]

The 'worry' I think fredx181 had was that he felt more time was needed to test all symlinks used get cleared up correctly whenever the sfs is unloaded

Not really, I think that the mechanism to remove the symlinks when unloading the SFS or after reboot is rather solid (even when the system crashes, the symlinks will be gone after reboot).
The 'worry' that I have is that not all cases are tested, e.g. what can happen if the package manager installs files that are just added to the system as being symlinks (by SFS-load), or: what happens when remastering the system while an SFS is loaded, and probably more that I cannot think of right now.
As I said earlier, I like the "Alternative SFS Load" much more: viewtopic.php?t=8004 as it doesn't "disturb" the system (running in chroot) so much more safe IMO.
(but the disadvantage is that applications won't show in the default Menu (has it's own "menu" by displaying a small GUI).
To be honest, I almost never use 'Sfs-load on the fly', so only did basic testing.