Remaster KLV-Airedale

[fredx181]

Here's a remaster program (GUI) for KLV-Airedale (xbps package, install and run from Menu > System > KLV Remaster).
It's using gtkdialog for the main GUI and Xdialog for some other dialogs and the progress bar (mksquashfs progress creating SFS)

klv-remaster-1.0_0.noarch.xbps

klv-remaster

(4.65 KiB) Downloaded 83 times

Based on a concept that I earlier shared (for Weedog) but much more extended.
The (simple) concept is to make a SFS from the running system "/" and use the (advanced) mksquashfs exclude and wildcard options (no mounting or overlay involved this way) .
Everything will be merged in a single .sfs, not only the changes you made, but also the contents of the extra .sfs modules loaded at boot in the frugal install.

Screenshot.png (55.02 KiB) Viewed 2386 times

Read the "Help" and click the "Exclude" button to specify which files/folders you'd like to exclude (e.g. cache files or browser profile folder).
This exclude option is not the most user-friendly perhaps, but you can specify exactly what to exclude from the remaster.
If the purpose is to share the remaster with other people, you must know exactly what to exclude (e.g. don't want to share personal files or passwords stored in the browser profile)

[fredx181]

Noticed that with the remastered .sfs, the system boots quicker and runs more smoothly too, not sure why, perhaps because it's single SFS only

[geo_c]

Noticed that with the remastered .sfs, the system boots quicker and runs more smoothly too, not sure why, perhaps because it's single SFS only

I'll give this a try, it might boot faster because it's not necessary to read various levels of upper changes and white out the deleted files. In my case that is now 13 layers of upper_changes. Although KLVseems to load those pretty quickly. The overall boot speed for KLV seems to me faster than fossapup. I'll have to clock them both on the same machine.

One question. Once the remaster is made, should the /work directory should be deleted? And I suppose it would be easy enough to leave the 13 levels of upper_changes in the /home directory as a backup by prepending them with a non numerical character to keep them from being loaded.

Oh also, the remastered sfs should replace the 00KLV-airedale_rootfs.sfs, and not the others? Or should it also replace 00modules and 01firmware and 10gtkdialog?

[fredx181]

One question. Once the remaster is made, should the /work directory should be deleted? And I suppose it would be easy enough to leave the 13 levels of upper_changes in the /home directory as a backup by prepending them with a non numerical character to keep them from being loaded.

Mmm... to be honest, I don't know about multiple levels of upper_changes, can you explain a little about it ? (I have only one upper_changes loaded at boot, or perhaps I misunderstood what you mean to say).
Yes the /work directory should also be deleted (or back-upped) along with upper_changes.
EDIT: I think I see now, you load one upper_changes at boot, but having multiple backups, you can leave these, since they are not loaded at boot.

Anyway, it's in fact very simple, running this remaster script just makes a new .sfs from everything contained in the current running KLV system '/' (including current changes made/loaded (deletions also) and contents of extra sfs modules loaded), excluded are e.g. contents of /proc/, /sys/, /dev/, /run/, /mnt/ and what you (optional) manually chose to be excluded (in excludelist).

Oh also, the remastered sfs should replace the 00KLV-airedale_rootfs.sfs, and not the others? Or should it also replace 00modules and 01firmware and 10gtkdialog?

Should replace all, as everything is merged into one, so then 'remastered' frugal install folder should contain ONLY "01KLV-airedale_rootfs-remaster.sfs' (or how you named the remaster .sfs) , vmlinuz, initrd.gz, w_init (and 'fresh' changes setup).

Hopefully above info is clear enough, if not I'd be happy to explain more.

[wiak]

One question. Once the remaster is made, should the /work directory should be deleted? And I suppose it would be easy enough to leave the 13 levels of upper_changes in the /home directory as a backup by prepending them with a non numerical character to keep them from being loaded.

Oh also, the remastered sfs should replace the 00KLV-airedale_rootfs.sfs, and not the others? Or should it also replace 00modules and 01firmware and 10gtkdialog?

You shouldn't delete the work directory while a booted run session is in progress of course. If thereafter you are booted into a different distro you could delete that KLV work directory if you wished, prior to rebooting into KLV, and work/ directory would be auto-regenerated on boot, but I don't think you need to delete it at all.

Since the remaster contains whole of previously running system, you are correct, you would need to disable the previous upper_changes (including the non-numbered previous top read-write upper_changes folder) via that renaming method you describe (for example, just put a D in front of all such file names) which allows them to be kept as a backup (for use with pristrine original 07KLV-airedale_rootfs.sfs - in your post you accidentally I think used the wrong number, 00, for the KLV sfs. We normally reserve 00 for modules).

An alternative to an all-system remaster such as the one described would be a 'merging' of the upper_changes sequence of files only - I think rockedge mentioned an alpha version of a 'merge utility' also being in existence, which may or may not need some work on it to ensure it works correctly; I believe that alternative "merge utility" probably creates a new overlay only consisting of all the upper_changes components and the merged result is then resquashed by the utility - that should certainly work.

Probably the most common firstrib-based system wouldn't have separate 00modules and 01firmware layers since modules and firmware would be built-in to the underlying root filesystem by default anyway. However, if a huge kernel is being used (such as one from Puppy Linux that has all media access drivers built directly into the kernel) then there is an advantage to having modules and firmware provided as addons (00<modules_name> and 01<firmware_name>) since makes it very easy to swap kernel/modules/firmware then. I'm surprised the system is at all significantly faster or smoother when everything merged together though - I suppose there will be some speed overhead having to keep track of layers though so that must be why (also a little bit more RAM uses for each mounted layer I imagine).

What I know does make a significant difference to smoothness/speed of running system is the level of compression used in the sfs files - lz4 having been tested as pretty fast, but uncompressed layers being fastest since no decompression by CPU required. Bearing that situation in mind, I suppose one main sfs only would therefore be a bit faster/smoother anyway (and could be improved further by using low compression on the result or unsquashed version). Boot time is a different matter since depends also if copying addon layer files to RAM (personally I don't do that and not the default), which takes long..... time if uncompressed or low compression being used.

I can't say I know anything about the technical underlying system-level operation of overlayfs though - so your guess would be as good as mine in terms of performance when multiple layers are involved and so on.

[fredx181]

.... I'm surprised the system is at all significantly faster or smoother when everything merged together though - I suppose there will be some speed overhead having to keep track of layers though so that must be why (also a little bit more RAM uses for each mounted layer I imagine).

What I know does make a significant difference to smoothness/speed of running system is the level of compression used in the sfs files - lz4 having been tested as pretty fast,....

Yes, forgot to say that my remaster test was with using LZ4 compression, most probably that's a factor to make it run more smoothly.

[wiak]

EDIT: I think I see now, you load one upper_changes at boot, but having multiple backups, you can leave these, since they are not loaded at boot.

I believe geo_c is meaning where after some sessions he renames upper_changes to, say, 50upper_changes (and optionally compresses that to an sfs if wanting to save space); a new upper_changes read-write directory gets created and used on subsequent boot. Later can rename new upper_changes to say 51upper_changes and so on. All such upper_changes (numbered and the non-numbered one) are required for subsequent boots - they are not backups per se but rather incremental upper_changes rollback addons.

[geo_c]

EDIT: I think I see now, you load one upper_changes at boot, but having multiple backups, you can leave these, since they are not loaded at boot.

I believe geo_c is meaning where after some sessions he renames upper_changes to, say, 50upper_changes (and optionally compresses that to an sfs if wanting to save space); a new upper_changes read-write directory gets created and used on subsequent boot. Later can rename new upper_changes to say 51upper_changes and so on. All such upper_changes (numbered and the non-numbered one) are required for subsequent boots - they are not backups per se but rather incremental upper_changes rollback addons.

Yes @wiak has it right. I don't compress the numbered upper_changes as I rename them, but I've been incrementally renaming upper_changes to 51upper_changes and so on. I currently have upper_changes directories numbered from 51-63. So the system is quite developed. This allows me to experiment, perhaps break the system, and delete the top level upper_changes and reboot to where the system was previously. But at some point I need to consolidate. Preferably that would be merging all the upper_changes directories to one, like a pupsave, but I'm open to a remaster. No reason not use both actually. Where things are now would be a decent base system to remaster as a root file system, but I am wondering how that will work since the total size of numbered upper_changes is in the 3GB territory, that might make root file system too large to load in ram. I expect it to decrease in size on a merger as some of the early upper changes may contain large data that I later deleted. But I won't know until I try.

edit: actually those upper_changes folders are totalling 3.7GB, but that may be because I copied a lot of icons and things that I may have later deleted, but are sitting in the previous upper_changes. However I have about 10 icon sets installed, and I may need to experiment with symlinking those.

[rockedge]

EDIT: I think I see now, you load one upper_changes at boot, but having multiple backups, you can leave these, since they are not loaded at boot.

I believe geo_c is meaning where after some sessions he renames upper_changes to, say, 50upper_changes (and optionally compresses that to an sfs if wanting to save space); a new upper_changes read-write directory gets created and used on subsequent boot. Later can rename new upper_changes to say 51upper_changes and so on. All such upper_changes (numbered and the non-numbered one) are required for subsequent boots - they are not backups per se but rather incremental upper_changes rollback addons.

This is what this merge script does. It has not been fully finished as in a GUI control. But it will take the incremental XXupper_changes and merge as many as it finds into one XXupper_changes.sfs

This script uses file names XXchanges.sfs which were created with the save.sh script that accompanies it. These names can be altered in the merge script, Also the merged output SFS prefix number can be altered. As is the script finishes with 30changes.sfs

[geo_c]

This is what this merge script does. It has not been fully finished as in a GUI control. But it will take the incremental XXupper_changes and merge as many as it finds into one XXupper_changes.sfs

This script uses file names XXchanges.sfs which were created with the save.sh script that accompanies it. These names can be altered in the merge script, Also the merged output SFS prefix number can be altered. As is the script finishes with 30changes.sfs

Wow, that script is short and sweet. I was expecting something much more cryptic and epic!

edit: So I take it I would need to change the first line to the location of my KLV install. Would I need to edit any other lines?

[fredx181]

EDIT: I think I see now, you load one upper_changes at boot, but having multiple backups, you can leave these, since they are not loaded at boot.

I believe geo_c is meaning where after some sessions he renames upper_changes to, say, 50upper_changes (and optionally compresses that to an sfs if wanting to save space); a new upper_changes read-write directory gets created and used on subsequent boot. Later can rename new upper_changes to say 51upper_changes and so on. All such upper_changes (numbered and the non-numbered one) are required for subsequent boots - they are not backups per se but rather incremental upper_changes rollback addons.

Ah, thanks, hopefully I understand now, so then if all these XXupper_changes .sfs's are loaded at boot in the session at the time of running my remaster script, they will all be merged into single SFS, as all extra sfs modules.
EDIT: Late here now, saw earlier responses, read tomorrow.

[wiak]

EDIT: I think I see now, you load one upper_changes at boot, but having multiple backups, you can leave these, since they are not loaded at boot.

I believe geo_c is meaning where after some sessions he renames upper_changes to, say, 50upper_changes (and optionally compresses that to an sfs if wanting to save space); a new upper_changes read-write directory gets created and used on subsequent boot. Later can rename new upper_changes to say 51upper_changes and so on. All such upper_changes (numbered and the non-numbered one) are required for subsequent boots - they are not backups per se but rather incremental upper_changes rollback addons.

Ah, thanks, hopefully I understand now, so then if all these XXupper_changes .sfs's are loaded at boot in the session at the time of running my remaster script, they will all be merged into single SFS, as all extra sfs modules.

That's correct.

[fredx181]

...
Where things are now would be a decent base system to remaster as a root file system, but I am wondering how that will work since the total size of numbered upper_changes is in the 3GB territory, that might make root file system too large to load in ram.
...

You use w_copy2ram=0 at the boot commandline ? (that will copy the .sfs's to RAM)

Btw, I never understood the real advantage of copy2ram, only thing it can be useful for IMHO is that you possibly can unmount the disk (or unplug USB) where the frugal install is located, it doesn't speedup the system AFAIK, but anyone correct me if I'm wrong.

Can't calculate exactly, but if you use XZ compression, it will compress around 40% (or less), so let's say that your 3GB uncompressed will become around 1.2GB sfs.
Plus the existing sfs modules in KLV (600MB), total will become around 1.8GB (not that much, I'd say).
Depends on the amount of RAM you have of course, if copy2ram will work.

[wiak]

...
Where things are now would be a decent base system to remaster as a root file system, but I am wondering how that will work since the total size of numbered upper_changes is in the 3GB territory, that might make root file system too large to load in ram.
...

You use w_copy2ram=0 at the boot commandline ? (that will copy the .sfs's to RAM)

Btw, I never understood the real advantage of copy2ram

Actually, you don't need the =0 so can just use: w_copy2ram on its own. The mere fact that w_copy2ram is given as argument on grub kernel line causes all the layer components to be first copied into RAM (a simple grep in initrd just looks for word w_copy2ram in /proc/cmdline). If w_copy2ram not there then default is not to copy these layer sfs addons and so on to RAM. But I never use w_copy2ram - I agree that for most use there is no point to copying into RAM - just uses up more RAM! The argument used to be that it was faster accessing mounted sfs filesystems if they were first stored in RAM (which slows down boot of course) rather than left on slow hard drives or slower usb sticks. Maybe the latter can still be true but modern hard drives are fast, and older systems used to be RAM-challenged so wasn't such a great idea to copy into RAM anyway. Even running from usb 2.0 sticks seems to work fine without w_copy2ram, so why bother with it? - and yes, when lots of rollback files they would indeed occupy tons of RAM unnecessarily if using w_copy2ram argument.

[geo_c]

...
Where things are now would be a decent base system to remaster as a root file system, but I am wondering how that will work since the total size of numbered upper_changes is in the 3GB territory, that might make root file system too large to load in ram.
...

You use w_copy2ram=0 at the boot commandline ? (that will copy the .sfs's to RAM)

Btw, I never understood the real advantage of copy2ram

Actually, you don't need the =0 so can just use: w_copy2ram on its own.

Actually, I don't use the copy2ram parameter, but I wasn't sure if the main rootfs was copied to ram by default. It looks like from what you're both saying it's just mounted and layered.

[wiak]

@rockedge:
I think either you or I have somewhere posted how to make a pseudo-full-install. I think it would be good to make a sticky of that method somewhere since it provides a kind of other mechanism to a remaster. Not only does it allow new stuff to be added to the actual root filesystem (and not as an addon layer) but provides a great yet simple facility we haven't yet talked about:

Slimming down the system by removing unwanted apps and whatever...

Unlike most other frugal installed distros no special additional utilities are required to remove any base root filesystem apps or other components. If you are booting into a pseudo-full-install arrangement, simply using the package manager to remove an app actually removes it and doesn't simply 'white it out'. Similarly with anything else you remove; it is actually removed! That's because in pseudo-full-install the base root filesystem is placed at the top read-write layer in uncompressed form.

Great thing about this method is that it provides a new approach to building a system to a user's own liking. Traditionally, users wanting to build up their own system asked for some 'core' mini-build to which they could then add components of their choosing. Problem with that approach is that it is difficult to build a distro from basics because many components can be involved and need configured and the overall work can be difficult technically to determine. New approach would be:

Start with nicely built, polished and working distro (all the difficult component parts and dependencies all working and configured) and then simply remove as many applications as user wants (assuming not dependencies of anything else) and add alternatives or any extras they want. That new approach makes it much much easier for a user to craft their own variant be that smaller or larger in resulting size.

Of course, more experienced FirstRib builders can build from scratch via the build script f_plug file (or craft their own), but a pseudo-full-install provides an alternative quite simple and direct method (aside from need to use unsquashfs and, optionally, mksquashfs at the end of the process); a remaster, such as covered in this thread, is yet another method of course.

[Clarity]

...
I think either you or I have somewhere posted how to make a pseudo-full-install. I think it would be good to make a sticky of that method somewhere since it provides a kind of other mechanism to a remaster. ...

Good idea, time permitting. ... Maybe sticky here.

[fredx181]

Trick to remaster with excluded kernel modules and firmware (that I earlier wrote about, and thanks @dancytron ) : https://forum.puppylinux.com/viewtopic. ... 929#p77929
So that way you can keep the existing "00modules-.....sfs" and "01firmware-....-KLV.sfs" in the frugal install, just replace the 07KLV-airedale_rootfs.sfs with the resulting remastered .sfs .

[dancytron]

After there is a more finalish release, I'll make a Chromium remaster mirroring how we do it in Debian Dog and list out all the steps.

[wiak]

Sorry, I've never checked/looked-into any of this remaster work as yet. I can't keep up with the speed time seems to be travelling.

[dancytron]

That's okay.

Fred's script gives us all we need. It just needs a set of instructions people can follow and I can do that.

[fredx181]

Sorry, I've never checked/looked-into any of this remaster work as yet. I can't keep up with the speed time seems to be travelling.

That may be because you are (too much ) obsessed by your own development works.

[wiak]

Sorry, I've never checked/looked-into any of this remaster work as yet. I can't keep up with the speed time seems to be travelling.

That may be because you are (too much ) obsessed by your own development works.

I beg your pardon?
EDIT: Currently I am working flat out from morning till way after I should go to bed trying to improve/finaliise the several distros I have recently released whilst at same time trying to fix up issues in FR initrd code including, but not only for Ventoy - perhaps that is why I haven't found time to also get round to testing others utility additions. I won't be finding that time now.

[fredx181]

Sorry, I've never checked/looked-into any of this remaster work as yet. I can't keep up with the speed time seems to be travelling.

That may be because you are (too much ) obsessed by your own development works.

I beg your pardon?

Ok, was just a hint, to encourage you to try to be more open for additions by other people, feel free to ignore
EDIT: Also, to be honest, all the talking about Ventoy, SG2D, ISO booting etc... has not my interest, but that's just me ...

[wiak]

That may be because you are (too much ) obsessed by your own development works.

I beg your pardon?

Ok, was just a hint, to encourage you to try to be more open for additions by other people, feel free to ignore

I don't ever ignore comments of that type. A lot of piss actually

[fredx181]

I beg your pardon?

Ok, was just a hint, to encourage you to try to be more open for additions by other people, feel free to ignore

I don't ever ignore comments of that type. A lot of piss actually

Sorry, no intent to insult you, not sure what you mean btw with "A lot of piss actually".

[wiak]

Sorry, I've never checked/looked-into any of this remaster work as yet. I can't keep up with the speed time seems to be travelling.

That may be because you are (too much ) obsessed by your own development works.

Exactly what else is your comment supposed to be if not insulting?

[wiak]

not sure what you mean btw with "A lot of piss actually".

I don't know about other English speaking countries, but the Scottish often use the word 'piss' when others might use the word 'shit'

[fredx181]

Sorry, I've never checked/looked-into any of this remaster work as yet. I can't keep up with the speed time seems to be travelling.

That may be because you are (too much ) obsessed by your own development works.

Exactly what else is your comment supposed to be if not insulting?

That you are not giving much feedback on what other people contribute IMO.
But, I cannot look in your mind, if you want to give priority to things like booting with Ventoy etc... that's your choice of course.

[wiak]

That may be because you are (too much ) obsessed by your own development works.

Exactly what else is your comment supposed to be if not insulting?

That you are not giving much feedback on what other people contribute IMO.
But, I cannot look in your mind, if you want to give priority to things like booting with Ventoy etc... that's your choice of course.

It is none of your business what I give priority to, and your accusation aside from being unwarranted is indeed a lot of crap.