Overlayfs experiment - redundant

[ozsouth]

** EDIT 28/11/22 - Now that S15Pup64 can run a savefile with overlayfs, this project is redundant. **

I've been thinking about overlayfs for the last few months. I used Gyro's 2019 initrd.gz from the old forum as a test today with ScPup64 & my 5.19.5 kernel, for which I enabled overlayfs (as =y). I have a working desktop but without save capability. As I use a ydrv packed with my personal settings, that is not a huge loss.
Will experiment more over the next few months. I'm doing this as I'm concerned about the long-term future of aufs, so am building my understanding of overlayfs.

[rockedge]

@ozsouth did you see? https://forum.puppylinux.com/viewtopic.php?t=6753

might be something you can check out. This is built also from gyro's work on overlayfs and Puppy Linux.

This version has a basic save file system using tar.gz compression I think

[mikeslr]

I've been thinking about overlayfs for the last few months. I used Gyro's 2019 initrd.gz from the old forum as a test today with ScPup64 & my 5.19.5 kernel, for which I enabled overlayfs (as =y). I have a working desktop but without save capability. As I use a ydrv packed with my personal settings, that is not a huge loss.
Will experiment more over the next few months. I'm doing this as I'm concerned about the long-term future of aufs, so am building my understanding of overlayfs.

Thanks for experimenting. We all --even Devs-- suffer from a 'Human Failing'. We are subjective about the value of 'our things'. We value of the things we invest time, energy, and effort in in direct proportion to that time, energy and effort rather than their actual utilitarian value. So its good to have a 'disinterested' party with technical expertise view --what is to us-- the Emperor's new clothes.

While not exactly new to the developers of the Linux Kernel who haven't considered aufs important, overlays is relatively new to Puppys which has. I've emphasized one of your sentences because of my extensive use of the Save2SFS module in the nicOS-Utilities-Suite, https://www.forum.puppylinux.com/viewtopic.php?t=1694 to substitute an adrv.sfs or ydrv.sfs for a SaveFile/Folder. That utility can also 'update' those files-systems to include new applications, and changes to setting and customizations. But I'm hazy about whether it can do that when those new applications, settings and customizations are provided by an overlay.

Along the same lines, I wonder if the use of overlays will hamper a User's ability to Remaster. Perhaps with many gigabytes of RAM and terabytes of Storage Remastering may no longer be important. Still Remastering might be likened to 'Spring Cleaning'.

Newly OOTB from an ISO, a Puppy will have built-in infra-structure-frameworks, utilities and other applications. Overtime the User may update those. Those updates will be preserved in a SaveFile/Folder which --in the merge-file-system-- has priority over the files built-into the original ISO. At that point, the original infra-structure-frameworks, utilities and other applications are just wasting storage space. [I'm not certain, but also perhaps a little bit of RAM]. Remastering 'cleans house', discarding them. [I haven't mentioned built-in unwanted-by-the User applications: remastering can also remove of those].

Are Puppys currently available tools for Remastering adequate to manage a 'merge-filesystem' only employing overlays?

[rockedge]

@mikeslr Good question. I think I could run the Bionic32-OL and try and see what happens with a remaster attempt.

I would think it all depends on the merge mechanism to work, to have that remaster clean the system effect. Have to see the differences on how the layers are handled by overlayfs and AUFS

[ozsouth]

@rockedge - I hadn't read that post - I will have a closer look at the save mechanism as the one in Gyro's original didn't work for me.
@mikeslr - I haven't thought as far as remastering - I usually make a save folder/file and make a ydrv based on that. Nic's utilities look interesting.

[ozsouth]

I've further developed my overlayfs project & have made a 5.10.141 64bit kernel which has no aufs. I hacked away (commented out massively) all aufs settings in kernel-kit master 2018. Still get a warning during kernel compile that I don't have aufs available, but that's a good reminder. I made versions of fossapup64 & ScPup64 that can use it. Savefile/folder is still not available (just didn't work for me), but I can load .pets, & by adding .sfs files before bootup, can add other apps such as different browsers, office, devx, sources etc. They are simply renamed to one of the (letter)drv in the pup folder & should run upon reboot, provided you have dependencies sorted & no critical files overwritten, but normal pups require that too. Once system booted, sfs files can only be viewed (not loaded) by clicking on them in ROX-Filer. Initially, I have made 4 extra .sfs loadable in fossapup64 (b,c,d,e drv), and 8 in ScPup64 (g -> n drv). By adding settings to a special ydrv (via my updatesfs), I can customise to a fair degree. It needs a special vmlinuz & initrd.gz & zdrv, and a ydrv with very basic settings (including updatesfs), along with standard puppy & fdrv (& adrv optional) files. DISTRO_SPECS needs updating in initrd.gz & ydrv for each different puppy. Also needs a specific kernel/linux line in grub/syslinux cfg. Is quite usable for me. As it can tangle up your system if done wrongly, I won't post the generic version publicly now.

EDIT 21-09-22: I have been exploring overlay layers - ABOVE, MAIN & EXTRA. 'Main' are puppy drv, fdrv & zdrv. 'Above' are adrv & ydrv for browsers & settings (I've also added xdrv & wdrv as spares). They can overwrite some files, i.e. /usr/local/bin/default... scripts. 'Extra' (other apps) are those where the contents do not overwrite previous settings from other drvs. I reverted to just b,c,d,e drvs (more drvs would use more ram). Still no save session options. The 64bit 5.10.141 kernel I made is for either Ubuntu or Slacko derivatives. On UEFI, I can boot via grub, but prefer syslinux (which allows more boot parameters for my setup). I don't have any old (non-uefi) systems to try. As @dimkr has overlay options with sfs-load, I'll leave this as workable option, 'just in case'. (I did also make an overlayfs usable version of @Grey 's jammypup64-9.7b).

[ozsouth]

I've been experimenting more & learning more about layers, & released 3 hybrid aufs/overlayfs kernels. I now allow 16 possible sfs drvs - 3 Main, 6 Above, 7 Extras. I've also made suitable drv.sfs app files - FirefoxESR, LibreOffice, Masterpdf4. I included my updatesfs script to adjust sfs files (i.e. add settings to ydrv). I've now made 'conversion kits' for Bionicpup64-8.0, Jammypup64-9.7b & ScPup64-20.06+2. Those consist of initrd.gz, vmlinuz, zdrv & ydrv - all specific replacements for the Puppy involved. The grub.cfg kernel line must include, for example, pdrv=sda6 psubdir=pup (Syslinux can use a drive label).
Savefile/folder still not a goer - won't reload pup_rw layer data from an sfs, but I'm toying with saving just the home folder & maybe root folder & a docs folder, from the pup_rw layer. Still too experimental to release to the general public - tinkerers could break it.

EDIT 30-9-22: For the final test for this month, I made a version of ScPup64 which offers to back up folders /home & /root (minus .XLOADED) upon shutdown. I've reset FirefoxESR, LibreOffice & MasterPDF4 to their defaults & can save any settings or documents (I set /root/Documents). Adding a printer/scanner required some files to be added to the ydrv, but otherwise settings are reloaded upon bootup.

[ozsouth]

Think I finally succeded in making full persistence on my overlayfs system. I can now save via menu and upon exit. I have Jammypup64 & ScPup64 working versions. Will probably be impractical for slow, usb or low-ram systems due to significant disk writing needs. Once booted, the /initrd/mnt/tmpfs/pup_rw folder contains only changed files. By mounting the root drive & processing the session save there, only /mnt subfolder keeps changing. After making a backup of the persistence .sfs on the root drive, I unsquash it there, then copy only changed files which are updates, (I ignore /boot /dev /initrd /mnt /proc /run /sys /system /tmp folders) into the unsquashed folder (cp -r -u), then remove files no longer on main system from persistence file, then resquash it & rename to the persistence .sfs. Rather like pupmode 13. Upon reboot, my installed & created stuff is there. Some specific .sfs, initrd.gz & kernel are needed for each Puppy version. EDIT - cups issue with Jammypup64-9.7b - needed brlaser3 .deb, with similar driver to my printer .

[ozsouth]

I have a working version of ScPup64-21.04 with updated curl, openssl, wget (& with peebee's permission to release it).
I have a working version of Jammypup64-9.7b (& with grey's permission to release it).
I have a working version of Bionicpup64-8.0 with updated curl, openssl, wget (without permission to release it).
Classed as experimental, so will only release to devs/experienced users selectively, at own risk.

[je55eah]

Impressive. In what ways does your new system differ from the FirstRib system? I am getting the impression that your approach is more like Puppy linux with overlayfs instead of aufs and wiak's approach is divorced from everything puppy except for the ability to create frugal installations. If that is accurate, do you anticipate that all future puppies will migrate to this method?

[ozsouth]

@je55eah First Rib is an advanced, ground-breaking project. There are also other projects like Debian kernel conversion happening. Those could well be the future of Puppy - the community will decide. All I wanted to make was an overlayfs conversion kit for standard pups - a usable alternative to aufs-based pups. By using a specific initrd & saving then reloading changes via an edrv, I've largely achieved my original aim. Some minor shortcomings will diminish the appeal. It is not really for usb or low-spec systems (saving would take ages).
It is possible to change the kernel to 5.10.144-64oz-ao from my kits, & afterwards alternate between normal puppy & overlayfs version by setting the appropriate of the 2 initrd.gz files - as I use puppy, zdrv, fdrv, and then non-standard drv letters. Even the grub entry is standard (but must contain, for example, pmedia=atahd pdrv=sda5 psubdir=oje ).

[je55eah]

Thanks for the synopsis. Does it write a lot on all systems or only low ram systems which must write to swap? It wasn't clear to me why saving changes is slower with your overlayfs implementation. That's not your fault. It's because I don't know as much about the topic. It sounds like you are dissuading adoption of your solution, but I doubt you would have done the work if there was nothing to gain from it.

Since you are an expert on the init system in Puppy, what parts of init provide unique values which are not present in the FirstRib runit implementation?

[ozsouth]

@je55eah - I'm no expert on init - I've derived others work (my initrd came largely from user gyrog). I wouldn't use my method on a system with under 2gb ram (4 is better), as a lot is loaded into ram on boot, and swap would be too slow. As for saving, my save procedure expands the whole edrv, updates it (adds & removes), then repacks it. On usb or low-spec systems, that could take a few minutes. I used a 6 year old intel i3 with ssd & my average was 20 sec. The beauty of this method is that you can make a system to suit yourself, then never save again - or make special copies of the edrv to reinstate whenever you wish (for example, you may have a setup for video editing that you don't want altered). Once set up, that COULD be run on usb without undue load/save issues. I've given the original devs (of ScPup64 &jammypup64) copies to try. I'm cautious about new concepts & would rather experienced folk test it before a general release.

[je55eah]

@ozsouth
That makes a lot of sense. I have begun to believe that Puppy really shines when it is set up as an unchanging live system, and it might really come into its own as a system administrators tool similar to the sparkyLinux rescue edition or SystemRescueCD. That is how I have been using it for years anyway. If I learn enough there may be a GodModePuppy in our future. Thank you for doing this.