Comparing aufs vs overlay

[williwaw]

I understand they both intend to meet the same needs, and overlay is in the kernel vs. aufs not...

1. From the users point of view, what are the pros and cons?

and of course, if there are implementation issues that devs are more concerned with, I would like hear those concerns also.

thanks

[puddlemoon]

I have a user's perspective..
I use a realtime or low latency kernel for audio production so I have been working with DebianDogs that are non aufs so I can install kernels built for debian with no modification. But they are not interchangeable as they use a different enough method of layering out the system, so to use puppy (which I love dearly) we need aufs patched kernels.

So I for one am currently grateful for both.

[dancytron]

I understand they both intend to meet the same needs, and overlay is in the kernel vs. aufs not...

1. From the users point of view, what are the pros and cons?

and of course, if there are implementation issues that devs are more concerned with, I would like hear those concerns also.

thanks

Overlay won't add and remove layers after boot like aufs will, so you wouldn't be able to load and unload sfs files on the fly.

Fred has a work around using symlinks instead for Debian Dog, but it is literally a few weeks old, so hard to say whether it would work on Puppy or if there are as of yet unknown issues.

[s243a]

I understand they both intend to meet the same needs, and overlay is in the kernel vs. aufs not...

1. From the users point of view, what are the pros and cons?

and of course, if there are implementation issues that devs are more concerned with, I would like hear those concerns also.

thanks

Overlay won't add and remove layers after boot like aufs will, so you wouldn't be able to load and unload sfs files on the fly.

Fred has a work around using symlinks instead for Debian Dog, but it is literally a few weeks old, so hard to say whether it would work on Puppy or if there are as of yet unknown issues.

I think that in most cases symlinks would work. However, I can think of some potential issues:

- Calls to realpath, could take a script to a fragmented path where things aren't in the same relative location as expects.
- If a compiler links libs relatively (instead of searching for them in the usual places) then a lib may fail to load its dependencies. I noticed that this was an issue with java (package manager version) on upupGG+D. Creating a symlink for the java executable into one of the folders in $PATH led to java not being able to find some of the dependencies. This can however be patched by editing the elf file.

[user1111]

The dynamics of aufs can also introduce problems/risk, where the changing of layers and whiteout files for instance don't always work out correctly (I periodically see such instances even as a single seat desktop user). As such Linus was not in favour of including it within the core/kernel by default. Instead I guess figuring that you could always setup and use chroot's (or other means) to similar effect.

I use Fatdog, which is already fat with goodies that I use, I don't tend to load sfs's dynamically anymore, and can just reboot to do so whenever the need arises (such as infrequent loading of the devx sfs). Others load/unload sfs's much more frequently, and support of that involves a non-standard (aufs patched) kernel.

I also tend to boot, use, shutdown without any changes being saved i.e. a clean configured setup. For me being able to gslapt (package manager) things into that is comparable to the temporary loading of a sfs. I also have tar files that I sometimes extract to the root / folder, again which in not saving changes means that they were only loaded for the session. I haven't really come across a usage case of being able to unload things being beneficial/useful, as a simple reboot is relatively quick and unloads things back to the 'clean' state.

[s243a]

I haven't really come across a usage case of being able to unload things being beneficial/useful, as a simple reboot is relatively quick and unloads things back to the 'clean' state.

I suppose it depends on how frequently that you change things. If your system is quite static then sure overlayfs is likely sufficient.

[user1111]

Yes I tend to reboot regularly, multiple times/day, at least for my desktop system, such that the need to be able to unload sfs's isn't a necessity. Others have their systems up and running for weeks/months (I also do for 'servers') where the ability to unload sfs's could be useful (but where doing so could induce instability). Systems strive to accommodate a broad range of usage purposes, so having both choices is nice/useful.

[s243a]

Yes I tend to reboot regularly, multiple times/day, at least for my desktop system, such that the need to be able to unload sfs's isn't a necessity. Others have their systems up and running for weeks/months (I also do for 'servers') where the ability to unload sfs's could be useful (but where doing so could induce instability). Systems strive to accommodate a broad range of usage purposes, so having both choices is nice/useful.

rizalmart, suggested mounting less essential packages in opt could help improve puppies modularity.

Improving puppy modularity #1899

I originally, didn't think that this was necessary, until I realized that symlinks can block folders and Vice versa. The fewer files which are located in the folders specified by $PATH and $LD_LIBRARY_PATH, the lower the likelihood that one of these files will be blocked.

[step]

Similarly to @rufwoof, I rarely save to my save folder, and always start the system with a clean slate. I only reboot once a day. To me the real cost of rebooting is getting back to my previous, carefully-constructed work state; the speed/time of booting is irrelevant. I load and unload SFS files very frequently. A use case for unloading is application versioning, when one can't always use the latest/greatest version but needs to use different versions of a package for different projects. My R/RStudio SFS package series comes to mind. A use case for loading is when an application package is only available as an SFS because the package developers didn't release it in any other form. I'm one such developer--some of my packages are only available as SFS files. As @rufwoof said, having a choice of both aufs and overlayfs is good, especially since they are not the same and Fatdog/Puppy are heavily invested in aufs. So far I haven't seen a use case where overlayfs is needed because aufs can't do the task. I'm under the impression that a temporary shortage of updated aufs patches has led our community to worry and begin experimenting with alternatives. Overlayfs allows greater kernel version independence. That is good for people who need special kernels, can't build their own, so they get them more easily outside of Puppy. I don't have that need and often worry about upgrading the kernel. I have never seen an upgrade that didn't bring about some issues and regressions.

[dimkr]

(After seeing a link to this topic in Phoronix)

I'm under the impression that a temporary shortage of updated aufs patches has led our community to worry and begin experimenting with alternatives.

Indeed, recent issues with aufs (slower addition of branches for recent kernels, stability issues) make overlayfs a more attractive than it used to be, for Puppy's use cases.

woof-CE has a PR that adds optional (off by default) support for overlay on /, and it's included+enabled in Vanilla Upup 22.04:

https://github.com/puppylinux-woof-CE/woof-CE/pull/2963
https://github.com/puppylinux-woof-CE/w ... ssues/2962

[Duprate]

Hello! I only use overlayfs, on FatDog64, Vanilla Dpup64, Voidpuppy64, BionicPuppy32 and any system I still use. The AUFS, I buried it more than a year ago. Everything I might need to load after starting the system, I converted it to ROXapps. Some create symlinks, others run locally, others are installed on the system. It is also possible to load an SFS using scripts used in DebianDog from "Fredx181". I don't use savefiles. So when I turn off the PC, everything is as it should be. I didn't cry a tear at the AUFS wake!

The options are there, for anyone who wants to get rid of AUFS! I used them all...

[user1111]

Hello! I only use overlayfs, on FatDog64, Vanilla Dpup64, Voidpuppy64, BionicPuppy32 and any system I still use. The AUFS, I buried it more than a year ago. Everything I might need to load after starting the system, I converted it to ROXapps. Some create symlinks, others run locally, others are installed on the system. It is also possible to load an SFS using scripts used in DebianDog from "Fredx181". I don't use savefiles. So when I turn off the PC, everything is as it should be. I didn't cry a tear at the AUFS wake!

The options are there, for anyone who wants to get rid of AUFS! I used them all...

@Duprate how to you setup/use Fatdog with overlayfs?

[Duprate]

To use FatDog64 with Overlayfs boot, just follow the steps below:

1- Use your existing installation or install FatDog64 from the .ISO file and make the
its initial configuration, customizing it according to your needs;
2- Save your settings on HD;

3- Extract fd64.sfs and kernel-modules.sfs from the initrd file;
4- Rename kernel-modules.sfs as 00-modules.sfs;
5- Rename fd64.sfs as 01-FatDog64.sfs;
6- With the "savefile", create the 02-changes.sfs;

7- Use the "initrd" created by "Wiak", serving perfectly for that, the one included in the
wdlgo_focal64_3.0.0-rc1-allfirmware.iso, not requiring a more modern version;

https://www.weedolinux.rockedge.org/vie ... 9110e1&i=1

8- It is good to add the file "ucode.cpio" that can be obtained in several ways and is
included in versions of "Vanilla Dpup";

In grub, it looks like this:

menuentry "FatDog64-812 Kernel 5.18.2-fd64 OVERLAYFS" {
linux /FatDog64/vmlinuz w_bootfrom=UUID=xxxxx-xxx-xxx-xxx-xxxxxxx=/FatDog64 w_changes=RAM w_copy2ram
initrd /FatDog64/ucode.cpio /FatDog64/initrd
}

If you use "menu.lst" produced by grub4dosconfig-v1.9.2:

title FatDog64-812 Kernel 5.17.3-fd64 OVERLAYFS
uuid xxxxx-xxx-xxx-xxx-xxxxxxx
kernel /FatDog64/vmlinuz w_bootfrom=UUID=xxxxx-xxx-xxx-xxx-xxxxxxx=/FatDog64 w_changes=RAM w_copy2ram
initrd /FatDog64/ucode.cpio /FatDog64/initrd

Good luck!

[mikeslr]

Correct me if I'm wrong. But as I see it the 'Changes Folder' when used in an Overlay system is the functional (perhaps literal) analog of a SaveFolder when used in an AUFS system: settings, customizations, new applications and the updated versions of build-in applications are written to it and on boot-up its contents will have priority in a 'merged in RAM' file system.

Especially with regard to application updates, this results in 'wasted space' in storage and unnecessary complexity in the system-as-a-whole. Under an AUFS system, there are applications to Remaster, eliminating the older application versions, including added applications into the base/core.sfs and making it easier to transfer the remastered OS to another computer or publish it.

Has anyone Remastered a Puppy employing Overlays? What remastering application was used? Were there any problems? hurdles to be aware of?

Under an AUFS Puppy, in addition to compiling, it has often been useful, sometime necessary, to rebuild applications designed for or ported from other LInuxes. That rebuilding has taken various forms such as pets, SFSes, Rox-Apps, Portables and Portables employing an AppImage via a wrapper. There have been posts about converting such applications to Rox-Apps for use in an Overlay system as it lacks the ability to SFS-load-on the fly during a session. I think portables will work as well. And --as I understand it-- under an Overlay system application SFSes can be renamed with a number prefix so that on boot-up it will be 'merged in ram'.

But do any of the tools available to AUFS Puppys for creating and rebuilding applications work under an Overlay system? Are their other tools? And especially, is or something like nicOS-Utility-Suite's Save2SFS available. Under an AUFS Puppy it can create either an adrv.sfs or a ydrv.sfs which will automatically become part of the 'merge in RAM' file-system. If usable under an Overlay Puppy, the a/ydrv.sfs would have to be renamed to provide it with a numeric prefix. But that would extend its utility as an OVerlay system can employ up to 100 SFSes, IIRC.

[mikeslr]

@ Duprate, or anyone working with Overlays.

Before posting the above questions, I had intended to explore them on my own by following the recipe Duprate provided here, viewtopic.php?p=73485#p73485. Unfortunately, I think that recipe requires the initrd created by Wiak and previously available by downloading wdlgo_focal64_3.0.0-rc1-allfirmware.iso. That ISO is no longer available, https://firstrib.rockedge.org/viewtopic.php?p=332&i=2

Any suggestions for an alternative? I do have a weedog'd manjaro on my hard-drive.

Initrds in WeeDog'd Manjaro Folder.png (53.54 KiB) Viewed 2813 times

[rockedge]

I might have the initrd.gz for it. I will check later when I am able to access my dev machines

[wiak]

@ Duprate, or anyone working with Overlays.
...
the initrd created by Wiak
...
Any suggestions for an alternative?

KLV-airedale uses the most recent updated version of that initrd which configures overlayfs to provide all its functionality. There is no advantage in using the older version; it is not simpler to use - it is the same except it does not provide some of the extra facilities the newer one incorporates. Overall, the code stucture is identical - it was designed to be extensible in a logical (not hacked) manner, and that is all that has happened over time, though nothing really in my mind to add to it now (who knows...) since now has all I envisioned for it.

What you described in your post was in fact the coded-in features of that particular initrd, indeed of any and all versions of it. For example the up to 100 addon layers - that is not a feature per se of overlayfs - could also be done with aufs - it is the 'wiak' initrd code logic that chose to use numbered add-ons up to 100. It also provides the equivalent of Pupmode 13 (save2flash) as well as direct write to the save folder or save file if you want that alternative (savefile can be used on non-Linux formatted partitions like ntfs if you like). The only major omission to an aufs-based layered system is indeed sfs load on the fly (though KLV provides that anyway - just by a different method than via aufs or overlay). You can of course load whatever sfs addons you wish at boot time - with 'wiak' initrd that just needs you to put a 2-digit numeric in front of the sfs filename (which is used to say what layer you want it at, so you can easily arrange layers or you can disable an addon by putting non-numeric character at start of filename if you wish).

There is indeed the possibility to remaster (fredx181 wrote a utility script) - which is pretty much same whether using aufs or overlayfs - you just write back the whole filesystem to external media and create new main sfs out of that excluding whatvever system cache and so on not wanted or needed. You can also merge the addon sfs files together if utility is written to do so, or indeed just merge all the ones currently in RAM (since everything is already merged when using any layered system - just a matter of writing it back as one file to external media). Rockedge is experimenting with various merge scripts and the remaster utility with KLV-airedale.

Finally, even the sfs-load-on-the-fly is achieved in KLV-airedale, but by a different method than using the overlay - rather it is done the way tinycorelinux loads on the fly its various addons - a process of symlinking that wanderer has advocated for years. So no functionality is lost, and in fact KLV-airedale because of utilising that 'wiak' initrd actually offers extra facilities to a typical puppy distro - particularly the fact that the initrd has been written to allow uncompressed normal directories and/or sfs files to be used as the addon layers.

firstribit (was weedogit) is just a variant of using firstrib component parts (basically an extra firstrib build system script), so any such distro using 'wiak' initrd inherits the same possible functionality, though the utilities need copied in to effect them. Next firstribit, for example, will likely provide save2flash (like Pupmode13) functionality to these firstribbed frugal installed upstream/mainstream distros out-of-the-box. Only probably a few lines of code alteration would be needed to change that initrd to use aufs instead of overlayfs, but I consider that an unnecessary backwards step and prefer ability to use upstream official tested kernels that come with overlayfs included and don't need aufs patching.

EDITED cos too long/complex

[wiak]

@rockedge I think I should maybe make a very small commandline only Void build kit containing the build scripts, very small build plugin and a tar.gz of the save2flash utility to allow people to explore the initrd and the overlayfs functionality it provides via that simple/expandable system. Would obviously still need to provide firmware/modules and kernel to boot it of course, so actually doesn't end up all that small once built, but at least shows the simple build process.

Only three scripts are required nowadays, along with a suitable firmware sfs:

1. The main root filesystem build script: build_firstrib_rootfs.sh (that and a user-created plugin, which is simply a list of all the build config commands the builder wants - of course that becomes very complex for a complete system like KLV-Airedale, but can be provided as a simple few lines of code for a simple commandline system).

2. The fetch and configure initrd script: FRmake_initrd.sh

3. A script to make the modules addon layer component: FRextract_kernel.sh

Then you just boot it using your pre-installed grub (oh, the utility script wd_configgrub even provides the exact grub.cfg stanza to use for successful boot)

So a simple build kit just for experimentors would be easy to put together - just a tar archive of these tiny few components.

Of course, if you end up wanting to produce isos, an extra script would be needed to run the appropriate xorriso command or similar (and include the numerous EFI-related components needed for iso-booting); but no need to build isos - frugal install built from these simple scripts is perfectly fine enough IMO.

On second thoughts, I can't be bothered right now - too many other things on my rapidly growing to-do-list, but would be good future kit for those interested in developing or simply to better understand how the likes of KLV-airedale is put together in the first place. KLV-airedale is a special case of course since it incorporates anything and everything useful from the Puppy/Dog/forum world - particularly the useful utility scripts we all become spoilt with. Other KL distros could start with a Puppy aufs initrd of course, though without a lot of modification that would require some form of Puppy root filesystem underneath since these are closely tied together.

[geo_c]

On second thoughts, I can't be bothered right now - too many other things on my rapidly growing to-do-list, but would be good future kit for those interested in developing or simply to better understand how the likes of KLV-airedale is put together in the first place. KLV-airedale is a special case of course since it incorporates anything and everything useful from the Puppy/Dog/forum world - particularly the useful utility scripts we all become spoilt with. Other KL distros could start with a Puppy aufs initrd of course, though without a lot of modification that would require some form of Puppy root filesystem underneath since these are closely tied together.

Hmmm. I think I would be interested in that. If I get what you're saying, I could use the script to build a command line psuedo-full install Void Linux, then if I got pretty good with command line xbps, build my own desktop version from the ground up, or at least experiment with the idea to see how it's done.

[rockedge]

@geo_c Yes that's it in a nut shell.

I have PLUG files that can build a really basic KLV command line only, or a bit more advanced with Xorg and JWM, or right up to pretty sofisticated KLV-Airedale base. Build one or two and get a rootfs you can chroot into or boot on bare metal ot a virtual machine!

PLUG file design can be simple simple or super complex and all in between.

I'll see what I can find.

some links to lots of information:
https://oldforum.puppylinux.com/viewtop ... 6#p1028956
viewforum.php?f=130

[ozsouth]

@mikeslr - I am now only using overlayfs with S15Pup64-22.12. The initrd.gz that woof-ce has produced (since july this year at least), can run either overlayfs or aufs, by specifying the punionfs=overlay (or punionfs=aufs) parameter in the kernel line of grub.cfg or syslinux.cfg, (but probably not grub4dos), when using an overlayfs (& aufs) capable kernel. I did make an iso earlier that would boot either from the menu, but only installs aufs directly.
Also since about july, savefile & savefolder work under overlayfs, but are different (containing upper & work folders), & can't be interchanged with aufs-created ones.
Main shortcoming I have is that I had to modify sfs loading, as mounting after boot did not merge with the system & required manual unmounting. Choosing to view works as expected - I made a .pet to modify filemnt & sfs-load to allow only viewing (attached below - use at own risk).
I also made a modified initrd.gz to allow b,c,d,e drv for apps. These must be loaded at boot time. However, they should NOT be used to load devx or sources upon reboot on a system with a savefile/savefolder, to avoid possible corruption. For compiling, a separate installation is recommended.

[rockedge]

@ozsouth might be able to import and use the same mechanisms to perform SFS-Load on the fly in the overlayfs mode that DebianDog and KLV-Airedale are using.

Also take the filemnt from KLV-Airedale, it can mount ISO's and SFS files also in overlayfs mode. filemnt is in 10gtkdialogGTK3_filemnt64.sfs

[ozsouth]

@rockedge - very interesting - with sfs-load on-the-fly available too, I wonder why would anyone persist with aufs? Puppy has come a long way forward this year.

[mikeslr]

Thanks, ozsouth and wiak for your responses. From wiak's responses I have the feeling that the quickest way for me to internalize how overlays function and how it differs from Aufs is to download and run and explore KLV-airedale. That was on my 'ToDo List' anyway. I'll just move it to the top.

Once I have a handle on that I see what I can do with using S15Pup64-22.12 with Overlays. As I already have S15Pup64-22.12 set up using Aufs, using it with overlays should provide both a good comparison and transition.

[dimkr]

Once merged, https://github.com/puppylinux-woof-CE/woof-CE/pull/3723 should close the main gap that prevents "mainstream" Puppy from using overlay. It emulates dynamic SFS loading using linker search path modifications and symlinks, only under PUPMODE 5 and 13. If you want to try it out, this feature is included in the Vanilla Dpup 9.3.0 beta and 10.0.x development builds, which don't have aufs at all (download links at https://vanilla-dpup.github.io/).

[peebee]

FWIW......... probably won't affect many people............

using aufs, a 64-bit kernel can be used with a 32-bit system build
but
a 64-bit overlayfs kernel cannot be similarly used (crashes out just before getting to desktop)

This means that 64-bit-compatibility sfs for 32-bit systems are not feasible when booting with an overlayfs kernel.

[dimkr]

@peebee overlay doesn't break support for 32 bit userspace, this is probably conicidence (a broken kernel, a broken SFS, ...) and nothing more. I run old games and other 32 bit applications on a 64 bit kernel (even tried pure 32 bit userspace at some point) and everything works great with overlay.

[peebee]

@peebee overlay doesn't break support for 32 bit userspace, this is probably conicidence (a broken kernel, a broken SFS, ...) and nothing more. I run old games and other 32 bit applications on a 64 bit kernel (even tried pure 32 bit userspace at some point) and everything works great with overlay.

I was trying to use:

kernel-kit-output-usrmerge-debian-bookworm-x86_64
from: https://github.com/puppylinux-woof-CE/w ... kernel.yml

in:
BookwormPup32-23.12-231221.iso
https://sourceforge.net/projects/pb-gh- ... 2_release/

[dimkr]

@peebee Did a fresh 32 bit build with a 64 bit kernel and it boots fine,

X doesn't start but this is common in 32 bit builds on this 2015 laptop. And everything is super sluggish (boot time is horrible), probably because of xz.

[peebee]

@peebee Did a fresh 32 bit build with a 64 bit kernel and it boots fine,

X doesn't start but this is common in 32 bit builds on this 2015 laptop. And everything is super sluggish (boot time is horrible), probably because of xz.

Mine wasn't a new build - it was an existing 32 bit build with the 32-bit kernel swapped for a 64-bit kernel.....