Comparing aufs vs overlay

[dimkr]

As long as the swapping operation was done correctly and the kernel is not corrupt or misconfigured, result should be the same.

[peebee]

As long as the swapping operation was done correctly and the kernel is not corrupt or misconfigured, result should be the same.

The result is the same - I downloaded your test build and it fails to get to a working desktop .... just like my initial test.

If I swap the kernel for a 64-bit kernel-kit usrmerge version with both aufs & overlay - a boot with aufs works just fine but a boot with punionfs=overlay fails just like the dpup-kernel......

Attached /var/log/Xorg.log from the 2 boots and a diff

FWIW - LxPupSc32 with 64-bit kernel & punionfs=overlay boots to a sort of desktop - but unusable - all sorts of graphical glitches and peculiarities.....

[mistfire]

@peebee
Could you please enable these OVERLAYFS options to your kernel?

Code: Select all

CONFIG_OVERLAY_FS_REDIRECT_DIR=y
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y

[peebee]

Could you please enable these OVERLAYFS options to your kernel?

Could you explain why these are needed?

[ozsouth]

I'm interested in the reasoning too, as for my uses, I don't see a need.

From kernel.org overlayfs ref:

Renaming directories

When renaming a directory that is on the lower layer or merged (i.e. the directory was not created on the upper layer to start with) overlayfs can handle it in two different ways:

1. return EXDEV error: this error is returned by rename(2) when trying to move a file or directory across filesystem boundaries. Hence applications are usually prepared to handle this error (mv(1) for example recursively copies the directory tree). This is the default behavior.

2. If the "redirect_dir" feature is enabled, then the directory will be copied up (but not the contents). Then the "trusted.overlay.redirect" extended attribute is set to the path of the original location from the root of the overlay. Finally the directory is moved to the new location.

Kernel config options:

OVERLAY_FS_REDIRECT_DIR:
If this is enabled, then redirect_dir is turned on by default.

OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW:
If this is enabled, then redirects are always followed by default. Enabling this results in a less secure configuration. Enable this option only when worried about backward compatibility with kernels that have the redirect_dir feature and follow redirects even if turned off.

[mistfire]

Could you please enable these OVERLAYFS options to your kernel?

Could you explain why these are needed?

To allow more granular control on overlayfs. I made experiment were some files are added on the lowerdir even creating a new directory. It turns out that the added files folder on the lower directory was not reflected on the merged mountpoint of overlayfs. even issue remount command on the overlayfs "/" mount. That changes on the lowerdir performs the critical role regarding on-the-fly loading of sfs modules. Where my concept was creating symlinks of sfs module contents on a separate lower directory instead of upper directory which supposed to be reflected on the merged mountpoint. The current on-the-fly loading of sfs modules creates symlinks on the upper directory, however the biggest drawback was the configuration files were read-only and does not allow to modify because it was linked to read-only sfs modules. If the symlinks were made at the lowerdir then the merged point sees the config files from sfs module was can be write/delete

[dimkr]

If your intention is to add a layer with symlinks below the upper, you won't be able to modify it freely. Modifications of layers other than upper interfere with file system cache and can lead to data loss and things like contents of lower layers and whiteouts in the upper layer running out of sync. Been there, done that - not recommended. The overlay documentation warns against direct changes to layers for very good reasons.

[peebee]

To allow more granular control on overlayfs. I made experiment

I think you probably need to "grow your own" kernel for your experiments - it's not too hard to do using Woof-CE kernel-kit.....

https://github.com/puppylinux-woof-CE/w ... kernel-kit

Download Woof-CE and run it locally.

https://github.com/puppylinux-woof-CE/woof-CE - green Code button - Download Zip

[mistfire]

If your intention is to add a layer with symlinks below the upper, you won't be able to modify it freely. Modifications of layers other than upper interfere with file system cache and can lead to data loss and things like contents of lower layers and whiteouts in the upper layer running out of sync. Been there, done that - not recommended. The overlay documentation warns against direct changes to layers for very good reasons.

Too bad only one upperdir is allowed. That overlayfs was crippled to deal with. Creating symlinks on the upper layer creates symlink mess that's why I supposed isolate the symlinks. Another workaround must be made

[ozsouth]

(Sorry to depart from subforum subject).

@mistfire - Symlink issues & live sfs loading issues - that's why, in my upcoming remaster of s15pup64, I've restricted sfs-loading to pupmode 5, as bad user choices cause issues. Even that can go bad if devx is loaded live, compiling fails.
However, queue & unqueue does work (for non-pupmode 5), if static location for sfs is used. I'm testing that, without live-loading. Just means a reboot.
Safest option is loading sfs on bootup & using .pets (or whatever package manager your system uses).

[bigpup]

Maybe it has changed.

But SFS-load-on-the-fly would not let an SFS load until a save was made and being used.
Mainly to have a place to save the settings, to load the SFS on next boot up, if you still had it selected to load.
Always have SFS packages stored in /mnt/home, so the SFS is never in the save.

It has always been the practice to have a save made and booted using it, before doing any compiling.

[peebee]

Thoughts from BarryK's blog on his experiences with overlayfs:

I just have to give up. When Dr. Okajima retires from developing Aufs, I will also retire from Linux development.

https://bkhome.org/news/202402/overlay- ... aster.html

[dimkr]

https://bkhome.org/news/202402/overlay- ... aster.html

This is the error you get if you modify lower layers despite what the warning in overlay documentation says. aufs allows this, overlay doesn't.

[fredx181]

Re: SFS loading 'on-the-fly' for overlayfs : My vision would be: Better give up thinking about possible options to simulate AUFS behavior (e.g. by using symlinks or whatever) and go for some "poor man's" SFS loading by using e.g. unionfs and chroot with a simple applications menu GUI (e.g. with gtkdialog or yad, for a single application SFS it's useful I think ), not ideal, but should work well in combination with overlayfs booted system.
See also here: (works for KLV-Airedale) : viewtopic.php?p=81066#p81066

[fatdoguser]

Overlay won't add and remove layers after boot like aufs will, so you wouldn't be able to load and unload sfs files on the fly.

If/when the likes of brtfs (that uses copy on write, snapshots, compression etc) commonly supersedes ext then a btrfs device mounted with compression and populated, which is then later dynamically added/deleted to/from a set as a additional read only volume/device is pretty much the same as dynamically adding/removing a sfs. In that context overlayfs isn't even required. Less a case of overlay + sfs vs aufs + sfs, more a case of just btrfs vs ext 4 + aufs + sfs

[rockedge]

go for some "poor man's" SFS loading by using e.g. unionfs and chroot with a simple applications menu GUI (e.g. with gtkdialog or yad, for a single application SFS it's useful I think ), not ideal, but should work well in combination with overlayfs booted system.
See also here: (works for KLV-Airedale) : viewtopic.php?p=81066#p81066

This mechanism for loading a SFS in KLV-Airedale, KLV-Spectr (and will also work in KLV-Hyprland-CE-4.8) has so far worked as expected and has performed reliably. And I agree that SFS load and unload for single application is the target use case.

[ozsouth]

Looks like the way to go. Imitating aufs has issues. In my s15pup64ovrly, I have load/unload (view only) working, install/uninstall in pupmode 5 only, & queue/unqueue working otherwise (with a recent mitigation). Is functional, but less so than aufs. Good enough for me, but unionfs & chroot sounds like the next step forward. BTW, I believe kernel 5.15.150+ has an aufs problem. To me, this is the real reliability issue.

[BarryK]

https://bkhome.org/news/202402/overlay- ... aster.html

This is the error you get if you modify lower layers despite what the warning in overlay documentation says. aufs allows this, overlay doesn't.

I am NOT modifying lower layers!!!

A note about aufs in EasyOS. I have compiled the kernel with writing to lower layers disabled. EasyOS does not write to lower layers, and does not support loading SFSs on-the-fly.

[dimkr]

@BarryK

Maybe your kernel is misconfigured - from what I see, https://github.com/bkauler/woofq/blob/5 ... /init#L571 mounts / with the defaults, but CONFIG_OVERLAY_FS_METACOPY=y is not a good fit for the Puppy use case (snapmergepuppy doesn't have a file to copy to pup_ro1).

This is what I use:

Code: Select all

CONFIG_OVERLAY_FS=y
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set
# CONFIG_OVERLAY_FS_DEBUG is not set

and I mount the overlay at / with xino=on,metacopy=off (force-enable xino, force-disable metacopy if user replaced the kernel with a misconfigured one).

[BarryK]

@BarryK

Maybe your kernel is misconfigured - from what I see, https://github.com/bkauler/woofq/blob/5 ... /init#L571 mounts / with the defaults, but CONFIG_OVERLAY_FS_METACOPY=y is not a good fit for the Puppy use case (snapmergepuppy doesn't have a file to copy to pup_ro1).

This is what I use:

Code: Select all

CONFIG_OVERLAY_FS=y
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set
# CONFIG_OVERLAY_FS_DEBUG is not set

and I mount the overlay at / with xino=on,metacopy=off (force-enable xino, force-disable metacopy if user replaced the kernel with a misconfigured one).

I will try that thanks.

Just had a thought... the read-write layer is zram, with ext2 filesystem.
My thought was that the ext2 f.s. may be deficient, as overlay needs xattr, but no, kernel config has everything enabled:

Code: Select all

CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y

[mikewalsh]

A note about aufs in EasyOS. I have compiled the kernel with writing to lower layers disabled. EasyOS does not write to lower layers, and does not support loading SFSs on-the-fly.

.....which I guess explains why folks have had success running some of my portables under EasyOS, AND why packages you've provided for EasyOS are all in .pet format. Thanks for the clarification.

Mike.

[fatdoguser]

https://bkhome.org/news/202402/overlay- ... aster.html

This is the error you get if you modify lower layers despite what the warning in overlay documentation says. aufs allows this, overlay doesn't.

I am NOT modifying lower layers!!!

A note about aufs in EasyOS. I have compiled the kernel with writing to lower layers disabled. EasyOS does not write to lower layers, and does not support loading SFSs on-the-fly.

Fatdog uses aufs, but I use overlayfs in addition to that. As part of standard bootup Fatdog's main sfs is copied into ram /aufs/pup_ro, I have a chrome sfs that is loaded into /aufs/pup_ro11 and my re-configuration of base Fatdog sfs gets loaded into /aufs/pup_ro10, which I then use to set up a overlayfs with those at the bottom layer, work/upper (changes) stored in /tmp, and top layer mounted as /mnt/Xephyr - that I then chroot into and set things up to run as spot. Similar I believe in some respect to EasyOS containers. Like EasyOS that also uses little additional resources. You can open and close that quick-n-easy, no need for reboots and the bottom layer could be extended to include any other sfs(s), similar to dynamic loading/unloading of sfs. So in a sense, EasyOS containers are a potential means to load/unload SFS's on the fly, but with the restriction that a new container has to be loaded

Code: Select all

#!/bin/sh

if [ ! -d /mnt/Xephyr/root ] #                        Not already previously run
then

################################################################################
                                                    #       Prepare script files
                                                    #     one for chroot as root
  cat > root_chroot << EOF
#!/bin/ash

if [ -f /etc/shadow ]; then    #       not already been run before
            

  echo Tweaking the layered filesystem and starting sound
  chmod 1777 /dev/shm          #               Chrome requires shm
  chown -R spot /dev/snd       #  Make sound & mouse owned by spot
  chown spot /dev/input/mice

  rm /etc/wpa_supplicant.conf  #             contains wifi details
  rm /root/.wifi.db            #           contains wifi passwords
  rm -rf /root/.ssh            #                ssh keys/passwords
  rm -rf /root/.pki            #                              keys
  rm /etc/shadow*              #  spot can't su (no passwd hashes)

  alsactl restore              #                      restart alsa

  cd /home/spot
  sync

fi
su spot -c /home/spot/su_spot
EOF
################################################################################
                                            # another script file for su as spot
  cat > su_spot <<EOF
#!/bin/ash

echo ". /etc/profile" > /tmp/my_xinitrc
echo "jwm" >> /tmp/my_xinitrc
xinit /tmp/my_xinitrc -- /usr/bin/Xephyr \
:2180 -ac -br -dpi 144 -glamor -fullscreen
rm /tmp/my_xinitrc
EOF
################################################################################

                                        #        Prepare and setup overlayfs
  mkdir /tmp/overlay /tmp/overlay/changes
  mkdir /tmp/overlay/work /mnt/Xephyr
  mount -t overlay -o \
lowerdir=/aufs/pup_ro10:/aufs/pup_ro11:/aufs/pup_ro,\
upperdir=/tmp/overlay/changes,\
workdir=/tmp/overlay/work none /mnt/Xephyr
                                            #                 Prepare for chroot
  sync
  mount --bind /proc /mnt/Xephyr/proc
  mount --bind /sys /mnt/Xephyr/sys
  mount --bind /dev /mnt/Xephyr/dev
  mount --bind /tmp /mnt/Xephyr/tmp
  mount --bind /dev/shm /mnt/Xephyr/dev/shm
  mount -t devpts devpts /mnt/Xephyr/dev/pts

  echo "Preparing sound ... "
  cp /root/.asoundrc /mnt/Xephyr/root/.
  cp /root/.asoundrc /mnt/Xephyr/home/spot/.asoundrc
  chown spot:spot /mnt/Xephyr/home/spot/.asoundrc

  cp /etc/resolv.conf /mnt/Xephyr/etc/resolv.conf  #         So networking works
  cp /etc/hosts /mnt/Xephyr/etc/hosts              # Our adblock like hosts file

  mv root_chroot /mnt/Xephyr/root/.
  chmod +x /mnt/Xephyr/root/root_chroot
  mv su_spot /mnt/Xephyr/home/spot/.
  chown spot:spot /mnt/Xephyr/home/spot/su_spot
  chmod +x /mnt/Xephyr/home/spot/su_spot
  

  cp /etc/localtime /mnt/Xephyr/etc/localtime
  cp /etc/X11/xorg.conf.d /mnt/Xephyr/etc/X11/xorg.conf.d
  cp /root/.fatdog/xkeymap /mnt/Xephyr/root/.fatdog/xkeymap
  cp /root/.fatdog/language /mnt/Xephyr/root/.fatdog/language
  

  echo chrooting into the layered system

fi
                                          # chroot initially in as root, secures
chroot /mnt/Xephyr /root/root_chroot      #   session and then that su's as spot

sync                                      #                              Tidy up
cd
umount /mnt/Xephyr/dev/pts 1>/dev/null 2>/dev/null
umount /mnt/Xephyr/dev/shm 1>/dev/null 2>/dev/null
umount /mnt/Xephyr/tmp 1>/dev/null 2>/dev/null
umount /mnt/Xephyr/dev 1>/dev/null 2>/dev/null
umount /mnt/Xephyr/sys 1>/dev/null 2>/dev/null
umount /mnt/Xephyr/proc 1>/dev/null 2>/dev/null
umount /mnt/Xephyr 1>/dev/null 2>/dev/null
umount /dev/zram0 1>/dev/null 2>/dev/null
rm -rf /tmp/overlay

################################################################################

It's not unreasonable to run two such overlayfs, so for example a second version could be started with a additional sfs added to the bottom layer, and the 'changes' layer copied from the first overlayfs, and then close that first overlayfs, so the second overlayfs was the exact same as the first, but has the additional sfs added in. And similar in the other direction to 'unload' the sfs.

[ozsouth]

Interesting concept! Incomplete uninstalling was why I restricted installing to pupmode 5. Does this system allow persistence, saving data files associated with installed apps?

[fatdoguser]

Interesting concept! Incomplete uninstalling was why I restricted installing to pupmode 5. Does this system allow persistence, saving data files associated with installed apps?

Who is your question directed at @ozsouth?

If me then yes the 'changes' (upper) folder content is a record of changes. I don't preserve/persist that myself, but no reason why it couldn't be copied and reapplied again (persisted), even if not directly by it being copied/extracted to the top layer after reforming the overlayfs.

overlayfs 'work' folder has to be empty and on the same filesystem as the 'upper' (changes), basically its just a tmp type folder into/from which transitory files/folders are moved by overlayfs as/when required - i.e. being on the same filesystem such moves are just inode/pointer type operations, not actual full data set moves/copies. When files are flagged as deleted rather than creating a .wh whiteout file in the changes/upper layer overlayfs seems to create a character device of the filename instead.

[BarryK]

Overlay won't add and remove layers after boot like aufs will, so you wouldn't be able to load and unload sfs files on the fly.

If/when the likes of brtfs (that uses copy on write, snapshots, compression etc) commonly supersedes ext then a btrfs device mounted with compression and populated, which is then later dynamically added/deleted to/from a set as a additional read only volume/device is pretty much the same as dynamically adding/removing a sfs. In that context overlayfs isn't even required. Less a case of overlay + sfs vs aufs + sfs, more a case of just btrfs vs ext 4 + aufs + sfs

@fatdoguser
Interesting statement about btrfs. I am just getting into learning about it. Started off, I got interested in bcachefs, which got mainlined in the 6.7 kernel; but after learning how immature it is, I then thought about btrfs. And discovered that native-encryption is coming for btrfs. I posted about it to my blog:

https://bkhome.org/news/202403/linux-ke ... crypt.html

...plan to compile the 6.8.x kernel with those 52 patches very soon.

Anyway, what you mentioned about btrfs subvolumes and snapshot and compression pretty much eliminating the need for SFSs and overlay/aufs; yeah very interesting. Then for me, per-folder native encryption would be the icing on the cake.

Now all I have to do is understand how all that btrfs subvolume/snapshot stuff works. Absolute beginner with btrfs.