aufs future

[ozsouth]

With the end of support for aufs on kernels prior to 5.15, once 5.15 goes EOL in Oct 2023, there will be no pre-6 longterm kernels supported by aufs.
We thus have a year to settle on plans B, C, D . . . Pups with overlayfs; Hope that kernel 6.x doesn't cause too much drama; Or?

[dimkr]

Pups with overlayfs

Already implemented and the default in my development builds

[rockedge]

From the old forum, a few years ago, somebody made a Puppy Linux using only overlay. I ran that for some tests and it all worked including Load SFS. At the time I just played around with the distro and didn't investigate further.

I have mentioned this Puppy version before. Perhaps I have a copy of the ISO or maybe @ally has archived it. Anyway it might be worth looking at how it was setup.

Interesting link -> https://www.mediafire.com/folder/inbkdjmie89hm/overlay
Interesting link -> https://www.mediafire.com/folder/q04fxq43exkfr/mio
Interesting link -> http://www.fishprogs.software/puppy/

[ally]

do you have a name of the ISO to look for?

[rockedge]

Looking back I think I had a delta file to apply to a stock Bionic32 and 64 ISO and I made the customization to the stock distro and switched to Overlayfs.

[ally]

https://archive.org/download/Puppy_Linu ... .iso.delta ?

https://archive.org/download/Puppy_Linux_Delta, F3 "overlay" brings up a dozen or so

hth

[peebee]

Pups with overlayfs

Already implemented and the default in my development builds

@dimkr does the kernel have to be specifically configured to make your version of overlayfs support work properly?

In particular must the kernel be configured with:
CONFIG_OVERLAY_FS=y
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y

and must it also be a so-called "usrmerge" kernel?

Thanks for clarification.

[rockedge]

For KLV I made a kernel with the kernel-kit that has no AUFS patches at all and has built in Overlayfs. Puppy Linux huge kernels work in KLV that use overlayfs as a module and is not built in at compile time.

I think the usermerge is for converting the kernel modules component from the Puppy Linux used file system structure to the type used in Void Linux and other newer distros (/usr/lib instead of /lib)

[dimkr]

@dimkr does the kernel have to be specifically configured to make your version of overlayfs support work properly?

CONFIG_OVERLAY_FS=y

and must it also be a so-called "usrmerge" kernel?

Nope, that's irrelevant.

[peebee]

CONFIG_OVERLAY_FS=y

@gyrog's work on overlayfs in 2017 ( https://oldforum.puppylinux.com/viewtopic.php?t=110636 ) only required:
CONFIG_OVERLAY_FS=m
so that is how I suspect most Puppy kernels are configured (except presumably the Github Woof-CE kernel-kit 5.10 and 5.15 kernels)

[dimkr]

overlay is small and modprobe overlay would make Puppy slower to boot.

(Let's not forget that Puppy kernels are built with CONFIG_AUFS_FS=y, although aufs is an out-of-tree file system, bigger and more complex than overlay. In addition, most Puppy kernels contain many unused built-in drivers, and they're much bigger than overlay. If CONFIG_AUFS_FS=y is OK, I don't see a reason why CONFIG_OVERLAY_FS=y isn't.)

I can add a warning, so kernel-kit warns people who try to build kernels with CONFIG_OVERLAY_FS=n or CONFIG_OVERLAY_FS=m. That would make them more future-proof.

[ozsouth]

Kernel 6.0 is now at rc6 - expect final release by end of sept. At that point, kernels prior to 5.15 will not have aufs updated & will fail to compile with aufs sometime in the next few months. A solution is to compile with overlayfs only. I have been experimenting & have made a usable solution, with some limitations.
See: https://forum.puppylinux.com/viewtopic. ... 0ef#p67200

[dimkr]

@ozsouth woof-CE already supports overlay, the weekly kernel builds now have an overlay-only flavor, and initrd uses overlay if aufs is unsupported. My builds use this "native" overlay support.

https://github.com/puppylinux-woof-CE/w ... nt-3622699

https://github.com/puppylinux-woof-CE/woof-CE/pull/3398 will take it one step further by simulating dynamic SFS loading on PUPMODE 5 and 13.

Contributors and feedback are welcome

[ozsouth]

@dimkr - Interesting, particularly the second link. My solution works sufficiently for me now, but I don't see huge value in my inventing an inferior wheel, so now I understand a bit more, I will look more closely at your work in future.

[dimkr]

I don't see huge value in my inventing an inferior wheel

Same, but overlay doesn't allow (safe) modifications of lower layers, hence the PUPMODE 5/13 limitation.

[ozsouth]

Today I discovered that the only AUFS ACTIVE branches are 5.x-rcN, 5.10.117, 5.10.140, 5.19.
All 5.15 is listed as stale, meaning no further updates after 5.15.41. Later 5.15 kernels will compile for a while (some months?) but one will eventually be the breaking point, as has happened in the past.
The good news is that as 5.10 EOL is more than 4 years away, updating it's aufs, along with the current mainline branch (as appears to be the plan now), makes great sense and is more efficient use of sfjro's time. We are so grateful to him for his efforts. I expect 6.x-rcN will become active in the next few weeks.
Overlayfs is unaffected by this situation, so I plan to only release aufs/overlayfs combined kernels in future.

[rockedge]

I was looking over aufs5 and found some information that might be helpful so I made a page to look over.

Chart for aufs5

Screenshot(32).jpg (33.59 KiB) Viewed 7460 times

[dimkr]

The good news is that as 5.10 EOL is more than 4 years away

If aufs breaks against 5.10.145 (which will be out soon) and any Puppy that uses 5.10.x and aufs is stuck with 5.10.144 (the last 5.10.x that works with aufs), why does it matter that 5.10.x is a longterm release with 4 (unused) years of support?

Also, if 6.0 is the next longterm kernel, will sfjro maintain it for years to come? That seems very unlikely, if the current policy is to drop support for longterm kernels relatively early in their support window. aufs will force those who adopt 6.0.x to switch to 6.1.x few months later, even if 6.0.x is a longterm release and 6.1.x isn't.

aufs is not a viable option for Puppy anymore, unless you chase the latest kernel and constantly need to deal with deprecation of drivers for old hardware, increased resource usage and new bugs introduced by new features. IMO, Puppy is better off using longterm kernels without third party patches like aufs.

[ozsouth]

@dimkr - I was pleasantly surprised that 5.10 looks like getting longer term aufs patches now, as a couple of weeks ago it was destined to be dropped. I do understand it is a release-by-release proposition - one never knows when it will break & be followed a week or so later with an aufs point release update. Maybe hanging back a few releases is wise? I'm following up on overlayfs as well. It appears to be the future choice.
I'm glad the 5.10.141 I made a few days ago was overlayfs only, or it would be broken & I would have only found out why today.
I made a 5.10.144 combined aufs/overlayfs kernel today & am doing final testing. Seems fine, so I'll make wireless drivers & release it.
The average puppian doesn't seem enthralled by overlayfs just yet.

[peebee]

Today I discovered that the only AUFS ACTIVE branches are 5.x-rcN, 5.10.117, 5.10.140, 5.19.

I think this is just a Github classification of branches that have not seen any changes for a period of time - the other branches are still there and can be used as normal.

[ozsouth]

Here's the link I took my picture (5 posts up) from: https://github.com/sfjro/aufs5-standalone/branches

Unless I'm reading it wrongly, it looks like aside from those 4 branches, everything else is now denoted 'Stale',
meaning they won't be worked on. Stale branches would work for a while, but within a few months, likely fail.

[dimkr]

the other branches are still there and can be used as normal.

Even if the patches in the "stale" branches still apply cleanly to the latest 4.19.x, 5.4.x, 5.10.x or 5.15.x, nothing guarantees that aufs is safe to use against a kernel it was never tested against.

If nobody who knows the aufs codebase in and out looks at the changelog between x.y.z and x.y.z+1 to see if aufs is broken or needs adjustments, we just trust aufs blindly. Maybe a newly introduced fix triggers a bug in aufs, and now it needs to copy a file up or down through the stack in some situation, although it didn't need to do that before?

(I'm not trying to spread FUD, only show that "still builds" is not the same as "still works exactly as before", and "still works" is not the same as "still works, apparently", "still works and it's a miracle" or "still works, because somebody did the dirty work to guarantee that")

[peebee]

https://github.com/marketplace/actions/stale-branches

Stale Branches
Creates issues for branches that have become stale. By default it aligns with this definition, but can be configured for other use cases.

When a branch has been inactive for more than the days-before-stale input, an issue is opened with the title [branch-name] is STALE. The branch will be deleted once it has been inactive longer than days-before-delete.

By default, a stale branch is defined as a branch that:
has had no commits in the last 120 days.
has no protection rules.
has no open pull requests.
is not the default branch of the repository.
See inputs for more info.
See example workflow.

[ozsouth]

I guess my concern is also about unforeseen breakage. Case in point: 5.4 kernel series -
Work on it stopped in Sept 2021. I made 2 kernels - 64bit 5.4.159 & 32bit 5.4.158, for which source was released
in Nov 2021. Both worked fine. Then I tried 5.4.163 which was broken, as was my every 5.4 aufs attempt following.
So it took about 3 months from when worked stopped on it for 5.4 to break. If that is the norm, we have limitations.
I had thought about asking our aufs man to make a 5.4.160 -ish patch, but as I saw yesterday, he recently declined a
similar request. Earlier this week, I was able to build a 5.4.210 overlayfs-only (no aufs) kernel, which works fine.

[amethyst]

I guess my concern is also about unforeseen breakage. Case in point: 5.4 kernel series -
Work on it stopped in Sept 2021. I made 2 kernels - 64bit 5.4.159 & 32bit 5.4.158, for which source was released
in Nov 2021. Both worked fine. Then I tried 5.4.163 which was broken, as was my every 5.4 aufs attempt following.
So it took about 3 months from when worked stopped on it for 5.4 to break. If that is the norm, we have limitations.
I had thought about asking our aufs man to make a 5.4.160 -ish patch, but as I saw yesterday, he recently declined a
similar request. Earlier this week, I was able to build a 5.4.210 overlayfs-only (no aufs) kernel, which works fine.

It seems I am using just the right aufs version then. Using your 5.4.158 version with all my Puppys from Precise upwards. Works great.

[dimkr]

Using a old kernel from 2021 is risky, see https://www.linuxkernelcves.com/streams/5.4. So many known and documented vulnerabilities affect 5.4.158

[peebee]

Github kernel-kit now has a 5.4.x patch:
https://github.com/puppylinux-woof-CE/w ... .4.x.patch

[LateAdopter]

the other branches are still there and can be used as normal.

Even if the patches in the "stale" branches still apply cleanly to the latest 4.19.x, 5.4.x, 5.10.x or 5.15.x, nothing guarantees that aufs is safe to use against a kernel it was never tested against.

If nobody who knows the aufs codebase in and out looks at the changelog between x.y.z and x.y.z+1 to see if aufs is broken or needs adjustments, we just trust aufs blindly. Maybe a newly introduced fix triggers a bug in aufs, and now it needs to copy a file up or down through the stack in some situation, although it didn't need to do that before?

(I'm not trying to spread FUD, only show that "still builds" is not the same as "still works exactly as before", and "still works" is not the same as "still works, apparently", "still works and it's a miracle" or "still works, because somebody did the dirty work to guarantee that")

Ubuntu HWE kernels for Focal include AUFS because it is required by docker, so 5.15 kernels with aufs are tested as much as anything in the linux kernel is.
https://git.launchpad.net/~ubuntu-kerne ... h=hwe-5.15

The 5.19 series of kernel.org stable kernels has been particularly bad as 5.19.10 was the first without reverts that suggest the release was not adequately tested to start with.

I have been wondering whether JRO is trying to ease himself out of the job without leaving users in the lurch.

[dimkr]

Ubuntu HWE kernels for Focal include AUFS because it is required by docker

Are you sure it's required? AFAIK Docker should default to overlay2 and the Ubuntu live CD uses overlay too, but I could be wrong. I've been using Docker with overlay2 for years, and nowadays I just use Podman with overlay. Outside of the live distro world, I don't see any use of aufs.

[LateAdopter]

I only saw the commit that dropped aufs from Ubuntu versions after Focal. It was at the time kernel 5.8 was introduced and JRO was off the job for a while. Debian, Ubuntu and Puppy had their own patches for aufs for 5.8.

The Ubuntu policy was to include aufs in the kernel source, but only build it for Focal. Docker was given as the reason. Later versions of Docker don't depend on aufs, but Ubuntu's policy was not to backport the feature change to Focal.

The current 5.15 kernel source for Jammy still includes aufs
https://git.launchpad.net/~ubuntu-kerne ... ee/fs/aufs

Whether aufs gets built depends on the config. Ubuntu have numerous kernel variants for Focal and I haven't looked to see which ones build it but there is a 5.10 config that builds it.
https://kernel.ubuntu.com/~kernel-ppa/c ... lavour.oem