aufs future

[peebee]

What causes these problems with aufs?

The kernel developers have backported changes from 5.12 into 5.10.220 which mean that the aufs-5.10.140 patches no longer work. The changes are considerable. aufs-5.10 is no longer supported. aufs-5.15 is no longer supported but continues to work. Shortly, aufs-6.1 will become unsupported but should still work. aufs-6.6 will be the only supported LTS branch.

[greengeek]

What particular extra abilities of aufs makes it so valuable to you?

Good question. Alas - I have no answer as i fail to understand why a software layering system reaches end of life.

Seems to me that the hardware manufacturers have somehow created a monster.

What is it about modern hardware that prevents it from handling software layers in RAM?? I don't get it.

[dimkr]

i fail to understand why a software layering system reaches end of life.

Its developer can't or doesn't want to support it to the same degree as before. Some unmaintained software can live on because it still works: ROX-Filer is dead for years but still works because GTK+ 2 is dead as well so nothing ROX-Filer relies on changes. Howver, aufs is special. It's really tied to a particular kernel version because it's a modification of the kernel and not some external addon with a clear and stable boundary between the two: the same aufs won't work on kernel versions other than the one it targets, and it can break even when you advance from kernel version x to version x plus one bug fix. It's fragile and needs maintenance work - if nobody does this work, it remains broken. aufs is pretty much guaranteed to break if its sole developer decides to abandon the project completely instead of reducing the lifetime of each version and reducing the number of supported versions, unlike things like ROX-Filer.

[greengeek]

aufs is special. It's really tied to a particular kernel version because it's a modification of the kernel and not some external addon with a clear and stable boundary between the two: the same aufs won't work on kernel versions other than the one it targets, and it can break even when you advance from kernel version x to version x plus one bug fix.

Thanks. That's disturbing. Seems like at least a decade of Pup development becomes irrelevant because the kernel has no choice but to keep up with ever changing hardware - and layering techniques are apparently forced to change in lockstep.

Makes me feel I have little choice but to continue sticking with Vista-era hardware.

Still don't understand why the Puppy method of building layers is impacted by kernel structure. Isn't there some other way? (Or am I totally misunderstanding all this...?)

Will future Puppies be able to layer (and live-load/unload) sfs files as the kernel "progresses" ??)

[wiak]

What particular extra abilities of aufs makes it so valuable to you?

Good question. Alas - I have no answer as i fail to understand why a software layering system reaches end of life.

Seems to me that the hardware manufacturers have somehow created a monster.

What is it about modern hardware that prevents it from handling software layers in RAM?? I don't get it.

No it isn't that, greengeek, it is simply that aufs, in a way like Puppy Linux, doesn't have sufficient development support and because aufs is not provided by kernel team so needs specially patched kernel sources and thus specially compiled kernels; more effort and less tested for reliability and security.

[dimkr]

Still don't understand why the Puppy method of building layers is impacted by kernel structure. Isn't there some other way? (Or am I totally misunderstanding all this...?)

aufs is a modification of the kernel, so when a new kernel version comes out, the modifications need to be validated against the new version, The changes can break the correctness of the aufs code, introduce new bugs in aufs or simply break aufs (if aufs relies on something removed from the kernel). I've seen many cases where the kernel crashes or the system locks up because aufs kept building fine but a kernel change broke aufs at runtime.

overlay is part of the kernel itself, it's not a third-party addon, so it doesn't break like aufs does and doesn't need this extra maintenance work. A Puppy that uses overlay instead of aufs doesn't have these frequent issues. In addition, overlay is smaller and has narrower scope (less edge cases to handle, less room for bugs). The downside is that aufs has extra features and Puppy uses them, with imperfect fallbacks when using overlay instead. If and when aufs dies completely, you'll need to decide between using Puppy but with overlay, or another distro. Personally, I switched long ago because I hate these surprises and prefer something stable and future-proof.

[wiak]

Will future Puppies be able to layer (and live-load/unload) sfs files as the kernel "progresses" ??)

Tinycore linux is composed of dozens of sfs files, always has been, but it doesnt use layering but instead symlinks everything in, and you can load in live via that method. That methodology has since been implemented by fredx181 for live loading sfs in DebianDog. Later, same idea added by dimkr I believe into Puppy Linux systems that use overlay fs rather than becoming problematic aufs. KL distros can also do it because fredx181 ported over his debiandog utility.

[jamesbond]

overlay is part of the kernel itself, it's not a third-party addon, so it doesn't break like aufs does and doesn't need this extra maintenance work.

I beg to differ.

overlayfs was once upon a time, an add-on too. Just like aufs, overlayfs was once a external kernel module, just like aufs. There is no difference in that aspect.

The only difference is, the guys in charge of the kernel decided to "adopt" it and include it in the kernel source code that they release, while aufs is not included. To use overlayfs, you just need to enable it, and compile the kernel. To use aufs, you need to "bring it into the kernel" - this is what "aufs patches" do. But once brought-in, aufs works and compiles exactly like overlayfs. There is no difference.

So why didn't the kernel guys adopt aufs too? Who knows. The official reason is because aufs is "too complex" - but then they accepted untested, brand-new, 27,000 SLOC of code for the NTFS3 driver without even blinking. Hmmm. Might need to look up who is the sponsor behind overlayfs. It might give you a clue.

There is no difference in terms of "bugs" whether the code is in the kernel source release, or outside. People who follows the overlayfs development would remember that the first release of overlayfs as a in-built kernel module was extremely buggy to the point of it was almost useless (except for a very niche use case - the only use case used and tested by the promotor of the overlayfs code before it was merged; as opposed to aufs which had been stable for __years__).

Another case in point, regarding that NTFS3 code example. Have you got any corrupted filesystem because of aufs? How about because of in-kernel NTFS3? Hmmm, it makes you think, doesn't it?

Things have gotten better since (for overlayfs), but not because it is "part of the kernel", but because somebody somewhere spent some effort to fix those bugs.

Which brings us to "bug-fixing", or maintenance. As far as maintenance is concerned, __everything__ related to kernel has to be maintained, whether it is "part of the kernel source code", or anything outside it. The kernel changes all the time, and the changes always break things (inside and outside kernel). There is no magic that because something in included in the kernel source, it is maintenance-free. The maintainers of certain part of the kernels (whether or not the code is part of the kernel source code, or outside; whether the guys maintaining it are part of the kernel team, or outside) have to do the work, or their stuff will break on the next kernel release. Overlayfs, just like aufs (and any other parts of the kernel), has to be maintained, or it will break.

The only difference is, if the kernel module is included in the kernel, then other people in the kernel team might do the work for the actual maintainer if the maintainer of that module disappears for a while. Most changes are only trivial interface changes, and such changes can be adapted so that the module still builds, without changing the functional part of the module code itself. But, doing such changes is no guarantee that the module still performs correctly. It only enables the module to build.

Contrast this to the external kernel module, where there is no holiday for the maintainer. If the kernel changes its interfaces, the maintainer has to adapt to it or the module won't even compile. For all other aspects, however, the maintenance process is identical.

Hence, it is clearly a myth that because the module is included in the kernel source, it will always be "more correct" or "less buggy" that external module. The correctness or bugginess depends on the effort put by the maintainers of that piece of the code, irregardless of whether the code is inside or outside.

Here's a food for thought. Aufs isn't the only external module in the world. You people who have used some USB wifi dongle, especially the ones with Realtek chips inside, are aware that most of them aren't supported directly by the kernel. External kernel modules are required. Are they bad? Shall we not use them? Hmmm.

Or how about nvidia, then. The included kernel module to handle nvidia kernel module, known as "nouevau", is thoroughly and completely buggy. It may be enough if what you do on your laptop is using libreoffice, but once you start something more serious about it, e.g. watching youtube, or worse if you start doing something with 3D in it (not necessarily for gaming, it can be for something else e.g. google earth, blender, etc), then you need some serious, non-buggy driver. Where do you get this non-buggy driver? From nvidia themselves, of course. The proprietary nvidia driver. This is, guess what, an external module, maintained by nvidia people, not from the kernel team. Are we going to dismiss this proprietary driver because it's not included in the kernel? Hmmmm.

One day, aufs will die. Like everything else.
Nobody or nothing will live or last forever.

But not today. Not yet.
In my opinion, aufs demise has been greatly exaggerated.

aufs is still maintained.
The author/maintainer has not indicated that he would retire very soon either.
As far as I'm concerned, it is still very much alive.

It's okay to prepare for the future.
But not by sacrificing the present.

I'm a pragmatist. I use what works. I use the best tool for the job.
And for now, for me, it is aufs.

[dimkr]

So why didn't the kernel guys adopt aufs too? Who knows. The official reason is because aufs is "too complex"

I think the main problem with aufs is the changes to things outside of aufs itself. I believe upstream kernel is very welcoming to new file system drivers if their developers don't touch code used by other file system, so the drivers are as independent as possible and a change in the ext4 driver won't affect you if you only use btrfs, because you don't use the ext4 driver at all and don't run any of the changed code lines.

That's not true in the case of aufs (it does touch shared code - the underlying API used by various file system drivers), otherwise you'd be able to build aufs as an out-of-tree driver, just like NVIDIA or Broadcom drivers.

[jamesbond]

I think the main problem with aufs is the changes to things outside of aufs itself. I believe upstream kernel is very welcoming to new file system drivers if their developers don't touch code used by other file system, so the drivers are as independent as possible and a change in the ext4 driver won't affect you if you only use btrfs, because you don't use the ext4 driver at all and don't run any of the changed code lines.

That's not true in the case of aufs (it does touch shared code - the underlying API used by various file system drivers), otherwise you'd be able to build aufs as an out-of-tree driver, just like NVIDIA or Broadcom drivers.

You have a point, but something that does what aufs/overlayfs does will have to touch shared filesystem code, like mmap, fcntl, file open/read/write, etc. It's unavoidable. I'm quite sure if overlayfs is not in-kernel, in order to use it, you will have to patch it in as well, just like what aufs does.

Hence it doesn't explain why they chose to accept the overlayfs version of "changing shared code", instead of not the aufs one.

[dimkr]

@jamesbond If I understand correctly, overlay uses generic VFS APIs to access lower layers, and that's where page cache kicks in and the reason why overlay forbids direct changes to lower layers. aufs, on the other hand, doesn't work like this. It has the layer attach/detach logic and deals with direct changes, so it needs to 'invalidate' inodes in various situations and there are many edge cases. It's not number of code lines but the complexity cost of the extra logic. But I understand the pragmatic argument of 'it works for now'.

[wanderer]

hi all

this is my 2 cents

as i have been saying for many many years

just use symlinks instead of a layered filesystem

and you will permanently get away from all this - its broken - has to be fixed - is the maintainer dead - clean up the white out files - garbage

wanderer

[fredx181]

...
just use symlinks instead of a layered filesystem
...

The "just" in what you propose makes me think that you don't realize how much change it needs to accomplish such system, it requires major modifications to be made for the existing puppy(-like) build system(s). (in fact a complete re-write, I'd say)
edit: and the solution to use overlayfs instead of aufs is not ideal, but serves well as replacement IMO (although with some options missing).

[rockedge]

Having used and worked with wd_multi for awhile now and many many other symlink scenario's with several different variations of operating systems I know that symlinks DO NOT always work as expected.

At root level in partitions can be problematic. Also during certain source code compilation if the actual file or directory is not present but instead a symlink, the build will not work.

Saving to persistence for example can be tricky with symlinked based distro structure. With wd_multi systems there has to have attention paid to where the persistence is actually being saved. The symlinks pointing to targets on other places.

[dimkr]

Symlinks can't achieve the same effect as an additional layer. You can't replace existing files and roll back on 'unload', among other issues. You can find workarounds for everything but at the cost of complexity and inefficiency or slowness. IMO requiring users to reboot to add a layer is both simpler for developers and less confusing for users, and I don't really see the value in dynamic loading and unloading (doesn't solve any problem for me).

[Duprate]

Hello! Reading this thread, I couldn't contain myself and I have to give my testimony: Since the covid epidemic, I have completely abandoned the use of AUFS and only use the kernel with overlayfs. It was difficult at first, but I adapted and use it on all systems installed on my PC. One of the things that motivated me was the desire to compile my own kernel and use the latest LTS, downloading the tarball directly from the website https://kernel.org/

[dancytron]

Symlinks can't achieve the same effect as an additional layer. You can't replace existing files and roll back on 'unload', among other issues. You can find workarounds for everything but at the cost of complexity and inefficiency or slowness. IMO requiring users to reboot to add a layer is both simpler for developers and less confusing for users, and I don't really see the value in dynamic loading and unloading (doesn't solve any problem for me).

It's handy to be able to right click on the LibreOfficeAndJava.sfs file to load it, but it doesn't really do anything a portable version or a version symilnked from /mnt/home to /opt doesn't do.

[mistfire]

hi all

this is my 2 cents

as i have been saying for many many years

just use symlinks instead of a layered filesystem

and you will permanently get away from all this - its broken - has to be fixed - is the maintainer dead - clean up the white out files - garbage

wanderer

There is one problem on symlinks method. The files on /etc and /var folders are read-only. The best solution was copy /etc and /var from sfs module to root filesystem

[ozsouth]

IMO requiring users to reboot to add a layer is both simpler for developers and less confusing for users

My 'ovrly' versions of s15pup64 do this, using queueing to install (single or multiple at once) sfs's, if a savefile/savefolder exists. They are then available at next bootup, which will be a bit slow due to once-only layering, but normal thereafter.
Without a savefile/folder sfs's can be installed & are available until shutdown/reboot unless uninstalled, just as with aufs.
Only other main restrictions are that puppy system sfs (including devx) can't be queued - must load at boot time, & software sfs's should be in same folder as puppy sfs (& savefile/savefolder). I've used some scripts to stop most bad choices.

My biggest beef with aufs is that when something breaks, it can be weeks before we're even sure it's broken & fix may be difficult, experimental or impossible. Doesn't inspire confidence. Overlayfs, with much wider testing/maintenance is safer.

[LateAdopter]

I have previously argued that AUFS is as tested as anything else because it is still used by Ubuntu Focal.

This is still true, but...
Ubuntu only maintains 5.4 and 5.15 up to date kernels for Focal, so later kernels do not have AUFS.

The Ubuntu 5.4 source has this AUFS patch, authored by JRO, that is not in the the last release of the 5.4.3 branch.
https://git.launchpad.net/~ubuntu-kerne ... 71b4f22cc4

The Ubuntu 5.15 source does not have any later patches.

Do any other major distributions still maintain kernels with AUFS?

[ozsouth]

@LateAdopter - Major distros, no aufs other than ubuntu AFAIK, although Porteus has some guys who suggested the 6.6.4 patch. Puppy has a 5.4.210 patch and a recent 5.15.150 one. As I've abandoned those, I don't have a copy.

[dimkr]

Old Ubuntu releases carry aufs but AFAIK it's unused. It was there because docker used aufs as the default (fuse-based alternatives were worse) but Ubuntu kept applying the aufs patches even after docker gained overlay support and made it the default. The aufs code in old Ubuntu releases that still receive updates doesn't really run, because AFAIK nothing uses it: if the list of (severe) aufs issues in Ubuntu's bug tracker is short or non-existent, maybe it's just because nobody uses aufs anymore. The number of major distros that actually use aufs in one way or another is more likely to be 0 than 1.

[wiak]

I stopped using aufs in March 2019 when I first developed FirstRib distros that have always used overlay fs, and I'm not sorry for that decision
Never since thus had to worry about it; I always try to keep my computing life easy as possible...

[peebee]

o news
- linux-v6.10 is released, so does aufs6.10 branch.
source code is unchanged since last aufs6.x-rcN.
- aufs6.1..aufs6.5 are not supported anymore.

J. R. Okajima

6.10 is not LTS so it won't be added to Github Woof-CE kernel-kit......

[nocsak]

Hi All!

I've tested many variants of kernels and versions, and found 5.10.x longterm as the only cross-compatible between EFI and LEGACY boot and with support of my oldest notebook with VIA soundcard in it. It seemed to compatible both way modern hardwares and old hardwares but on the way I could only test. So my tests are limited to approx 3 machines:

A 9gen intel platformed notebook,
A 7gen intel platformed desktop,
An AMD M320 and RS880M Chipset platformed notebook.

These are my test machines. The mentioned longterm 5.10.x kernels now the 5.10.218 could be compiled with no problems at all, but 5.10.223 and now the actual 5.10.224 are get error with aufs:

Code: Select all

  CC      fs/aufs/module.o
  CC      fs/aufs/sbinfo.o
  CC      fs/aufs/super.o
  CC      fs/aufs/branch.o
  CC      fs/aufs/xino.o
  CC      fs/aufs/sysaufs.o
  CC      fs/aufs/opts.o
  CC      fs/aufs/fsctx.o
  CC      fs/aufs/wkq.o
  CC      fs/aufs/vfsub.o
fs/aufs/vfsub.c: In function ‘vfsub_rename’:
fs/aufs/vfsub.c:420:26: error: passing argument 1 of ‘vfs_rename’ from incompatible pointer type [-Werror=incompatible-pointer-types]
  420 |         err = vfs_rename(src_dir, src_dentry, dir, path->dentry,
      |                          ^~~~~~~
      |                          |
      |                          struct inode *
In file included from ./include/linux/security.h:29,
                 from fs/aufs/vfsub.c:25:
./include/linux/fs.h:1822:16: note: expected ‘struct renamedata *’ but argument is of type ‘struct inode *’
 1822 | int vfs_rename(struct renamedata *);
      |                ^~~~~~~~~~~~~~~~~~~
fs/aufs/vfsub.c:420:15: error: too many arguments to function ‘vfs_rename’
  420 |         err = vfs_rename(src_dir, src_dentry, dir, path->dentry,
      |               ^~~~~~~~~~
./include/linux/fs.h:1822:5: note: declared here
 1822 | int vfs_rename(struct renamedata *);
      |     ^~~~~~~~~~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:286: fs/aufs/vfsub.o] Error 1
make[1]: *** [scripts/Makefile.build:503: fs/aufs] Error 2
make: *** [Makefile:1832: fs] Error 2
Error: failed to compile the kernel sources.

I needed to swicth the jobs parameter to -j1 to let the error come on one way.
The questions are what causes the error and is there a way to solve that - without making unable to compile other sub variants which are worked until 223 suffix?

Is it really an aufs source problem, or maybe these version of kernels are not support aufs anymore?

Thanks for answers in advance!

---edit----

Without aufs, with only overlayfs the 224 suffix version could be compiled without any other error.
I've checked the sources in kernel-kit, manually deleted aufs sources folders in it, to let the build.sh process download from 0.
But with aufs enabled, the error still occur, from version suffix 223.

[ozsouth]

@nocsak - peebee found that a backport from kernel 5.12 killed aufs after 5.10.220. The fix is so huge, no-one is attempting it. Aufs 5.x is no longer supported. I had someone else who had no luck with 6.x, but I made a very vanilla 6.1 kernel which works on his system (in xenialpup64, bionicpup64, fossapup64). Downside is many touchpads won't work with it.
If you wish to try it (at own risk), is here: viewtopic.php?p=127947#p127947

[nocsak]

Sad news... I understand that kernel development brings with it such phenomena, some things become obsolete and others are renewed, but this has hit me hard right now.

Thanks for the quick reply! And thanks all efforts by keeping aufs development alive! Now I have to accept there is a limitation in it.

[peebee]

@nocsak

5.15 with aufs is still building OK at the moment - you could try that

https://github.com/puppylinux-woof-CE/w ... el-kit.yml
https://github.com/puppylinux-woof-CE/w ... 0397151768

https://github.com/sfjro/aufs-standalone/issues/41

[nocsak]

Thanks @peebee !

With 5.15.x kernel versions and above the legacy boot is not work somehow at me and no sound anymore on my oldest VIA soundcard Chip based machine. And that is why I tried many kernels to find a cross-compatible one. If I don't insist for 5.10.x then 5.15.x ++ are okay.

Another thing: If anyone could correct aufs for the problematic kernel versions, maybe the corrections are work for new longterms but not for olders from that point.
Am I see right?

Here are my kernel and ucode+fdrv compilations. zen kernels are / were a bit tricky to compile but possible.

[peebee]

Another thing: If anyone could correct aufs for the problematic kernel versions, maybe the corrections are work for new longterms but not for olders from that point.
Am I see right?

A version of aufs with the further patches needed for 5.10.220 would only work for kernels after 5.10.220 - just like 5.10.140 only works for kernels after 5.10.140 until 5.10.219
https://github.com/sfjro/aufs-standalon ... fs5.10.140