aufs future

ozsouth · Post by **ozsouth** » Fri Aug 28, 2020 2:54 am

Our aufs man sfjro has been silent for 3 months. Kernel 5.7 is near eol, 5.8 & LTS 5.9 rc are out already, but no aufs updates. Hope he is OK - covid19 cases have spiked in Japan in August.

garnet · Post by **garnet** » Fri Aug 28, 2020 5:23 am

Scary isn't it? ( : ౦ ‸ ౦ : )
Let's write an email to him and ask him how is he doing?

peebee · Post by **peebee** » Fri Aug 28, 2020 6:07 am

https://sourceforge.net/p/aufs/mailman/ ... /37050373/

From: J. R. Okajima <hooanon05g@gm...> - 2020-07-01 01:59:39
Hello aufs-users,

I was going to start supporting linux-v5.8-rcN, but unfortunately I have
to suspend it due to some personal issues. For something new in aufs
development, I set its priority lower in my todo list. But bugfix,
supporting users, such like answering the questions, etc. are still have
higher priority.
I shall return to aufs5.x-rcN in a few months.

Thanx for your patience.
J. R. Okajima

This patch:
https://u.pcloud.link/publink/show?code ... AqChaW2XoV
means 5.8.x builds ok with aufs-5.x-rcN

LateAdopter · Post by **LateAdopter** » Mon Sep 21, 2020 12:01 pm

Hello peebee
I tried the modified aufs base patch from the link you posted. It works OK and I compiled 5.8.10. I'm using that now.
It was a bit tricky slipstreaming it into the kernel kit build process. In the past the script checked the date and only fetched the aufs source once per day, but the version that I'm using now fetches it every time you run it.

I ran the build script until it failed, commented out the section that finally fetches aufs_sources, copied over the new aufs base patch and ran the script again. That worked correctly.

Thanks for the link.

peebee · Post by **peebee** » Thu Sep 24, 2020 6:04 am

Unfortunately, the situation is unchanged from
https://www.mail-archive.com/aufs-users ... 05822.html

J. R. Okajima

https://github.com/sfjro/aufs5-standalo ... -698025107

Duprate · Post by **Duprate** » Sun Sep 27, 2020 6:52 pm

One question: is there no other alternative to aufs that is suitable for the purposes of Puppy Linux and FatDog, and that causes less dependency?

peebee · Post by **peebee** » Sun Sep 27, 2020 9:33 pm

Duprate wrote: Sun Sep 27, 2020 6:52 pm One question: is there no other alternative to aufs that is suitable for the purposes of Puppy Linux and FatDog, and that causes less dependency?

Yes - overlayfs - https://en.wikipedia.org/wiki/OverlayFS
@gyro has done a lot of work - http://www.murga-linux.com/puppy/viewtopic.php?t=116059
but....
it doesn't work quite the same way as aufs, so puppy would be "different" if the changeover had to occur.

user1111 · Post by **user1111** » Mon Oct 05, 2020 2:49 pm

gyro is using tar files and exclude lists ... http://www.murga-linux.com/puppy/viewto ... 91#1031191

1. Files can be "removed" by simply forgetting to "save" them.
For the save layer:
On shutdown a tar archive of the current RW directory is written, but specifying an exclude file that contains a list of all files that need to be "removed".
In 'init' the RW directory is clobbered with this tar archive, and so the new stack is built without the "removed" files.

A variation might be to use hard links rsync's. Very fast to create snapshots and restore those snapshots, provided within the same filesystem, which if you use a file filesystems for the main sfs ... it is.

I run fatdog, so rather than puppyxxx.sfs its fd64.sfs that contains the main system. I unsquashed that 450MB sfs and ran du -sh * that indicated 1.3GB size for squashfs-root (extracted content), so I opted for a 1.5GB file filesystem

dd if=/dev/zero of=fd64fs.ext3 bs=4096 count=393296

and ls -l indicated 1610612736 filesize

Formatted that to ext3
mkfs.ext3 fd64fs.ext3

mounted it

mkdir m
mount fd64fs.ext3 m

unsquashed fd64.sfs into it using the force (so extracts even though folder already exists)

unsquashfs -f -d ./m fd64.sfs

and should have created a hard link rsync of that (and obviously excluding the hardlinks folder from that), but I didn't

df -h indicated m filesystem has 128MB available free space

# tidy up
umount m
rm -rf squashfs-root
rmdir m

Booted initrd, mounted that file filesystem and I used chroot (instead of the more usual switch root) into that, and it started up OK.

NEW_ROOT=/m
mount -o move /sys $NEW_ROOT/sys
mount -o move /dev $NEW_ROOT/dev
mount -o move /proc $NEW_ROOT/proc
exec /sbin/chroot $NEW_ROOT /sbin/init

After using it a while obviously the content of the file filesystem was changed, but had I taken a hardlinks rsync snapshot at the point indicated above, you could at any time roll back to that.

Fundamentally hard links take up little space (but can consume a lot of inodes), involve no actual data copying, just filename pointers, and runs through very quickly. So for instance to shutdown without saving any changes involves rm -rf of the main copy, rename the hard links copy to the main copy, and make another hard link copy of that. Or to shutdown and save changes, rm the hard link copy and make another hard link copy. Either way that's very quick. Obviously you have to exclude the hard links top level folder from the rsync.

Conceptually you could maintain a number of hardlink sets, such that a sfs could be loaded whilst using the system by making a hardlink copy (snapshot), extract the sfs into the main system, and then later to 'unload' that sfs you just restore the 'snapshot'. Same as how a sfs can be 'loaded' into a full install simply by extracting it, but where the hardlinks rsync provides the capacity to 'undo' those changes. Could be fiddly for when multiple sfs's had been loaded but a record of the order of loading might enable you to roll-back to immediately prior to the sfs that was to be unloaded occurred, and then roll forward again (reload) the other sfs's (again) on top of that (for the sfs's that were loaded after the now removed sfs).

No aufs required, and all using standard tools. But restricted to having to use file filesystems (and involving non-compressed storage space), a.k.a savefile type saving (save folder could also be used). Due to the larger (non compressed) sizes involved however (1.5GB instead of 500MB for instance in the case of Fatdog) that's more prohibitive to being loaded into ram - or at least that is the case until 4GB+ of ram pretty much becomes the 'norm'.

peebee · Post by **peebee** » Mon Oct 12, 2020 6:14 am

Kernel 5.9 has been released (not yet identified as LTS)
https://www.kernel.org/

These VERY UNOFFICIAL aufs changed files allow 5.9 to be built apparently successfully...
https://u.pcloud.link/publink/show?code ... 2kPYwrBWQk

Running with it now:
Kernel Release: 5.9.0-lxpup64
Build Date: Mon Oct 12 08:54:17 BST 2020
...

: Screenshot.png (88.44 KiB) Viewed 6401 times

LateAdopter · Post by **LateAdopter** » Mon Oct 12, 2020 9:31 am

Hello peebee
It looks like AUFS is in-tree for debian and ubuntu kernel git, so presumably the code gets updated automatically when it is rebased on mainline. There doesn't seem to be any noticeable delay since they built 5.8 in July.

Have you ever tried building a Puppy kernel from the Debian or Ubuntu kernel git rather than mainline? At least you know it's been properly tested.

peebee · Post by **peebee** » Mon Oct 12, 2020 9:14 pm

LateAdopter wrote: Mon Oct 12, 2020 9:31 am Hello peebee
It looks like AUFS is in-tree for debian and ubuntu kernel git, so presumably the code gets updated automatically when it is rebased on mainline. There doesn't seem to be any noticeable delay since they built 5.8 in July.

Have you ever tried building a Puppy kernel from the Debian or Ubuntu kernel git rather than mainline? At least you know it's been properly tested.

No...........
Can't see how they can have any later version than the AUFS main git???
And do their kernels include AUFS? I can't see why they should as they don't need layering? and they'd be more likely to use overlayfs if they did....
Do you have links?

LateAdopter · Post by **LateAdopter** » Tue Oct 13, 2020 1:16 pm

Hello peebee
Debian/Ubuntu fork the software they use and maintain it themselves. That includes the kernel. They can cherrypick commits and add their own.
Here are some links for Ubuntu:
https://kernel.ubuntu.com/git/ubuntu/unstable.git/
If you look in the master-5.8 tree it has root/fs/aufs and was finished 27th July
If you look in master it doesn't have aufs.

https://git.launchpad.net/~ubuntu-kerne ... aster-next
If you look here the latest 5.8 kernel for groovy has aufs

https://kernel.ubuntu.com/~kernel-ppa/config/
If you look at typical configs they have aufs=m

But Ubuntu don't do rolling release so they are unlikely to do a 5.9 branch.

Debian do rolling release for sid which is currently equivalent of about 5.8.14 and they are likely to do a 5.9. I can't make sense of their repository that uses gitlab. I think their kernels have aufs too, but I can't prove it. As far as I can decode it, they removed aufs at 5.9-rc4 and haven't put it back yet. Their source is currently at 5.9-rc8, I think.
I expect they will update to 5.9 stable soon.

user1111 · Post by **user1111** » Tue Oct 13, 2020 3:21 pm

https://packages.debian.org/stretch/aufs-dkms

Duprate · Post by **Duprate** » Wed Oct 14, 2020 11:44 pm

"Peebee wrote:
Kernel 5.9 has been released (not yet identified as LTS)
These altered VERY UNOFFICIAL files allow 5.9 to be apparently built successfully ...
Running with it now: Launch of
kernel: 5.9.0 -lxpup64
Compilation date: Mon 12 October 08:54:17 BST 2020 "

I'm testing your kernel on FatDog64 811 ... it works ... Firefox, OK ... Chromium, Chrome, Vivaldi, crash ... Wine, OK ... Wine + games, crash ... Unfortunately, for my knowledge, I can only test ...

Duprate · Post by **Duprate** » Sun Oct 18, 2020 12:59 am

Complementing the tests of the Kernel 5.9.0-lxpup64, which worked well: LibreOffice 6.3.3, Gimp 2.10.7, Firefox 81.0.2, Seamonkey 2.53.3, ClamAV-0.103.0, all of the ROXapps type;
Wine-3.3-V.2.1 + 32bit-fd64_811.sfs, Windows AppsPortables (The system freezes on heavier games).
What does not work (freezing) everything that is based on Chrome.

It's not a lack of memory, the PC has 8 GB. For an experimental kernel, the result is great! I like the peebee works. I am always using and testing your kernels.

peebee · Post by **peebee** » Sun Oct 18, 2020 9:48 am

Duprate wrote: Sun Oct 18, 2020 12:59 am What does not work (freezing) everything that is based on Chrome.

No problems here with both chrome64 and chromium64 on fossapup64 with kernel 5.9.1
...

: screenshot1.png (247.28 KiB) Viewed 5805 times

...

: screenshot2.png (248.57 KiB) Viewed 5805 times

Duprate · Post by **Duprate** » Sun Oct 18, 2020 5:34 pm

It's okay if it doesn't work perfectly on FatDog64. Kernel-5.9.1-lxpup64, you did it for LxPupSc and, without the support of the developer, J. R. Okajima. We live in a difficult time. You did a great job!
I also tested the 5.9.1 kernel with Debian live 10.6, and the result was similar to FatDog64. In three attempts, in one Vivaldi took but it worked.
I will also test with FossaPup64 ...

Post by **fredx181** » Fri Oct 23, 2020 1:12 pm

peebee wrote: Mon Oct 12, 2020 6:14 am Kernel 5.9 has been released (not yet identified as LTS)
https://www.kernel.org/

These VERY UNOFFICIAL aufs changed files allow 5.9 to be built apparently successfully...
https://u.pcloud.link/publink/show?code ... 2kPYwrBWQk

Running with it now:
Kernel Release: 5.9.0-lxpup64
Build Date: Mon Oct 12 08:54:17 BST 2020
...

Thanks a lot for the changed files, successfully built kernel 5.9.1 with working aufs for DebianDog.
Normally I use existing Debian kernel (linux-image-*), build single aufs module (using dkms) and add it to the kernel, but for 5.9 it doesn't work anymore because Debian didn't apply the aufs patches for it (probably due to the on hold development of aufs, as they wouldn't work anyway), so had to build the kernel myself (which was a good learning experience for me

).
Shared the 5.9.1 kernel "packages" here: viewtopic.php?p=8266#p8266

Fred

peebee · Post by **peebee** » Fri Nov 27, 2020 11:16 am

5.10 assistance required........

https://www.zdnet.com/article/linux-5-1 ... rity-bugs/ : "Torvalds drew attention to the removal of an addressing tool, called set_fs()"

this causes problems in https://github.com/sfjro/aufs5-standalo ... ufs/xino.c
e.g. "fs/aufs/xino.c:894:2: error: implicit declaration of function ‘set_fs’; did you mean ‘sget_fc’?"

Is there a simple replacement for set_fs and get_fs in xino.c ??

https://github.com/sfjro/aufs5-standalo ... -734782410

Post by **fredx181** » Fri Nov 27, 2020 7:17 pm

Looks like aufs development will continue (hopefully soon):
https://github.com/sfjro/aufs5-standalo ... -734830521

peebee · Post by **peebee** » Thu Dec 03, 2020 1:54 pm

Good news - an unofficial aufs patch for the next LTS Kernel 5.10 has been produced by J. R. Okajima....
see:
https://github.com/sfjro/aufs5-standalo ... -738004389

Post by **fredx181** » Thu Dec 03, 2020 3:33 pm

Thanks peebee ! That's great.

Duprate · Post by **Duprate** » Thu Dec 03, 2020 9:42 pm

Hi! It is certainly good news! We have used aufs for years, it is great, we got used to the facilities that aufs provides. Mr. J.R.Okajima, has been striving to maintain (alone?) this project for all these years and in return for his effort, Linus Torvalds refused to include it in the Kernel.
So, the future of aufs, may just be an over-life.
Overlayfs has been part of the kernel since version 3.18 (Debian Stretch). I'm using DebianDog Sid from fredx181, which uses Overlayfs and I'm really enjoying it. A very complete system, with the facilities that are the traditional mark of Puppy systems. I can compile the kernel from official sources (Kernel.org).
However, to compile a kernel with aufs support, I depend on a patch.
Okay, the aufs-5.10-changed-files.tar.xz patch has been released. What do I do with the pach?
I'm sure that other users like me also want the freedom to compile their own kernel and not just depend on the distro repository, which is not always up to date. Don't be offended, whoever uses Linux has the desire to learn ...
If we didn't want to evolve, we would use Windows ...

We should continue to use aufs but also give Overlayfs a chance!

(Sorry for any misunderstanding, I'm using Google Translate)

ozsouth · Post by **ozsouth** » Sat Dec 05, 2020 1:02 am

I think sfjro would be better off forgetting interim kernels (like 5.5 thru 5.9) and just concentrating on LTS (like 5.4 & 5.10).
Would save a lot of time & effort on his part & not interfere too much with puppy's progress.

peebee · Post by **peebee** » Tue Dec 22, 2020 4:39 pm

Good news........

Hello all,

I am going back to aufs development, and a new release will come early
in next year (aufs5.{8,9,10}). Also I am going to stop supporting
aufs4.19--aufs5.3 branches.
My new development base version is aufs5.4.

J. R. Okajima

peebee · Post by **peebee** » Sun Jan 03, 2021 6:52 pm

Subject aufs5 GIT release (v5.{8,9,10})
From J. R. Okajima
Date Today 16:21
I'm back.
Thanks for your patience and sorry for you inconvenient.
Also I appriciate PB who has made his aufs patch for v5.{8,9,10}.
I hope I can keep my development pace as I had.

o news
- As previously announced, aufs4.19--aufs5.3 has ended.
The new development base is aufs5.4.3. Not aufs5.4, because NFSv3 ACL
is broken in linux-v5.4. (aufs5.4 is still supported)
- Since v5.8, proc_mounts.patch in aufs5-standalone.git became unnecessary.
- linux-v5.10 has removed an internal function 'set_fs()' (on some
architectures), and aufs had to follow that change. Also aufs removed
its f_op->read() and ->write() functions.

J. R. Okajima

----------------------------------------
- aufs5-linux.git#aufs5.4..aufs5.7
aufs: minor, pseudo keyword 'fallthrough'
aufs: minor, MAINTAINERS
aufs: cosmetic, fix a comment

- aufs5-linux.git#aufs5.8
Addition to above,
aufs: for v5.8-rc1, a_ops->readahead

- aufs5-linux.git#aufs5.9
Addition to above,
aufs: for linux-v5.9-rc1, arg of ->handle_event()

- aufs5-linux.git#aufs5.x-rcN
Addition to above,
aufs: linux-v5.10-rc1, no more f_op->read() and ->write()
aufs: linux-v5.10-rc1, no more set_fs()
for aufs: linux-v5.10-rc1, no more vfs_(read|write)f_t

- aufs5-standalone.git
Addition to above,
aufs: unnecessary proc_mounts.patch for linux-v5.8 and later

- aufs-util.git#aufs5.8..aufs5.x-rcN
Revert "minor, make-var ProcMounts_Times"
Revert "minor, check AUFS_SUPER_MAGIC"
Revert "au_proc_getmntent: optimize reading, retry"

LateAdopter · Post by **LateAdopter** » Mon Jan 04, 2021 9:33 am

Hello peebee
Thanks for tracking this...
The version of kernel-kit that I use needs the new aufs-util 5.8 branch added to funcs.sh manually. I don't know scripting so I can't say whether the new 5.x-rcN would also work, but I'm unlikely to need that.

peebee · Post by **peebee** » Mon Jan 04, 2021 9:49 am

LateAdopter wrote: Mon Jan 04, 2021 9:33 am
The version of kernel-kit that I use needs the new aufs-util 5.8 branch added to funcs.sh manually.

Thanks - I've added aufs-util 5.8 to Woof-CE.

ozsouth · Post by **ozsouth** » Mon Jan 04, 2021 10:45 am

Thanks Peebee - have now built 5.8.18 64bit - works well - new AMD drivers help with my e2-9000e pc.

peebee · Post by **peebee** » Tue Jan 05, 2021 3:31 pm

Hopefully temporary problem......

I have raised a github issue as the patch files are missing in the 5.10 branch:
https://github.com/sfjro/aufs5-standalone/issues/3

Puppy Linux Discussion Forum

aufs future

aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future

Re: aufs future