QuickPup64 24.01 RC 15

[wiak]

@wiak

PUPMODE 5 means the read/write layer was tmpfs only (at the ramdisk)

On initrd I mount tmpfs at /overlay then I create writable folder and working folder like this

/overlay/writable
/overlay/workdir

Then mount the overlayfs like this

Code: Select all

mount -t overlay overlay -o workdir=/overlay/workdir,upperdir=/overlay/writable,lowerdir=< sfs mountpoints here >,xino=on /pup_new

Then move the /overlay mountpoint to /pup_new/overlay

And lastly switch_root

See the init script attached

I'm just going out, but I'll look at the attached init script later. I can say that the mount line looks pretty much okay to me and you setting up the tmpfs prior to it, but the following doesn't seem right off the top of my head:

Then move the /overlay mountpoint to /pup_new/overlay

You should simply be doing something like:

Code: Select all

exec switch_root /pup_new /sbin/init

EDIT: Back from the cafe now; ignore my above comment since I see you are doing switch_root like that anyway - must be another problem that needs looked at more closely - I don't as yet see any obvious error but I haven't read your init.txt itself yet. I really need to try to boot with that, but problem I have is that the computer I use won't boot Fossapup so I should try your quickpup and see if normal aufs will boot on this particular machine of mine - not sure why it won't boot normal aufs fossapup but likely firmware/modules related

though prior to that you can mount --move parts of the pre-switch root directories over to /pup_new, such as the directory you are booting from and that you have mounted to access vmlinuz and so on.

[wiak]

Turns out that using your normal aufs-based initrd.xz, QuickPup64 does boot okay on my nvme SSD machine (unlike Fossapup...). So at least I can now experiment with your new overlayfs-based initrd/init. Whether I find anything is another matter of course - it isn't easy since I'm totally unfamiliar with Puppy-original initrd so will only be trying to see if I can see anything that might be causing the overlayfs mount to fail or whatever the Pupmode 5 issue is. I'll report back if I find anything or fail to...

[wiak]

Sorry mistfire, I took a look at init.txt, but wow that is a huge script!

I wasn't too worried by the size in terms of trying it, so I took normal quickpup initrd, uncompressed it and replaced its init with your new overlayfs init.txt (as init). What I wanted to do was try booting with that but putting in a few quick debug statements of my own around line 3750 to see what was going on at the mount -t overlay stage. However, it wouldn't boot that far for me anyway - just gave error messages about not finding various /dev/tty and just kept doing that. Dawned on my that you may be using a different kernel or root filesystem arrangement in your own tests since need overlay module loaded or in the kernel. I did try quickly copying all the zdrv modules into the initrd.xz/lib, but didn't help - as I say I don't know the operation of Puppy initrd and no time to try and work that out. That's why I've never bothered studying Puppy's way of doing it - the init script, even the traditional one, is pretty huge and I couldn't be bothered... (call me lazy...).

As for simply reading your init.txt; well, not likely to spot subtle issue doing that (hence my attempt to actually boot it with debug statements). It 'looked' like it might be okay, but as you know even one tiny detail can make an init fail on boot, so simply browsing the init.txt wasn't enough for me to find anything. The only thing I 'wonder' about is your use of umount -l (lazy umount). I'd try without these I since they may be effectively forcing a umount when that shouldn't actually be happening. For example in WDL initrd I just use plain: umount /dev

[wiak]

Sorry mistfire, I took a look at init.txt, but wow that is a huge script!

I wasn't too worried by the size in terms of trying it, so I took normal quickpup initrd, uncompressed it and replaced its init with your new overlayfs init.txt (as init). What I wanted to do was try booting with that but putting in a few quick debug statements of my own around line 3750 to see what was going on at the mount -t overlay stage. However, it wouldn't boot that far for me anyway - just gave error messages about not finding various /dev/tty and just kept doing that. Dawned on my that you may be using a different kernel or root filesystem arrangement in your own tests since need overlay module loaded or in the kernel. I did try quickly copying all the zdrv modules into the initrd.xz/lib, but didn't help - as I say I don't know the operation of Puppy initrd and no time to try and work that out. That's why I've never bothered studying Puppy's way of doing it - the init script, even the traditional one, is pretty huge and I couldn't be bothered... (call me lazy...). My brain can handle reading WeeDog initrd/init because it isn't terribly long (internal initrd/init being 231 lines and external w_init, which is the part containing most complex logic and the overlay stuff, 439 lines, of code; more than that and my eyes begin to water... ).

As for simply reading your init.txt; well, not likely to spot subtle issue doing that (hence my attempt to actually boot it with debug statements). It 'looked' like it might be okay, but as you know even one tiny detail can make an init fail on boot, so simply browsing the init.txt wasn't enough for me to find anything. The only thing I 'wonder' about is your use of umount -l (lazy umount). I'd try without these I since they may be effectively forcing a umount when that shouldn't actually be happening. For example in WDL initrd I just use plain: umount /dev and so on... In fact, I doubt these should be umounted at all (and rather feel that the umount fails, but that the initrd switch_root is happy about that - so it is my only thought: maybe don't use -l with your umount statements? Just an unlikely guess sorry.

[dimkr]

On my latest experiment I successfully boot QuickPup in overlayfs using modified initrd init script. However only works if a save file is loaded. It will threw WRONG FILESYSTEM error message upon switch_root if no savefile was loaded. I need some clues on how to mount overlayfs on PUPMODE=5

woof-CE supports overlay as an alternative to aufs, just set UNIONFS=overlay in _00build.conf and use a kernel where CONFIG_OVERLAY_FS=y.

(It uses bind mounts instead of mount --move.)

[peebee]

woof-CE supports overlay as an alternative to aufs, just set UNIONFS=overlay in _00build.conf and use a kernel where CONFIG_OVERLAY_FS=y.

(It uses bind mounts instead of mount --move.)

I think there must be more to it than just that..........

I did a test build of VoidPup32 with UNIONFS=overlay but no other changes to the build definition

It booted fine but sfs handling was problematic (to say the least):
- right clicking an sfs and selecting install just mounted it
- right clicking and unmounting the mount gave errors
- sfs-load from the menu refused to detect any sfs to load no matter where I placed it

Screenshot.png (156.89 KiB) Viewed 10215 times

[dimkr]

I think there must be more to it than just that..........

overlay doesn't support dynamic addition or removal of layers (so a SFS cannot be added to the stack, below the upper layer) and modification of lower layers (so a SFS cannot be extracted to a lower tmpfs layer). Dynamic SFS loading (as opposed to, boot-time loading) is unsupported, until someone comes up with a safe solution.

[peebee]

I think there must be more to it than just that..........

overlay doesn't support dynamic addition or removal of layers (so a SFS cannot be added to the stack, below the upper layer) and modification of lower layers (so a SFS cannot be extracted to a lower tmpfs layer). Dynamic SFS loading (as opposed to, boot-time loading) is unsupported, until someone comes up with a safe solution.

I fully understand that........ but an overlayfs build shows anomalous behaviour as detailed above.

[mistfire]

@dimkr, @wiak, @peebee

Finally I able to boot on PUPMODE 5 via overlayfs. I found the root cause of the problem. I forgot to add mkdir a mountpoint before creating tmpfs. I will now try to refine sfs_load scripts

Odd numbered PUPMODE on overlayfs was seems feasible, instead of using mv command from upper to lower. Use cp instead from upper to lower directory after copying delete the file on the upper directory. After doing that run this command to refresh the merge overlayfs

Code: Select all

mount -remount /

I found that command somewhere on ubuntu forums. I'm not sure if that trick works

By the way here is the revised init that I made for your reference. This init script was still using 2013 revision but almost based from the upstream

[dimkr]

Odd numbered PUPMODE on overlayfs was seems feasible, instead of using mv command from upper to lower. Use cp instead from upper to lower directory after copying delete the file on the upper directory.

Yep, that's what snapmergepuppy.overlay does, but without the deletion part. If you delete a file from the save layer, you might run into undefined behavior due to:
1. Caching: i.e. cat /path/to/file will return the same output even after the file is deleted from the save layer, because it's cached, so it won't show the file from a lower read-only layer. This is one of the reason why overlay documentation says that behavior is "undefined" if you mess with the layers while overlay is mounted.
2. Race conditions: for example, if cp reads the file while a process is writing to it, the result of the copying is garbled; that's why https://github.com/puppylinux-woof-CE/w ... ssues/3187 happens with aufs. It's super easy to corrupt browser settings and history while snapmergepuppy is running, unless you close the browser. Therefore, snapmergepuppy.overlay copies files without deleting them from pup_rw. In addition, this makes everything faster (at the expense of RAM), because files modified since boot time still sit in RAM and not on a slow disk.

Code: Select all

mount -remount /

AFAIK this doesn't do anything.

[mistfire]

AFAIK this doesn't do anything.

I made a test about it. I setup an ovelayfs mountpoint. I copied a file from upper to lower directory then delete the file on upper directory. When file was still shown but when I try to delete the file on overlayfs mountpoint. It triggers file stale error. When I issued remount, it deletes the file on overlayfs mount when I try to delete it.

[wiak]

@dimkr, @wiak, @peebee

Finally I able to boot on PUPMODE 5 via overlayfs. I found the root cause of the problem. I forgot to add mkdir a mountpoint before creating tmpfs.

Good you found that mistfire. I didn't look closely enough to notice you hadn't made a mointpoint for that, but naturally you need it to be one when making it type tmpfs.

Mind you, I thought that's what you were doing here:

Code: Select all

mount -t tmpfs -o size=${ALLOCK}k tmpfs /overlay

Ah, but I see now, you put the mkdir -p after the mount -t tmpfs:

Code: Select all

mkdir -p /overlay/writable

and so on...

At least, I guess that's what you are talking about.

[mistfire]

Good you found that mistfire. I didn't look closely enough to notice you hadn't made a mointpoint for that, but naturally you need it to be one when making it type tmpfs.

I found out that error by running mount command, show the result /dev/console, and put sleep to 30 seconds right after overlayfs was mounted. mount command showed that there is no /overlay mountpoint shown when I test the init script

[Ramachandra Iyer]

Dear Mist

I strongly believe that the culprit is not Kernel alone. I have tested almost all mainline puppy and easyos. However none of them booting from nvme. All of them designed for legacy or ancient systems. Why Os looking for usb drive or slow storage option while booting from ssd. Mounting issues like etc/mount etc.
Considering your advice definitely I will try another attempt to boot after changing the kernel with zdrv. Any particular kernel versions?.

[mistfire]

Dear Mist

I strongly believe that the culprit is not Kernel alone. I have tested almost all mainline puppy and easyos. However none of them booting from nvme. All of them designed for legacy or ancient systems. Why Os looking for usb drive or slow storage option while booting from ssd. Mounting issues like etc/mount etc.
Considering your advice definitely I will try another attempt to boot after changing the kernel with zdrv. Any particular kernel versions?.

Try the latest puppy linux kernel version. Also did you boot puppy without pfix boot parameter?

[mistfire]

On my latest experiment of QuickPup I successfully boot PUPMODE 13 using overlayfs, and yes, it can move files from upperdir to lowerdir. However it requires some refinements but still usable.

[wiak]

using overlayfs, and yes, it can move files from upperdir to lowerdir. However it requires some refinements but still usable.

Could you give example or examples of what you mean by above comment please? My understanding from kernel docs is that effect of changing other than the top rw layer is undefined, or are you talking about moving files followed by a reboot, which is a different matter?

[Ramachandra Iyer]

Dear Mist

I strongly believe that the culprit is not Kernel alone. I have tested almost all mainline puppy and easyos. However none of them booting from nvme. All of them designed for legacy or ancient systems. Why Os looking for usb drive or slow storage option while booting from ssd. Mounting issues like etc/mount etc.
Considering your advice definitely I will try another attempt to boot after changing the kernel with zdrv. Any particular kernel versions?.

Try the latest puppy linux kernel version. Also did you boot puppy without pfix boot parameter?

Tested with new kernel. Same error. I would like to test "full install". Is it possible.

[mistfire]

Tested with new kernel. Same error. I would like to test "full install". Is it possible.

Yes it is. Go to Puppy Universal Installer. You will see full install option there.

[mistfire]

using overlayfs, and yes, it can move files from upperdir to lowerdir. However it requires some refinements but still usable.

Could you give example or examples of what you mean by above comment please? My understanding from kernel docs is that effect of changing other than the top rw layer is undefined, or are you talking about moving files followed by a reboot, which is a different matter?

In Puppy, PUPMODE=13 was using ramdisk as write layer while save file/folder was writable but lower layer. When ramdisk gets full, puppy can move the files from ramdisk to save file/folder to free up some ramdisk space while the files were visible to rootfs. In aufs it's possible to do that, but in overlayfs it's possible but it requires to refresh the rootfs by issuing "mount -o remount /"

The trick that I made was copy files from ramdisk to save file/folder first. If the copy was successful, the file on the ramdisk will be deleted. When all moving processes were completed. the rootfs will be remount to refresh rootfs

[wiak]

When ramdisk gets full, puppy can move the files from ramdisk to save file/folder to free up some ramdisk space while the files were visible to rootfs. In aufs it's possible to do that, but in overlayfs it's possible but it requires to refresh the rootfs by issuing "mount -o remount /"

Oh... that's very interesting. Yes I know it is done by Puppy when using aufs. I didn't think it could be done using overlayfs (I've tried before, but never did that "mount -o remount /"). Great that worked since as you say is the difference is in the RAM that would be saved. The question is whether that mechanism will always work - as others (and myself) have said on here, it is considered "undefined" what happens with overlayfs when moving files into a lower directory when using overlayfs - that would imply it "could" work but isn't guaranteed. However, you are remounting, which could be considered as re-starting the overlay mount - and then I can see it might be okay since it worked. If so, that is a great find mistfire! The question hanging in my mind is whether an sfs could be loaded to a lower layer and that same mount command used to remount the layers as if done for the first time? I suppose that is unlikely since it would be a completely new overlay structure and not simply a remount of existing one?

I will try that mount remount of / trick to see at least if it works for me. Good thing about it is that is done after the switch_root so not going to effect the initrd itself. In WeeDog it would be used in what is referred to as RAM2 mode - in that mode the save device is mounted rw but to a read-only layer in same fashion you describe.

[wiak]

When ramdisk gets full, puppy can move the files from ramdisk to save file/folder to free up some ramdisk space while the files were visible to rootfs. In aufs it's possible to do that, but in overlayfs it's possible but it requires to refresh the rootfs by issuing "mount -o remount /"

Oh... that's very interesting. Yes I know it is done by Puppy when using aufs. I didn't think it could be done using overlayfs (I've tried before, but never did that "mount -o remount /"). Great that worked since as you say is the difference is in the RAM that would be saved. The question is whether that mechanism will always work - as others (and myself) have said on here, it is considered "undefined" what happens with overlayfs when moving files into a lower directory when using overlayfs - that would imply it "could" work but isn't guaranteed. However, you are remounting, which could be considered as re-starting the overlay mount - and then I can see it might be okay since it worked. If so, that is a great find mistfire! The question hanging in my mind is whether an sfs could be loaded to a lower layer and that same mount command used to remount the layers as if done for the first time? I suppose that is unlikely since it would be a completely new overlay structure and not simply a remount of existing one?

I will try that mount remount of / trick to see at least if it works for me. Good thing about it is that is done after the switch_root so not going to effect the initrd itself. In WeeDog it would be used in what is referred to as RAM2 mode - in that mode the save device is mounted rw but to a read-only layer in same fashion you describe.

Unfortunately, I have not been able to confirm what you say above regarding using this method for overlayfs, mistfire. I would be grateful if you could explain it in more detail. If correct, it would be a very important result, but didn't work for me when tried with KLV-Airedale. Using KLV, I mount (rw) the existing save folder to a mountpoint /mnt/layers/uc_ro (called ro because used in overlayfs lower layer, but in practice it is mounted rw for a utility script to rsync back changes via save2flash). I then make a change (such as creating a new empty file via touch /root/newfile). I then saved that back to the save folder via save2flash utility in KLV-Airedale; that does successfully copy back all changes via rsync routine. The actual top overlayfs layer in KLV-Airedale is at /mnt/layers/RAM/upper_changes. Following what you said, I then tried deleting the contents of /mnt/layers/RAM/upper_changes (I've tried this in the past too, but without your remount command thereafter). I then tried: mount -o remount /,
but that did not restore the up-to-date overlayfs layer structure - i.e. it didn't work.

[mistfire]

When ramdisk gets full, puppy can move the files from ramdisk to save file/folder to free up some ramdisk space while the files were visible to rootfs. In aufs it's possible to do that, but in overlayfs it's possible but it requires to refresh the rootfs by issuing "mount -o remount /"

Oh... that's very interesting. Yes I know it is done by Puppy when using aufs. I didn't think it could be done using overlayfs (I've tried before, but never did that "mount -o remount /"). Great that worked since as you say is the difference is in the RAM that would be saved. The question is whether that mechanism will always work - as others (and myself) have said on here, it is considered "undefined" what happens with overlayfs when moving files into a lower directory when using overlayfs - that would imply it "could" work but isn't guaranteed. However, you are remounting, which could be considered as re-starting the overlay mount - and then I can see it might be okay since it worked. If so, that is a great find mistfire! The question hanging in my mind is whether an sfs could be loaded to a lower layer and that same mount command used to remount the layers as if done for the first time? I suppose that is unlikely since it would be a completely new overlay structure and not simply a remount of existing one?

I will try that mount remount of / trick to see at least if it works for me. Good thing about it is that is done after the switch_root so not going to effect the initrd itself. In WeeDog it would be used in what is referred to as RAM2 mode - in that mode the save device is mounted rw but to a read-only layer in same fashion you describe.

Unfortunately, I have not been able to confirm what you say above regarding using this method for overlayfs, mistfire. I would be grateful if you could explain it in more detail. If correct, it would be a very important result, but didn't work for me when tried with KLV-Airedale. Using KLV, I mount (rw) the existing save folder to a mountpoint /mnt/layers/uc_ro (called ro because used in overlayfs lower layer, but in practice it is mounted rw for a utility script to rsync back changes via save2flash). I then make a change (such as creating a new empty file via touch /root/newfile). I then saved that back to the save folder via save2flash utility in KLV-Airedale; that does successfully copy back all changes via rsync routine. The actual top overlayfs layer in KLV-Airedale is at /mnt/layers/RAM/upper_changes. Following what you said, I then tried deleting the contents of /mnt/layers/RAM/upper_changes (I've tried this in the past too, but without your remount command thereafter). I then tried: mount -o remount /,
but that did not restore the up-to-date overlayfs layer structure - i.e. it didn't work.

Can you list the folder structure you make?

I wonder what is your overlayfs command did you execute?

[wiak]

Can you list the folder structure you make?

I wonder what is your overlayfs command did you execute?

Structure and overlayfs command and mount details attached.

If you want to try, you could just boot KLV-Airedale64 distro with following grub2 stanza (use your own uuid and so on of course):

menuentry "WDL_klv ext4" {
insmod ext2
search --no-floppy --fs-uuid --set 424d8f42-e835-4111-9053-dd086b3d38e8
linux /WDL_klv/vmlinuz w_bootfrom=UUID=424d8f42-e835-4111-9053-dd086b3d38e8=/WDL_klv w_changes=RAM2
initrd /WDL_klv/initrd.gz
}

Note that w_changes=RAM2 mode mounts the savefolder rw to /mnt/layers/uc_ro mountpoint. Thereafter, session changes are held in /mnt/layers/RAM/upper_changes (which is tmpfs) and then can be saved back anytime using save2flash command (which uses rsync to copy everything back).

Having done the above I tried as you seemed to suggest: deleting contents of /mnt/layers/upper_changes

and then running command: mount -o remount /

but the system did not update with the changes that had been saved back at that stage. Of course, on reboot the saved changes are there, but that is how it always has been.

Anyway, once you release your overlayfs version of Quickpup that does that in RAM deleting after the save back I'll check it out to see what the difference is.

[mistfire]

Having done the above I tried as you seemed to suggest: deleting contents of /mnt/layers/upper_changes

and then running command: mount -o remount /

The correct command if you are testing overlayfs was

Code: Select all

mount -o remount <overlayfs merged mountpoint>

In that case

mount -o remount /mnt/layers/merged

Sorry for my mistake.

[mistfire]

QuickPup64 22.01 BETA 23 released

Changes:
* Linux kernel 5.19.4 kernel
* Overlayfs support (experimental, to use overlayfs use punionfs=overlay boot parameter. Use under your own risk)
* Updated glibc
* Added ibus and seahorse password manager

Download: https://drive.google.com/file/d/1Hey-A2 ... pafjbpzTYe
MD5 Checksum: 7a1c09eb4fc8410d7e1b0d239756a246

[wiak]

Having done the above I tried as you seemed to suggest: deleting contents of /mnt/layers/upper_changes

and then running command: mount -o remount /

The correct command if you are testing overlayfs was

Code: Select all

mount -o remount <overlayfs merged mountpoint>

In that case

mount -o remount /mnt/layers/merged

Sorry for my mistake.

Ah okay, that makes sense. I will try. Really hope this works. It is a great idea.

EDIT: tried that, still didn't work for me. Don't worry, I will download your quickpup overlayfs initrd version and puzzle out what the difference is. Since it works for you, it can work for me! and that will be excellent discovery. Actually, I suspect now that I know why it is not working for me... in my initrd I move (--move) the merged mount and so likely not allowing remount; I will play with that arrangement until it allows the remount since that is certainly the key - of course I had a reason for the --move and not having it will likely give me another problem to solve, but I do want that flush mechanism so will have to do it...

[dimkr]

Does this remount trick work for existing processes? When you mount something at / (say, mount -t tmpfs whatever /), existing processes still see the same mount as before, as their /. If the browser keeps writing to the same (wrong) file, the next "downwards" copy from the top layer won't copy the right file.

(By the way, have you checked dmesg? overlay warns when you modify files in lower layers and trigger undefined behavior)

EDIT: also, it's pretty easy to run into inconsistency if you mess with lower layers, unless you echo 3 > /proc/sys/vm/drop_caches. For example, if you modify a file that exists in a lower layer, then delete the copy from the upper layer and cat the file, you might receive the modified file's contents and not the unmodified file from the lower layer (because the contents are cached).

[wiak]

Well, I have had some limited 'success' - something definitely happens (when trying same mount remount 'trick' after booting with WDL initrd). I'm still not sure if this is safe, reliable, or even if it is working same for me. But what I did was copied a big 500MB file into /root whilst using upper_changes in RAM and then did a save2flash and then deleted the copy of the 500MB file that was showing of course in RAM/upper_changes/root (the save folder in RAM I use). Then I tried 'free' command and noted the values, and then it actually was the command:

mount -o remount /

that had an effect.

I ran 'free' command and there was indeed now around 500MB more RAM free...

I am tired tonight so my brain is a bit dazed and so not good time for me to test further, to see what works and what doesn't. I did notice I can't simply delete everything from RAM/upper_changes - I only tried deleting the big file... Deleting whole of RAM/upper_changes/* stopped my system working properly - hard to explain (more than likely to be expected?) since I haven't explained any of it to my own brain yet...

If the technique proves safe and reliable overall then it would certainly be a great discovery that makes me wonder if mount -o remount / could help with sfs load issues - seems unlikely, but even this much I wasn't expecting as possible. Or is it just 'undefined' so best not to go there?!!! Hurts my brain testing this one. I will test on quickpup I think ... no idea how to test how reliable this method is - my gut feeling is that it is risky and asking for trouble! ... but interesting nevertheless.

[wiak]

QuickPup64 22.01 BETA 23 released

Changes:
* Linux kernel 5.19.4 kernel
* Overlayfs support (experimental, to use overlayfs use punionfs=overlay boot parameter. Use under your own risk)

I'm trying it with overlayfs. First boot into Pupmode 5 was fine, but on reboot with savefolder created it said couldn't mount into pup_new and wouldn't boot past that.