using snapmergepuppy with FirstRib

[williwaw]

I booted with a an empty upper_changes, then used xpbs to install a new app,
I ran snapmergepuppy and then renamed upper_changes and created a new upper_changes

I expected to be able to install a second app and save it to the new upper_changes. snapmergepuppy did not make the save in the new upper_changes.

however it did save both apps to the earlier renamed upper changes.
s there a way to make seorate upper_changes in a single session?

rock, I see no place to ask general questions about the workings of firstrib,

all sections in the kl subform seem to be specific to a particular KL project
Move this or delete as you see fit.

[wiak]

I booted with a an empty upper_changes, then used xpbs to install a new app,
I ran snapmergepuppy and then renamed upper_changes and created a new upper_changes

I expected to be able to install a second app and save it to the new upper_changes. snapmergepuppy did not make the save in the new upper_changes.

however it did save both apps to the earlier renamed upper changes.
s there a way to make seorate upper_changes in a single session?

rock, I see no place to ask general questions about the workings of firstrib,

all sections in the kl subform seem to be specific to a particular KL project
Move this or delete as you see fit.

Not sure of the answer to this williwaw. So many FirstRib-related creations it is hard to pin down the exact arrangement you are using above. I presume that Void Linux root filesystem? And w_changes=RAM2 on grub or limine kernel line? Is it GUI distro - normally would expect you to use save2flash script with that, though that just calls snapmergepuppy I think (I should know, but no time to recheck right now). You never need to make upper_changes folder - that always gets made automatically on boot via FR initrd. But there can only ever be one read/writable upper_changes directory (it has exactly that name and no numbers in it). Yes you can take a previous upper_changes and make it a normal read-only layer by putting a 2-digit number in front of the name (actually the overall filename is irrelevant, only the 2-digit number determines its use and layer position, and doesn't matter if a squashed filesystem or an uncompressed directory. However anything with 2-digit number isn't relevant to snapmergepuppy - its just a read-only overlay layer. So alway the folder called upper_changes is the one that allows writing session changes (save on demand) via snapmergepuppy. That should just work, and always does unless something is wrong with your system somehow. Exact details would be required to try and work out any issue.

[wiak]

In terms of what section to put help requests. Yes, we don't really have a user's help area per say. The KL section is already very 'busy'. I'd suggest, since you are presumably developing your own KL system, you simply start a thread for what you are working on under section KL-Dev_work. That seems to be what most do. Doesn't really matter though, except if you had a special thread for exactly what you are working on it might be easier to work out what might be going wrong when that happens. i.e. watching the development proceed in detail so any issue that arises makes sense overall. Save2flash/snapmergepuppy is well-tested so with enough detail we should certainly be able to tell you what is happening, the why and the why nots - but needs detail of how you are installing the pieces.

[fredx181]

I ran snapmergepuppy and then renamed upper_changes and created a new upper_changes

I expected to be able to install a second app and save it to the new upper_changes. snapmergepuppy did not make the save in the new upper_changes.

however it did save both apps to the earlier renamed upper changes.
s there a way to make seorate upper_changes in a single session?

Renaming upper_changes while running the system has no real-time effect, it's still part of the overlay stack, that is a fixed thing AFAIK.
But perhaps there's a workaround possible (I doubt it, btw), to accomplish what you'd expect, that I don't know of.
Similar thing is if you would rename (or delete) one of the SFS's, it's mountpoint stays the same (overlay stack is "fixed / static", no "online" changes can be made), so nothing weird happens then unless you reboot.

[wiak]

That would explain it, Fred. I didn't catch on to the idea the changes, renaming of upper_changes and so on, were being made on running system.

[williwaw]

Renaming upper_changes while running the system has no real-time effect, it's still part of the overlay stack, that is a fixed thing AFAIK.

That helps make sense of what is happening, thanks

But perhaps there's a workaround possible (I doubt it, btw), to accomplish what you'd expect, that I don't know of.

A reboot is good enough of a workaround. I see no useful reason for finding a way to unmount anything during a session. Just a matter of convienence if one were trying to make discrete saves into sfs or such.

What would be of interest is to be able to mount multiple upper_changes at boot, or perhaps mount one at boot , and mount the other during the session.
The goal would be to merge two directories or layers with snapmergepuppy.
If one had to rename or change permissions on one of the directories before booting, that would be ok.

what makes snapmergepuppy ignore mounted sfs's when creating a change?

[fredx181]

what makes snapmergepuppy ignore mounted sfs's when creating a change?

I'm not sure what you mean, please explain.
(what I said about mounted sfs's was only example of that the overlay stack cannot be altered while running the system)

[williwaw]

what criteria does snapmergepuppy use to ignore some layers (mounted sfs's) and include others when saving.

can additional layers be added during a session that snapmergepuppy will act on?

the overlay stack cannot be altered while running the system

additional sfs can be mounted during a session,
why not a layer snapmergepuppy can act on?

possibly snapmergepuppy could be modified to pick and choose layers to flush down?

[fredx181]

additional sfs can be mounted during a session,

If you mean mounted as being part of the overlay stack, integrated in the system , no.
With aufs, yes, but not with overlayfs. (if sfs_load included (experimental IMO), it works by creating symlinks, not mounting and altering the stack, as with aufs)
edit: so AFAIK, snapmergepuppy can only deal with the layers created at boot.

[williwaw]

sfs_load ..... works by creating symlinks, not mounting and altering the stack

sfs_load unsquashes (possibly into /tmp)?

and the links?
do they persist (along with any configuration changes to the unsquashed app), in upper_changes at shutdown?

does sfs_load mksquashfs anything?

[fredx181]

sfs_load ..... works by creating symlinks, not mounting and altering the stack

sfs_load unsquashes (possibly into /tmp)?

Not really sure, may have forgot some details, I think the sfs is mounted first and from there creates symlinks in the actual system

and the links?
do they persist (along with any configuration changes to the unsquashed app), in upper_changes at shutdown?

In KLV-Airedale (that I know of) the links will be removed at shutdown, and has kind of plan B (remove links at next boot if failed at shutdown, e.g. due to crash).

[williwaw]

I think the sfs is mounted first

so it is mounted as an sfs?
I am confused now, as you said earlier, mounts after the boot are not permitted with overlay.

[fredx181]

I think the sfs is mounted first

so it is mounted as an sfs?
I am confused now, as you said earlier, mounts after the boot are not permitted with overlay.

Mounts are permitted on itself, e.g. ISO SFS etc... but adding a mount to the overlay stack is not permitted.

[wiak]

The overlay we are talking about is just the one FR initrd makes at boot time, which merges together (like graphic layers) several sfs files to become one resulting root filesystem for vmlinuz to use. It thereafter behaves like a normal Linux so you can mount as many other filesystems as you like, but these are separate mount points and not further 'merged-in' as if a graphic layer. Rather they hang on the mountpoint wherever it is so you can't over-write existing files with these later mounts.

In the FR initrd boot one directory specially named upper_changes is also merged in, but at the very topmost layer and is the writeable (as well as readable). Anything named with 2-digit number is treated by FR initrd as layer and mounted during boot, but as read-only layers in order of the number and then upper_changes directory (a single directory) is topmost layer (readable and writeable).

Prior to booting you can take an existing upper_changes and rename it to say 40upper_changes and therefore on booting whatever is in it will simply then be loaded as a layer at whatever numeric position you gave it (where it will be read but not writable. The only part of these numbered files or directories that is important is the 2-digit number; so you could change 60upper_changes to 60anyjunk and that's fine too. But then a new upper_changes special named directory will be auto-created by FR initrd and again that now empty dir will be used at topmost layer and read/writable.

With the system thus booted, in w_changes=RAM2 mode, that external on media upper_changes directory is mounted by FR initrd to a directory created at /mnt/layers/uc_ro (the name uc_ro wasn't well chosen since uc is actually readable and writable during RAM2 mode).

EDIT: oops had above a bit wrong. The temporary, in RAM changes are stored at /mnt/layers/RAM/upper_changes in originally empty hierarchy in form /root, /etc, /usr and so on, and it is these that are rsync'd by snapmergepuppy (when so demanded) back to writable /mnt/layers/uc_ro, which is the mounted rw external media upper_changes folder. Hence these rsyncd in changes become available still on reboot.

[williwaw]

Prior to booting you can take an existing upper_changes and rename it to say 40upper_changes and therefore on booting whatever is in it will simply then be loaded as a layer

nice, there might be a few old renamed upper_changes around. I only need to prefix them with a double digit to mount them

The temporary, in RAM changes are stored at /mnt/layers/RAM/upper_changes in originally empty hierarchy in form /root, /etc, /usr and so on, and it is these that are rsync'd by snapmergepuppy

if only those double digit prefixed old upper_changes could be unmounted during the session, but still remain in memory, and thus become elgible for being rsynced to the new upperchanges ........geo_c would buy the coffee

[wiak]

Prior to booting you can take an existing upper_changes and rename it to say 40upper_changes and therefore on booting whatever is in it will simply then be loaded as a layer

nice, there might be a few old renamed upper_changes around. I only need to prefix them with a double digit to mount them

But, and its a bit BUT... Since we are using official Void Linux package manager - that keeps track of what has been installed in its package manager database. If you use multiple old (rollback) upper_changes (renumbered to the likes of 60upper_changes, 61upper_changes, 62upper_changes, and so on) it is important to not omit any of them in the sequence since each will contain their parts of the package manager database. That's why we call it rollbacks: sequence is important - must be continuous to keep the package manager database uncorrupted (so okay to remove final rw upper_changes, or final rw upperchanges along with say 62upper_changes (or along with 62upper_changes and 61upperchanges): i.e. rolling back logically to earlier point in usage history.

However, there is a trick I mentioned earlier: At any stage you could take most recent upper_changes and remove /var (which includes its package database info) and then use that as a numbered filesystem (e.g. 70upper_changes) and whatever in that would not be known to the package manager at all (so might for example have just been created after installing the likes of firefox so result is like a firefox addon that doesn't effect package manager since that no longer knows about its existence...). Of course other problems might surface with that approach, but in tests it certainly works and that 'firefox' can almost be used like a portable addon as long as all its other dependencies are found somewhere on the system (if not in that 70upper_changes itself) or could be installed officially using xbps.

[wiak]

if only those double digit prefixed old upper_changes could be unmounted during the session, but still remain in memory, and thus become elgible for being rsynced to the new upperchanges ........geo_c would buy the coffee

I am presuming you are talking about an ability to merge them back into the topmost upper_changes so not needed in next boot since merged in?

Actually that could be done... They are mounted so their contents are available. One by one they could thus be rsync'd back to the on-media upper_changes folder (found mounted at /mnt/layers/uc_ro) followed finally by rsyncing /mnt/layers/RAM/upper_changes (the temporary new stuff held in RAM) back at the end. Just needs someone to write the rsync script to accomplish that merging. You would of course then need to delete (or disable/un-number) the old numbered upper_changes folders prior to rebooting (I have a suspicion you could in fact delete them from the running system and immediately reboot once all the rsyncing has been completed. I think it might be quite simple to arrange if, and I could be wrong, no whiteout char files are ever stored in upper_changes itself (and therefore none such in the number upper_changes folders either); reason I say that is the fact that merging complexity usually comes from having to deal with whiteouts, which if not present makes the merging script simple (one liner rsync command for each mounted numbered upper_changes; i.e. single rsync commandline done in a loop).

EDIT: On second thought I realise there will or can still be whiteout chars in each of these upper_changes filesystems. But... that just means we need to re-use the same snapmergepuppy code for each in turn since that handles the whiteouts correctly. So just a matter of writing a script that uses snapmergepuppy in a loop to not only merge back /mnt/layers/RAM/upper_changes, but also each of these rollback numbered changes one by one in turn. Needs a minor alteration to current snapmergepuppy so not just writing back /mnt/layers/RAM/upper_changes (i.e. that becomes a parameter to the function) and then run that in a loop that goes about merging each rollback file in turn - seems pretty simple still to me since snapmergepuppy already working). Main difficulty is just designing a user-friendly frontend and code to detect which filesystem addons are to be merged back (and the more automatic that is designed to work probably the better). I don't have time just now, but if the likes of fredx181 doesn't do it, I'll look into that early next year when my time frees up again.

[williwaw]

So just a matter of writing a script that uses snapmergepuppy in a loop to not only merge back /mnt/layers/RAM/upper_changes,..........

Thanks. It takes an architect to see the possibilities. Winter is coming in a few months, and I might take up the challenge then.

[wiak]

So just a matter of writing a script that uses snapmergepuppy in a loop to not only merge back /mnt/layers/RAM/upper_changes,..........

Thanks. It takes an architect to see the possibilities. Winter is coming in a few months, and I might take up the challenge then.

That would be highly appreciated.

[rockedge]

@williwaw some work was done very early on in Firstrib development with the ability for saving and persistence using RAM0 or RAM2 mode using a script. It works by creating a XXchanges.sfs on each run of the save.sh script. So eventually there is a set in numbered series. The idea is merge all the save SFS's into one

each save would produce a XXchanges.sfs each time a manual save was done. Like for example: 50changes.sfs, 51changes.sfs, 52changes.sfs with a second script that could merge all the available XXchanges.sfs into a single merged 30changes.sfs. The prefix digits for the final merged copy is hard coded in the script.

These are rough and in a very early stage of development, but may be a place to start with them as an example.

save.sh

Code: Select all

#!/bin/sh
# save.sh
cd /mnt/sdb1/KLV-Airedale-alpha6-RAM2
# Create a sfs of upper changes in ram so that changes persist across reboots
# nnchanges.sfs where nn is 02, 03 ... etc.  Starts from 02changes.sfs as
# 01.. filename is reserved for the main sfs layer filename

NEXT=1
for nextlayer in `ls *changes.sfs | sort -V | cut -c1-2`; do
       	if [ "$nextlayer" -ge 0 ]; then NEXT=$nextlayer; fi
done
NEXT=$(expr $NEXT + 1)
NEXT=$(printf "%02dchanges.sfs" $NEXT)  # add leading 0 if 2 to 9
mksquashfs /mnt/layers/RAM/upper_changes $NEXT -comp lz4 -Xhc -e var run mnt

merge-changes.sh

Code: Select all

#!/bin/bash
# merge-changes.sh
cd /mnt/sdb1/KLV-Airedale-alpha6-RAM2
# Merge multiple xxchanges.sfs into one (02changes.sfs and all prior NNchanges.sfs
# changes sfs files are removed once merged)

# format of what we're looking to achieve .. (note that c is the lowest (first)
# i.e. should correspond to 02changes.sfs, b is the next lowest (03changes.sfs)
# and a is the highest (04changes.sfs)) ...
#    mount -t overlay overlay -o lowerdir=/mnt/L/a:/mnt/L/b:/mnt/L/c /mnt/L/m
# As we're mounting read only (to create a sfs from) we don't need upper or work
# Note that we mount each NNchanges.sfs to a mount point named the same
# i.e. /mnt/L/NNchanges.sfs, which are all merged into /mnt/L/m ... and
# we use that /mnt/L/m to create the new 02changes.sfs 

mkdir /mnt/L 
mkdir /mnt/L/m
unset lower

for addlayer in `ls *changes.sfs | sort -Vr`; do # sorted in reverse order
	[ -z $lower ] && lower=/mnt/L/${addlayer} || lower="${lower}:/mnt/L/${addlayer}"
	mkdir -p "/mnt/L/$addlayer"
	mount $addlayer /mnt/L/$addlayer
done
sync
mount -t overlay overlay -o lowerdir=$lower /mnt/L/m
mksquashfs /mnt/L/m CHANGES.sfs -comp lz4 -Xhc
sync
umount /mnt/L/m
umount /mnt/L/*changes.sfs  # release all mounted sfs's
sync
rmdir /mnt/L/*changes.sfs
rm -rf /mnt/L
rm *changes.sfs
mv CHANGES.sfs 30changes.sfs