Puppy Linux Discussion Forum

mergelayers:
Create a merged sfs file from a list of sfs files,
and/or directories, specified on the command line.
Usage:
mergelayers -h
mergelayers <sfs file> <sfs file> [<sfs file>...] <output sfs file>
Notes:
The list is in stack order, i.e. the first sfs is the top of the stack.
<sfs file> must exist.
<sfs file> can be a directory.
<output sfs file> can NOT be a directory.
<output sfs file> is not included in the merge.
<output sfs file> does not need to exist.
If <output sfs file> does exist,
it will be replaced by the sfs file written by the merge.

If a file called "${prefix}*include.lst" exists for a particular sfs,
e.g 'fdrv_include.lst',
the files specified in this file are extracted to,
a directory called "./${prefix}Dir"
e.g. './fdrvDir'
and this diretory is used in the merge instead of the whole sfs.

About:

'mergelayers' is IO efficient, it does the merge by mounting the layers in a stack.
If the first parameter is a directory containg aufs style whiteout files, aufs will be used,
otherwise overlayfs will be used.
'mergelayers' then does a 'mksquashfs' of the stack's mount-point.

If you include a savefolder as the first parameter, make sure that all sfs files are listed in the following parameters.
Otherwise previously "deleted" files in the ommitted sfs files will reappear.
(Relevant whiteout files would have been "lost" in the merge.)

A utility to merge sfs files is not a new concept, there are a few existing utilities that do this.
However, this is my take on implementing the concept.

@gyrog

Thanks for sharing this with us all

I have had a cursory look at the script and realise that you are using XZ as the compressor.

For several months now, I have only been using ZSTD.

Is it possible to change this?

Jasper wrote: Tue Apr 09, 2024 6:39 am

For several months now, I have only been using ZSTD.

Is it possible to change this?

I prefer xz, I do some quite large merges with this, and I'm prepared to wait longer for the smaller size.
So it would have to be an extra parameter of some sort. I'll have to think on the "how".

Hmmm.., maybe something as obvious as '-zstd' as the first parameter.

mergelayers:
Create a merged sfs file from a list of sfs files,
and/or directories, specified on the command line.
Usage:
mergelayers -h
mergelayers [-zstd] <sfs file> <sfs file> [<sfs file>...] <output sfs file>
Notes:
If the optional first parameter '-zstd' is specified,
mksquashfs is called using the 'zstd' compressor instead of 'xz'.
The list is in stack order, i.e. the first sfs is the top of the stack.
<sfs file> must exist.
<sfs file> can be a directory.
<output sfs file> can NOT be a directory.
<output sfs file> is not included in the merge.
<output sfs file> does not need to exist.
If <output sfs file> does exist,
it will be replaced by the sfs file written by the merge.

If a file called "${prefix}*include.lst" exists for a particular sfs,
e.g 'fdrv_include.lst',
the files specified in this file are extracted to,
a directory called "./${prefix}Dir"
e.g. './fdrvDir'
and this diretory is used in the merge instead of the whole sfs.

About:
Now supports 'zstd' as well as 'xz'.

@gyrog Starting to experiment with this in RAM0 mode in KLV-Airedale and KLV-Spectr which usually does not have any save session option.

There was some work done on developing such a feature with a save.sh and merge_changes.sh scripts. The first takes the current RAM state and makes a XXchanges.sfs file. Each time a sequentially numbered XXchanges.sfs will be generated by running save.sh. The second script then merges all of the SFS files into a merged 30changes.sfs file.

I am going to experiment with this version of SFS merge along with the save.sh script, keeping in mind that the save.sh script is very rudimentary and no polish. But I feel like this might be a good fit in KLV to accomplish persistence manipulation in RAM0 mode in Kennel Linux variants.

@rockedge, hope you find it useful.

I have attached 'mergelayers-03.tar'.

This new version provides extended support for "fiddlers".

The '-zstd' optional parmmeter has now be enhanced with an optional trailing compression level.
If '-zstd19' is specified, mksquashfs is called using '-comp zstd -Xcompression-level 19'.

An '-lzo' optional parameter has been added, that accepts an optional trailing algorithm spec.
If '-lzo1x_1_15' is specified, mksquashfs is called using '-comp lzo -Xalgorithm lzo1x_1_15'.

Theee is also an optional '-b', blocksize parameter.
If '-b262144' is specified, mksquashfs is called using '-b 262144'.

In the previous version, "mergelayers a.sfs b.sfs o.sfs" failed if 'o.sfs' did not exist.
But "mergelayers a.sfs b.sfs ./o.sfs" worked.
This bug has been fixed.

@gyrog
I am going to try a bigger test with attempting to merge a 07KLV-vmhost_rootfs-xz.sfs with a upper_changes.sfs

rockedge wrote: Wed Oct 02, 2024 3:10 pm

@gyrog
I am going to try a bigger test with attempting to merge a 07KLV-vmhost_rootfs-xz.sfs with a upper_changes.sfs

Hmm, I haven't tried this script yet, I'm just now reading about it and seemed to have missed this topic before. I'm pondering the idea that it may be an alternative to psuedo-full install method of building a KL. Instead, layers can be added and all merged with upper_changes and the rootfs. A kind of remaster on the fly technique.

I'm inclined to see if I can make that work, as right now I haven't updated my psuedo full installs, and instead have been using multiple config layers and squashing upper_changes occassionally as a new layer. But that approach takes up a lot of storage space. This script approach seems that it will eliminate the lower layer's wasted files that are whited out.

@rockedge, and @geo_c,
Hope it works for you.
If it does, you are welcome.

I have attached 'mergelayers-04.tar'.

Now works with just a single input file, i.e. it can be used to recreate an sfs with different compression.

mergelayers -lzo puppy...sfs puppy-lzo.sfs

The "Usage" info you get from running 'mergelayers' without any parameters,
now documents a long-standing capability to specify an "exclude list" file to the final "mksquashfs".