mergelayers - merge sfs files and directories into sfs file

Moderator: Forum moderators

Post Reply
gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

mergelayers:
Create a merged sfs file from a list of sfs files,
and/or directories, specified on the command line.
Usage:
mergelayers -h
mergelayers <sfs file> <sfs file> [<sfs file>...] <output sfs file>
Notes:
The list is in stack order, i.e. the first sfs is the top of the stack.
<sfs file> must exist.
<sfs file> can be a directory.
<output sfs file> can NOT be a directory.
<output sfs file> is not included in the merge.
<output sfs file> does not need to exist.
If <output sfs file> does exist,
it will be replaced by the sfs file written by the merge.

If a file called "${prefix}*include.lst" exists for a particular sfs,
e.g 'fdrv_include.lst',
the files specified in this file are extracted to,
a directory called "./${prefix}Dir"
e.g. './fdrvDir'
and this diretory is used in the merge instead of the whole sfs.

About:

'mergelayers' is IO efficient, it does the merge by mounting the layers in a stack.
If the first parameter is a directory containg aufs style whiteout files, aufs will be used,
otherwise overlayfs will be used.
'mergelayers' then does a 'mksquashfs' of the stack's mount-point.

If you include a savefolder as the first parameter, make sure that all sfs files are listed in the following parameters.
Otherwise previously "deleted" files in the ommitted sfs files will reappear.
(Relevant whiteout files would have been "lost" in the merge.)

A utility to merge sfs files is not a new concept, there are a few existing utilities that do this.
However, this is my take on implementing the concept.

mergelayers.gz
gunzip to produce script
(1.95 KiB) Downloaded 66 times
User avatar
Jasper
Posts: 2067
Joined: Wed Sep 07, 2022 1:20 pm
Has thanked: 854 times
Been thanked: 480 times

Re: mergelayers - merge sfs files and directories into sfs file

Post by Jasper »

@gyrog

Thanks for sharing this with us all :thumbup:

I have had a cursory look at the script and realise that you are using XZ as the compressor.

For several months now, I have only been using ZSTD.

Is it possible to change this?

gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

Jasper wrote: Tue Apr 09, 2024 6:39 am

For several months now, I have only been using ZSTD.

Is it possible to change this?

I prefer xz, I do some quite large merges with this, and I'm prepared to wait longer for the smaller size.
So it would have to be an extra parameter of some sort. I'll have to think on the "how".

Hmmm.., maybe something as obvious as '-zstd' as the first parameter.

gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

mergelayers:
Create a merged sfs file from a list of sfs files,
and/or directories, specified on the command line.
Usage:
mergelayers -h
mergelayers [-zstd] <sfs file> <sfs file> [<sfs file>...] <output sfs file>
Notes:
If the optional first parameter '-zstd' is specified,
mksquashfs is called using the 'zstd' compressor instead of 'xz'.
The list is in stack order, i.e. the first sfs is the top of the stack.
<sfs file> must exist.
<sfs file> can be a directory.
<output sfs file> can NOT be a directory.
<output sfs file> is not included in the merge.
<output sfs file> does not need to exist.
If <output sfs file> does exist,
it will be replaced by the sfs file written by the merge.

If a file called "${prefix}*include.lst" exists for a particular sfs,
e.g 'fdrv_include.lst',
the files specified in this file are extracted to,
a directory called "./${prefix}Dir"
e.g. './fdrvDir'
and this diretory is used in the merge instead of the whole sfs.

About:
Now supports 'zstd' as well as 'xz'.

Attachments
mergelayers-02.tar
"tar xf mergelayers-02.tar" to produce script
(2.12 KiB) Downloaded 41 times
User avatar
rockedge
Site Admin
Posts: 6524
Joined: Mon Dec 02, 2019 1:38 am
Location: Connecticut,U.S.A.
Has thanked: 2736 times
Been thanked: 2615 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by rockedge »

@gyrog Starting to experiment with this in RAM0 mode in KLV-Airedale and KLV-Spectr which usually does not have any save session option.

There was some work done on developing such a feature with a save.sh and merge_changes.sh scripts. The first takes the current RAM state and makes a XXchanges.sfs file. Each time a sequentially numbered XXchanges.sfs will be generated by running save.sh. The second script then merges all of the SFS files into a merged 30changes.sfs file.

I am going to experiment with this version of SFS merge along with the save.sh script, keeping in mind that the save.sh script is very rudimentary and no polish. But I feel like this might be a good fit in KLV to accomplish persistence manipulation in RAM0 mode in Kennel Linux variants.

gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

@rockedge, hope you find it useful.

gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

I have attached 'mergelayers-03.tar'.

This new version provides extended support for "fiddlers".

The '-zstd' optional parmmeter has now be enhanced with an optional trailing compression level.
If '-zstd19' is specified, mksquashfs is called using '-comp zstd -Xcompression-level 19'.

An '-lzo' optional parameter has been added, that accepts an optional trailing algorithm spec.
If '-lzo1x_1_15' is specified, mksquashfs is called using '-comp lzo -Xalgorithm lzo1x_1_15'.

Theee is also an optional '-b', blocksize parameter.
If '-b262144' is specified, mksquashfs is called using '-b 262144'.

In the previous version, "mergelayers a.sfs b.sfs o.sfs" failed if 'o.sfs' did not exist.
But "mergelayers a.sfs b.sfs ./o.sfs" worked.
This bug has been fixed.

mergelayers-03.tar
"tar xf mergelayers-03.tar" to produce script
(2.39 KiB) Downloaded 14 times
User avatar
rockedge
Site Admin
Posts: 6524
Joined: Mon Dec 02, 2019 1:38 am
Location: Connecticut,U.S.A.
Has thanked: 2736 times
Been thanked: 2615 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by rockedge »

@gyrog
I am going to try a bigger test with attempting to merge a 07KLV-vmhost_rootfs-xz.sfs with a upper_changes.sfs

geo_c
Posts: 2877
Joined: Fri Jul 31, 2020 3:37 am
Has thanked: 2197 times
Been thanked: 872 times

Re: mergelayers - merge sfs files and directories into sfs file

Post by geo_c »

rockedge wrote: Wed Oct 02, 2024 3:10 pm

@gyrog
I am going to try a bigger test with attempting to merge a 07KLV-vmhost_rootfs-xz.sfs with a upper_changes.sfs

Hmm, I haven't tried this script yet, I'm just now reading about it and seemed to have missed this topic before. I'm pondering the idea that it may be an alternative to psuedo-full install method of building a KL. Instead, layers can be added and all merged with upper_changes and the rootfs. A kind of remaster on the fly technique.

I'm inclined to see if I can make that work, as right now I haven't updated my psuedo full installs, and instead have been using multiple config layers and squashing upper_changes occassionally as a new layer. But that approach takes up a lot of storage space. This script approach seems that it will eliminate the lower layer's wasted files that are whited out.

geo_c
Old School Hipster, and Such

gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

@rockedge, and @geo_c,
Hope it works for you.
If it does, you are welcome.

gyrog
Posts: 644
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 16 times
Been thanked: 231 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

I have attached 'mergelayers-04.tar'.

Now works with just a single input file, i.e. it can be used to recreate an sfs with different compression.

Code: Select all

mergelayers -lzo puppy...sfs puppy-lzo.sfs

The "Usage" info you get from running 'mergelayers' without any parameters,
now documents a long-standing capability to specify an "exclude list" file to the final "mksquashfs".

mergelayers-04.tar
"tar xf mergelayers-04.tar" to produce script
(2.42 KiB) Downloaded 12 times
Post Reply

Return to “Utilities”