mergelayers - merge sfs files and directories into sfs file

Moderator: Forum moderators

Post Reply
gyrog
Posts: 594
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 14 times
Been thanked: 180 times
Contact:

mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

mergelayers:
Create a merged sfs file from a list of sfs files,
and/or directories, specified on the command line.
Usage:
mergelayers -h
mergelayers <sfs file> <sfs file> [<sfs file>...] <output sfs file>
Notes:
The list is in stack order, i.e. the first sfs is the top of the stack.
<sfs file> must exist.
<sfs file> can be a directory.
<output sfs file> can NOT be a directory.
<output sfs file> is not included in the merge.
<output sfs file> does not need to exist.
If <output sfs file> does exist,
it will be replaced by the sfs file written by the merge.

If a file called "${prefix}*include.lst" exists for a particular sfs,
e.g 'fdrv_include.lst',
the files specified in this file are extracted to,
a directory called "./${prefix}Dir"
e.g. './fdrvDir'
and this diretory is used in the merge instead of the whole sfs.

About:

'mergelayers' is IO efficient, it does the merge by mounting the layers in a stack.
If the first parameter is a directory containg aufs style whiteout files, aufs will be used,
otherwise overlayfs will be used.
'mergelayers' then does a 'mksquashfs' of the stack's mount-point.

If you include a savefolder as the first parameter, make sure that all sfs files are listed in the following parameters.
Otherwise previously "deleted" files in the ommitted sfs files will reappear.
(Relevant whiteout files would have been "lost" in the merge.)

A utility to merge sfs files is not a new concept, there are a few existing utilities that do this.
However, this is my take on implementing the concept.

mergelayers.gz
gunzip to produce script
(1.95 KiB) Downloaded 49 times
User avatar
Jasper
Posts: 1595
Joined: Wed Sep 07, 2022 1:20 pm
Has thanked: 676 times
Been thanked: 357 times

Re: mergelayers - merge sfs files and directories into sfs file

Post by Jasper »

@gyrog

Thanks for sharing this with us all :thumbup:

I have had a cursory look at the script and realise that you are using XZ as the compressor.

For several months now, I have only been using ZSTD.

Is it possible to change this?

gyrog
Posts: 594
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 14 times
Been thanked: 180 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

Jasper wrote: Tue Apr 09, 2024 6:39 am

For several months now, I have only been using ZSTD.

Is it possible to change this?

I prefer xz, I do some quite large merges with this, and I'm prepared to wait longer for the smaller size.
So it would have to be an extra parameter of some sort. I'll have to think on the "how".

Hmmm.., maybe something as obvious as '-zstd' as the first parameter.

gyrog
Posts: 594
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 14 times
Been thanked: 180 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

mergelayers:
Create a merged sfs file from a list of sfs files,
and/or directories, specified on the command line.
Usage:
mergelayers -h
mergelayers [-zstd] <sfs file> <sfs file> [<sfs file>...] <output sfs file>
Notes:
If the optional first parameter '-zstd' is specified,
mksquashfs is called using the 'zstd' compressor instead of 'xz'.
The list is in stack order, i.e. the first sfs is the top of the stack.
<sfs file> must exist.
<sfs file> can be a directory.
<output sfs file> can NOT be a directory.
<output sfs file> is not included in the merge.
<output sfs file> does not need to exist.
If <output sfs file> does exist,
it will be replaced by the sfs file written by the merge.

If a file called "${prefix}*include.lst" exists for a particular sfs,
e.g 'fdrv_include.lst',
the files specified in this file are extracted to,
a directory called "./${prefix}Dir"
e.g. './fdrvDir'
and this diretory is used in the merge instead of the whole sfs.

About:
Now supports 'zstd' as well as 'xz'.

Attachments
mergelayers-02.tar
"tar xf mergelayers-02.tar" to produce script
(2.12 KiB) Downloaded 23 times
User avatar
rockedge
Site Admin
Posts: 5713
Joined: Mon Dec 02, 2019 1:38 am
Location: Connecticut,U.S.A.
Has thanked: 1992 times
Been thanked: 2097 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by rockedge »

@gyrog Starting to experiment with this in RAM0 mode in KLV-Airedale and KLV-Spectr which usually does not have any save session option.

There was some work done on developing such a feature with a save.sh and merge_changes.sh scripts. The first takes the current RAM state and makes a XXchanges.sfs file. Each time a sequentially numbered XXchanges.sfs will be generated by running save.sh. The second script then merges all of the SFS files into a merged 30changes.sfs file.

I am going to experiment with this version of SFS merge along with the save.sh script, keeping in mind that the save.sh script is very rudimentary and no polish. But I feel like this might be a good fit in KLV to accomplish persistence manipulation in RAM0 mode in Kennel Linux variants.

gyrog
Posts: 594
Joined: Thu Oct 01, 2020 8:17 am
Location: Australia
Has thanked: 14 times
Been thanked: 180 times
Contact:

Re: mergelayers - merge sfs files and directories into sfs file

Post by gyrog »

@rockedge, hope you find it useful.

Post Reply

Return to “Utilities”