Are lzo and lz4 compression algorithms kernel specific?

[amethyst]

I want to repack my puppy sfs files in lzo or lz4 compression format since it apparently has the fastest decompression values which speeds up Puppy operation. I'm using Bionic 32 and according to the help section of mksquashfs, lzo compression is available. So I successfully repacked the puppy files in that format but get a failure message at bootup ie. "lzo compression not supported, filesystem not supported". Tried with different kernels (newer too) and get the same message. Is this kernel related, perhaps a "flag" not set for this compression algorithm?

[amethyst]

No takers? I think decompression speed is an important topic to discuss since the traditional Puppy frugal install is all about a read-only compressed filesystem. I'm not using the newest Puppys but most are still released in xz compression format, if I'm not mistaken? This is fine for distribution purposes (smaller size for download) but is not best for optimum operational speed. This topic is especially important for those still using old and less powerful machines. I've repacked all my sfs files in gzip (level 9) compression format and there is a notable operational difference compared to using xz compression (especially in terms of running applications first time up - once loaded into memory it's okay depending on how much RAM you have available). I think the fastest decompression algorithm (which apparently is lz4) should be used for best speed. According to the limited research I have done, gzip's decompression ration is on average about twice as fast as xz whilst lz4 is apparently 4-times as fast as gzip. So how are we going to make this happen for all Puppys, older and new?

[amethyst]

It seems I can create an sfs file with lzo compression but can't mount it. Using sfs_load I get a "wrong fs" error in terminal. Any ideas what is going on here?

[rockedge]

@amethyst Those compression protocols do need the kernel to have the modules enabled at kernel build time.

I have made the last few kernels with those included. Do you need a 32 bit kernel with these enabled?

[amethyst]

@amethyst Those compression protocols do need the kernel to have the modules enabled at kernel build time.

I have made the last few kernels with those included. Do you need a 32 bit kernel with these enabled?

Yes please, I would like to check it out.

[rockedge]

something in the 5.4+ range or newer?

[amethyst]

something in the 5.4+ range or newer?

rockedge - Thanks, but I think give it a pass. I've just checked with the only 64-bit Puppy I have and which kernel has lz4 enabled. Initial tests indicate that it is a bit faster than gzip at decompression but I wouldn't say much much faster (compression is also a bit faster but the output size is horrific). The other thing is that almost all of the newer kernels hang on my machine at bootup for a few minutes so chances are that the one you produce will also hang. I'm actually running Tahr's kernel with Bionic I'm using now, it just works best with my hardware. Boots fast and the system is very stable. I think gzip compression is quite good for compression and decompression, sort of a good balance. Use the highest compression level for the fastest decompression. xz compression is really slow for compression and decompression, not a good option for the running system and extra sfs files compressed with that algorithm.
Edit: Actually, maybe not such a bad idea to produce a new 32-bit kernel for this. May be useful to others, lz4 is faster as mentioned (but almost the same at decompressing speed as gzip level 9 compressed files according to my personal observations testing in a Puppy environment).

[ljlj]

debiandog quick remaster
the best linux os!
with lz4 compression!!!

[amethyst]

BTW - From the DOTconfig file of the old kernel I'm using:

CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_KERNEL_XZ=y
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set

I see there is no CONFIG_KERNEL_GZIP=y entry (in comparison to CONFIG_KERNEL_XZ=y) but booting the gzip compressed Puppy files works. Why is this or is gzip just enabled by default? Also - DPBuster64 with its kernel supports booting of lz compressed files but its DOTconfig file looks basically the same as the above entries with only CONFIG_KERNEL_XZ=y set (and not lz). So why does booting of lz files work there?

[Jasper]

@amethyst

I have read your comments with great interest and would like to know how you created the SFS files?

Are you unpacking the SFS files and then repacking them with your desired compression format?

With this tool?

viewtopic.php?p=76706#p76706

I did attempt to do this on a NTFS drive and that was problematic. Also, to remove the uncompressed directories to a long time. Lesson learnt the hard way

[amethyst]

@amethyst

I have read your comments with great interest and would like to know how you created the SFS files?

Are you unpacking the SFS files and then repacking them with your desired compression format?

With this tool?

viewtopic.php?p=76706#p76706

I did attempt to do this on a NTFS drive and that was problematic. Also, to remove the uncompressed directories to a long time. Lesson learnt the hard way

I normally use my own right-click sfs editing tool (part of my suite). If the sfs file is already loaded, you can save time by compressing its contents directly from its mounting point in /initrd with Packit.

[Jasper]

@amethyst

Thanks for replying back.

Is there a particular directory I should be looking at? Or is it the entire contents of /initrd

[amethyst]

@amethyst

Thanks for replying back.

Is there a particular directory I should be looking at? Or is it the entire contents of /initrd

pup_ro2 contains the contents of the base sfs. Click to open the folder, then select all the items, right-click on any of the selected items and run Packit from the open with menu. Save the output to another location.

[Jasper]

Excellent

[amethyst]

Excellent

Instead of Packit you could have used the following commandline in terminal:

Code: Select all

mksquashfs /initrd/pup_ro2 /mnt/home/NameOfNewSFS.sfs -comp gzip -b 1M

The above command should compress the contents of the base sfs with gzip max level compression and save the new SFS to /mnt/home for example.

[Jasper]

@amethyst

Thanks for showing me how to do this and the tip regarding how to compress it

I do enjoy learning how to use the applications like this as it demonstrates knowledge and gives me an opportunity to experiment too

[amethyst]

@amethyst

Thanks for showing me how to do this and the tip regarding how to compress it

I do enjoy learning how to use the applications like this as it demonstrates knowledge and gives me an opportunity to experiment too

It's good to learn new things. As you can see the executable applicable here is, mksquashfs. Now if you run mksquashfs --help in terminal, you will find most of the compression options available to do this manually. The commandline is very powerful. GUI is nice and comfortable but it's always nice to learn how things work "under the hood" (and there are more options).