XZ, GZIP, LZ4 and ZSTD: Which Format is Best of compression for SFS?

[Duprate]

To solve this doubt, I decided to test the notebook Acer Aspire 4332, with processor Intel Celeron 900, 1 core; 2Gb DDR2 memory.
FatDog64 operating system, booted by the overlay mode created by Wiak.

First, with the system compressed into XZ, which is my usual mode, where the system has files SFS with a total size of 236Mb.
Booted in 42 seconds, with RAM usage between 152Mb and 168Mb. I installed Firefox and the size is busy in RAM it was 590Mb.
Running Firefox, watching a video on NETFLIX, RAM consumption was 909Mb and ZRAM was 98Mb.

Soon after, with the system compressed in ZSTD, the SFS file system got the size total at 296Mb.
Booted in 33 seconds, with RAM usage between 142Mb and 158Mb. I installed Firefox and the size is busy in RAM it stayed at 650Mb.
Running Firefox, watching the same video on NETFLIX, RAM consumption was 1Gb and ZRAM was 299Mb.

So, in my opinion, the gain of 9 seconds in initialization does not justify the use of ZSTD compression why do SFS files get bigger, take up more space in RAM and the consumption of RAM running applications as well got bigger.

1- This test is valid for the aforementioned notebook;
2- This test is useful for users who have a notebook with the same or similar configuration;
3- I also tested the GZIP format (similar results to ZSTD) and LZ4 (where the SFS files got much bigger) and the functioning of Firefox was hampered, with crashes.
4- The "bottleneck is the 2Gb memory. The 1 core Celeron, even from the service account ....

I hope this test is useful for some user of the forum, regardless of which system they use and, with other machines, other results!

Good luck to everyone!

----

Mod edit: Change title to reflect the question better.

[pp4mnklinux]

Of course that results are tested, contrasted an verifited for you, but my experience says that the best compression format depends on the specific use case and the type of data being compressed.

Each format has its own strengths and weaknesses, and it's important to choose the right one based on your needs.

Taking that into account we can say that:

XZ is known for its high compression ratio, making it ideal for compressing large files or archives. However, it can be slower than some other compression algorithms.

GZIP is a widely used compression format that offers good compression ratios and is relatively fast. It's commonly used for compressing web pages, emails, and other text-based documents.

LZ4 is a compression algorithm that is known for its high speed and low memory requirements. It's ideal for applications that require fast compression and decompression, such as real-time data processing.

ZSTD is a relatively new compression algorithm that offers a good balance of compression ratio and speed. It's designed to be used in a wide range of applications, from real-time data processing to archiving.

In general, if you're looking for high compression ratios and don't mind slower compression times, XZ is a good choice.

If you need a fast and efficient compression algorithm, LZ4 is a good option.

If you need a balance between compression ratio and speed, ZSTD is a good choice.

If you're looking for a widely supported compression format that's good for text-based documents, GZIP is a solid choice.

So.... a lot of factors, so each one must choose the best for him/ her.

Cheers

[Duprate]

Nice answer! Well explained, but as I said: "valid test for the said notebook" and "other machines, other results!".
In practice, the type of compression interferes with the boot time, but once everything is loaded into RAM, what matters is the amount of RAM available. The more RAM available, the better the apps will work! That's my reasoning, when using a frugal installation of Puppy Linux, or derivative on rickety and old hardware.

My Celeron 900 with 2 Gb of memory, it just wants to work, for a little while ..........

On my PC, with a better processor and 8Gb of memory, reducing boot time makes sense!

[amethyst]

I would use gzip with an older machine. Decent compression rate and speed/quite fast decompression... and important, very low resources. I also use older hardware and all my sfs's are compressed with gzip. When I tested the decompression rate comparing gzip and lz4, I couldn't really observe a difference.

[dimkr]

The SFS compression algorithm shouldn't affect Firefox RAM consumption. The differences in the browser's RAM consumption are probably the result of the browser's background activity (things like prefetching, extension updates, service workers, etc'). Even subtle things like temporary slowdown of your network can affect RAM consumption - if download is slow, Firefox has a smaller playback buffer of the same video.

[amethyst]

The SFS compression algorithm shouldn't affect Firefox RAM consumption. The differences in the browser's RAM consumption are probably the result of the browser's background activity (things like prefetching, extension updates, service workers, etc'). Even subtle things like temporary slowdown of your network can affect RAM consumption - if download is slow, Firefox has a smaller playback buffer of the same video.

CPU usage much less with gzip compression ( we talking about the actual act of compressing files and I think it will also be less with the decompression process), will be very difficult to beat. I have no doubt that memory usage is also lower. Gzip is great for challenged machines, no doubt about it.

[dimkr]

CPU usage much less with gzip compression

Compared to xz - yes. In every benchmark I've seen, zstd beats gzip (Deflate/zlib) in decompression speed.

[amethyst]

CPU usage much less with gzip compression

Compared to xz - yes. In every benchmark I've seen, zstd beats gzip (Deflate/zlib) in decompression speed.

Yes but how resource friendly is it especially on an old machine. CPU usage/ RAM compared to gzip? BTW - I've tested lz4 and gzip once upon a time and with normal Puppy operation I didn't notice any difference in decompression speed. Also - I've seen benchmark tests which indicates that pigz (multi-threaded gzip) is even faster than zstd at compression and decompression. Maybe something to explore for fastest available?

[Duprate]

Thanks to everyone who replied to this thread!
I also did this test with my PC, with a better processor and 8Gb DDR3 memory. I converted all SFS to ZSTD standard. Startup time reduced to 18 seconds (minus 5 sec). Firefox just got more responsive. I also tested it with Google Chrome and the RAM consumption was around 700Mb, proving to be a lighter browser, better for older hardware (maybe less secure, I don't know... I prefer Firefox).
Later, I will test the other formats. If there's anything relevant, I'll post it here.

[Duprate]

S15Pup32 (theme simulating JammyJellyfish) compressed files as ZSTD, faster startup and lower RAM usage, with Google Chrome (651Mb).

[amethyst]

S15Pup32 (theme simulating JammyJellyfish) compressed files as ZSTD, faster startup and lower RAM usage, with Google Chrome (651Mb).

Compared to?

[Duprate]

FatDog64, ZSTD compression, Google Chrome, RAM consumption = 700Mb
S15Pup32, ZSTD compression, Google Chrome, RAM consumption = 651Mb
But as Dimkr explained, "Differences in browser RAM consumption are most likely the result of browser background activity." Two different operating systems, run under the same conditions, produced nearly identical results. Boot time of the first system and the second system, 18 and 22 seconds respectively.

[amethyst]

FatDog64, ZSTD compression, Google Chrome, RAM consumption = 700Mb
S15Pup32, ZSTD compression, Google Chrome, RAM consumption = 651Mb
But as Dimkr explained, "Differences in browser RAM consumption are most likely the result of browser background activity." Two different operating systems, run under the same conditions, produced nearly identical results. Boot time of the first system and the second system, 20 and 22 seconds respectively.

You need to compare same operating system, same hardware, same software when using the different compression algorythms/formats (zstd, gzip,xz). What is the speed difference and resource requirements (RAM, CPU) comparing gzip with zstd for example.

[Clarity]

As this thread moves, it MUST isolate what is discussed?

Concerns of boot times may/may-not be an important item? But, if it is, the discussion MUST focus of that subject as it means NOTHING to other subjects in compression outcomes.

Concerns for system app's operations for I/O may/may-not be an important item? Similarly, if it is, the discussion MUST focus of that subject as it means NOTHING to other subjects in compression outcomes.

And if subsystems like browser behavior may/may-not be an important item? Same for system's subsystems like when used for NAS/media/streaming operations may/may-not be an important item in compression outcomes?

This post can/will hopefully make a discussion like this one more focused on which of the areas it intends to address.

[amethyst]

As this thread moves, it MUST isolate what is discussed?

Concerns of boot times may/may-not be an important item? But, if it is, the discussion MUST focus of that subject as it means NOTHING to other subjects in compression outcomes.

Concerns for system app's operations for I/O may/may-not be an important item? Similarly, if it is, the discussion MUST focus of that subject as it means NOTHING to other subjects in compression outcomes.

And if subsystems like browser behavior may/may-not be an important item? Same for system's subsystems like when used for NAS/media/streaming operations may/may-not be an important item in compression outcomes?

This post can/will hopefully make a discussion like this one more focused on which of the areas it intends to address.

MUST? Have you done any compression/decompression tests yourself in order to contribute constructively towards this specific thread?

[Clarity]

Can you rephrase your question?

Which of these are YOU most interested in? And which do YOU feel is best for this community?

I ask as I am not sure what YOU are looking for.

Hope that you're not trying to pick a fight as I am only interested helping when I understand which objective we are after.

Added later
Additionally, I am aware of several comparison tests seen in the internet against various scenarios and workloads.

Thanks. After examining your tone, I think it best I exit this thread.

[amethyst]

I am aware of several comparison tests seen in the internet against various scenarios and workloads.

Now. posting links to or attaching the results of these tests that you have come across on the internet would be valuable and applicable to this thread will be a good start, unlike the other gibberish you have posted. I'm particularly interested how this will impact on the specific systems (and different machines, old/new) we use here. For instance - I have compared lz4 and gzip compressed puppy files and in terms of decompression speed which should theoretically impact on basic operation of the OS, I couldn't notice a difference on my older machine. Comparing with xz decompression, a difference can be observed in operational speed, mostly only when opening an application for the first time in a session though. I don't think the decompression speed plays much of a role really as far as general operation is concerned. For older Puppys it's basically, gzip vs xz and gzip is better in terms of observed operational speed and use of resources especially on an older machine.

[Duprate]

Good morning everybody! This was my goal: to get people with more knowledge on this subject to speak up. My testing was amateurish, but now, with the participation of our community, I understand a little more about compression. Much has been said about the ZSTD format and I confess that I knew nothing about it. I'm going to test FatDog64, ZSTD compression, for at least a week and draw my own conclusions, regarding the functioning of the applications, speed, CPU heating, etc...
Later I will also test GZIP.

[jamesbond]

FACT 1: All compressed SFS have to be decompressed before you.
Once decompressed, the kernel will hold on of a copy of the decompressed file for a short while.

E.g. launching firefox for the first time, the "decompression" effect will show up. It will take a while to load up. But if you close firefox and launch it again, it would be faster. Because the decompressed copy of firefox (or parts of it) are already in the cache. How long it will stay in the cache, and how big is the cache, depends on your RAM size.

FACT 2: If you load your SFS to RAM, then some part of your RAM will be used to "store" that SFS. That part of the RAM cannot be reclaimed and cannot be used by the application. The bigger the SFS, the more RAM that you lose.

---

With that out of the way:
- xz compression (=making SFS) is very slow. xz decompression is decent (=fast). xz compression ratio is astounding, it's the best there is.

- gzip, lz4, lzo, zstd, brotli ... all have similar figures, as they are all based on variants of the same algorithm (LZ77). Overall gzip, provides __better__ compression compared to the others, but it is slightly slower. The others are tuned for speed, at the cost of compression ratio. Unless you're running benchmark and/or compressing/decompressing every minute (=that is, you're a server pumping out petabytes of data), the difference between these compressors are small. Which one you choose is a matter of preference.

The only comparison is between xz and the LZ77 compressors. Which one to choose?

It depends, I suppose. For me, I open an app and it stays there for the day. I don't open and close programs a hundred times a day. So an extra one, two or even three seconds to launch a program, is something that I can tolerate. Especially, knowing that because the __significantly__ SFS is smaller (about 2 or 3 times smaller than LZ77-compressed SFS), it means I have more RAM available for use by the programs themselves. It's a clear winner.

[Duprate]

Hello! I agree with your point of view, James Bond. Since the first post, my main interest was to find out the best compression format for my Intel Celeron 900 notebook, 1 core; 2Gb DDR2 memory. So, it was important to read the opinion of other users. But I also came to the conclusion that for notebooks and PCs with poor configuration and little RAM, the most important thing is to have more memory left over for applications to use, than a few seconds more to start a program.
However, the tests on my PC, with an Intel dual core processor and 8 Gb of RAM, using ZSTD compression on the SFS files, I observed a little more agility in the programs and less CPU heating.
However, the importance given to this varies from one person to another...