Page 1 of 1

Opening application archived as an sfs versus opening it from a folder

Posted: Fri Apr 14, 2023 6:52 am
by amethyst

Okay, I've mentioned this before but I think rather important as this impacts system performance. Scenario - On /mnt/home I have an application (let's say it's Firefox) archived as an sfs file. On the same partition I have the exact Firefox but in an uncompressed state in a folder (like a portable for example). Now, the application starts much faster for me at first run in the form of a loaded sfs (contents NOT copied to RAM) file than running the executable from the folder as mentioned. Since my sfs files are all pre-loaded during boot, this is a win/win situation for me. This may not be so noticeable with fast storage media, lots of RAM and a fast machine but definitely something to consider when your hardware is not so flash. Of course, if you have lots of RAM you can copy your sfs files to RAM at bootup which will make operation even faster (but then the frustration of longer boot time to allow for the copying).
The other thing of course is to use the fastest decompression method for your sfs files. Apparently zstd is best for the new Puppys (although I have read that multi-threaded gzip is even faster than zstd) so one should use that. For older Puppys use gzip. I've also experimented with different compression levels. This makes a noticeable difference for compressing speed but not really when decompressing. So use the fastest compressor for sfs's but compress with the highest level (will be slower compressing, but smaller size and you are probably only going to compress once anyway). Theoretically, an sfs compressed at a higher compression level should decompress slightly faster as when compressed at lower level however this is not really noticeable for me. I've also experimented with making sfs's using the non-compression data options. This results in quick "compression" (nothing is basically compressed) but interestingly does not make any noticeable difference when decompressing the archive. Some food for thought...


Re: Opening application archived as an sfs versus opening it from a folder

Posted: Fri Apr 14, 2023 11:11 am
by wiak

For a processor to perform instructions provided by a file, these instructions first have to be loaded into RAM, where some RAM cache/buffering will be involved.

Mounting a filesystem, does not 'pre-load' its data blocks. Mounting any device (be that something like a hard disk or a 'loop-mounted' file image (such as a squashed filesystem or the likes of a formatted savefile and so on) simply reads in sufficient information to thereafter allow its internal files and directories to be accessed and loaded when required (i.e. opened and accessed).

I believe that in some scenarios starting the likes of firefox from an uncompressed directory from a mounted hard disk will be faster than starting the same program from a mounted squashfs (sfs) file. It depends on various conflicting factors:

A certain number of data blocks need to be read in from the mounted device such that the application appears as running to the user.

If the hard disk is relatively slow then I believe it makes sense that it is advantageous that some compression is used on the stored hard disk data blocks for, say, firefox. The reason for that is that reading from the hard disk is a bottleneck so the smaller the size of each data block being read to RAM the faster it can be read in... As a binary, firefox, will be stored as a fixed number of uncompressed datablocks - that's the size of the uncompressed binary. Whilst data blocks may be combined in the sfs algorithm, for simplicity, let's say each datablock is compressed such that it is three times smaller than uncompressed - in that simple scenario the whole of firefox could be at least read into RAM (cache) three times faster as far as disk Input/Output part of the operation is concerned - however, it still has to be uncompressed (though likely done in chunks as it comes in... that's just a matter of the compression/decompression algorithm used).

So the second factor is how fast can the CPU decompress the data, since the program instruction code cannot be executed until decompressed. If the decompression algorithm is extremely CPU-intensive that could also be a bottleneck, but generally speaking (and particularly on older systems) it may be that hard disk I/O is a more problematic (slow) bottleneck than CPU decompression (by how much will of course depend on the decompression algorithm...). Hence, I'd say, it makes sense that a compressed version of firefox stored on hard disk is likely to start up faster than an uncompressed version on hard disk when that external hard disk media is slow in terms of I/O reading.

However, that situation is not fixed since it depends on several factors. When a faster hard disk is being used, and a relatively slow CPU (assuming RAM always relatively fast), AND a complex decompression algorithm that is CPU intensive, then the uncompressed firefox stored on hard disk could well prove to be faster to start up.

i.e. There is no one answer, but the most important thing to understand is that mounting an sfs file does not pre-load its data blocks into RAM.

As far as pre-loading the whole compressed sfs file into RAM. Well, thereafter of course, the problem with slow hard disk I/O is eliminated so the final startup speed will thereafter simply be determined by the CPU-intensive complexity of the particular decompression methodology used (e.g. xz or zstd or whatever) and the power of the CPU (and RAM speed and so on). But on modern systems the likes of SSD drives have fast I/O anyway, so no point really pre-loading compressed sfs files into RAM in that scenario (just wasted precious RAM). The dilemma is that older systems that may be using slow external media tend to have low amounts of RAM anyway and if you copy2ram large compressed sfs files the loss of RAM for actual system use can mean there isn't enough RAM left to run big progams like modern browsers... using conventional swap won't help since that is accessed on the slow hard disk again... using zram swap can help a bit though.

If a person happened to have a system with plenty of RAM (where the RAM was far faster than the external storage media) then the fastest starting firefox would be achieved by actually pre-loading uncompressed firefox (binaries and support files) into the like of a RAM disk at boot time (not simply pre-loading sfs files or mounting them). Since that would require loading in all the already uncompressed data blocks and so on of the program, the penalty would be the delay waiting for it to all load in at boot time...

Summary: on older hardware, which often includes slow I/O storage devices (or indeed slow usb sticks...) some type of compressed filesystem can result in faster application startup compared to uncompressed apps on disk. However, decompression itself is a bottleneck, which is why on old systems the likes of lz4 often made systems far more responsive compared to the likes of gz and certainly compared to xz on such machines. Certainly lz4 files were pretty large, but nothing like as large as uncompressed - so it is a bit of a balancing act. Fortunately modern zstd provides compression ratio pretty close to xz (a bit larger for sure) and decompression speeds similar to lz4, hence if the kernel supports it, provides a currently optimum solution. On fast modern machines the difference will hardly be noticeable from human perspective though. Nor is distro download size or the amount of initial RAM it takes up since modern machines tend to have tons of RAM and huge external storage (most of the time anyway).


Re: Opening application archived as an sfs versus opening it from a folder

Posted: Fri Apr 14, 2023 11:24 am
by pp4mnklinux

I agree with you, @amethyst but for me (I've got different puppies installed) the best option is having a folder with all the portable apps, and I only need to create a shortcut in my sistem to access it.

The best solution:

7 puppies installed.

ONLY ONE BROWSER with 7 direct access (one per system)

I dont nedd tonuse Firefox sync, Chrome sync.... make 7 configurations, remember passwords, mails....

Only one time .... 7 uses.

My vote for everything portable.

CHEERS

amethyst wrote: Fri Apr 14, 2023 6:52 am

Okay, I've mentioned this before but I think rather important as this impacts system performance. Scenario - On /mnt/home I have an application (let's say it's Firefox) archived as an sfs file. On the same partition I have the exact Firefox but in an uncompressed state in a folder (like a portable for example). Now, the application starts much faster for me at first run in the form of a loaded sfs (contents NOT copied to RAM) file than running the executable from the folder as mentioned. Since my sfs files are all pre-loaded during boot, this is a win/win situation for me. This may not be so noticeable with fast storage media, lots of RAM and a fast machine but definitely something to consider when your hardware is not so flash. Of course, if you have lots of RAM you can copy your sfs files to RAM at bootup which will make operation even faster (but then the frustration of longer boot time to allow for the copying).
The other thing of course is to use the fastest decompression method for your sfs files. Apparently zstd is best for the new Puppys (although I have read that multi-threaded gzip is even faster than zstd) so one should use that. For older Puppys use gzip. I've also experimented with different compression levels. This makes a noticeable difference for compressing speed but not really when decompressing. So use the fastest compressor for sfs's but compress with the highest level (will be slower compressing, but smaller size and you are probably only going to compress once anyway). Theoretically, an sfs compressed at a higher compression level should decompress slightly faster as when compressed at lower level however this is not really noticeable for me. I've also experimented with making sfs's using the non-compression data options. This results in quick "compression" (nothing is basically compressed) but interestingly does not make any noticeable difference when decompressing the archive. Some food for thought...


Re: Opening application archived as an sfs versus opening it from a folder

Posted: Fri Apr 14, 2023 11:37 am
by amethyst

As said, this will be noticeable on an older, less powerful machine. Wouldn't really notice on a fast machine (fast cpu, fast disks, lots of RAM), I think. On my old 2gb ram, 2GHz duo core CPU laptop, it's very noticeable.


Re: Opening application archived as an sfs versus opening it from a folder

Posted: Fri Apr 14, 2023 11:49 am
by amethyst

The best solution:

7 puppies installed.

ONLY ONE BROWSER with 7 direct access (one per system)

I dont nedd tonuse Firefox sync, Chrome sync.... make 7 configurations, remember passwords, mails....

Yes, portables are cool, however, I do not use portables but sfs files and I also share the same browser settings beween different Puppys. All you need to do, is to move the cache files (from .cache in/root) and the configuration files in /root to somewhere in /mnt/home and symlink it back. Takes 5 seconds. So, for every other Puppy you run, you just symlink back from the location you have moved it to. Quick, you only do it once for each Puppy. Advantage is that you have your browser in a read-only SFS format and you are saving space on your disk...and it may operate quicker as mentioned.