Alternative SFS Load for testing

[fredx181]

*** Alternative SFS Load for testing ***

UPDATE 2023-02-14, new version 1.2 attached below, see changes and fixes info here: viewtopic.php?p=81483#p81483

load-sfs-fuse-1.2_0.x86_64.xbps

load-sfs 'on the fly' alternative

(47.17 KiB) Downloaded 60 times

Here's an alternative way to load an SFS 'on the fly' (on system with overlayfs) by using unionfs-fuse and chroot.
(unionfs-fuse binary is included in the package, it's not in the Void repo AFAIK)
Installing this should add a 'Open with Load-sfs (fuse)' entry in Thunar (see pic below), or run from terminal e.g. loadsfs-fuse </path/to/my.sfs>.

NOTE, just sharing this for testing the idea of this alternative method, the application may not be finished yet.
(sometimes (annoying) a 'Volume' icon appears on the Desktop after clicking 'Quit' (edit, should be fixed in v1,2), but only sometimes, difficult to debug, perhaps a timing issue, don't know yet how to fix).

About the included SFS Load in KLV-Airedale I have my doubts on second thoughts (as I said earlier).
The sfs-load 'symlink' method is fragile IMO, may not always work as expected and hasn't been very much tested.

This alternative way is a more safe approach IMO.
It works by creating a mount in /mnt for the <app>.sfs and another mount of / (filesystem) (by unionfs-fuse) merged with the <app>.sfs.
Then by 'chrooting' into it, appearing a small 'Applications' menu (YAD gui, see below pic) to be able to run the application(s) included in the SFS.
The SFS will NOT REALLY be merged in the system, just a merged mounted directory in /mnt.
When clicking the 'Quit' button all will be unmounted again as it was.

Here's a gtk-pipe-viewer SFS that should work on KLV, and may be good test (includes mpv too) (pipe-viewer from 'trizen' https://github.com/trizen/pipe-viewer).
https://dl.dropboxusercontent.com/s/s9y ... v.sfs?dl=1

Screenshot.png (75.15 KiB) Viewed 1551 times

Screenshot(1).png (12.92 KiB) Viewed 1551 times

[wiak]

That's interesting, makes me wonder though if the overlayfs version of similar wouldn't also work with frugal installs since as I've said it is permissible to re-use filesystem layers in a new overlay. In the following I successfully loaded an sfs of gimp on top of a full install a few years ago, but may be advantage to using unionfs instead I don't know about since I've tried it:

https://oldforum.puppylinux.com/viewtopic.php?t=115915

Note that this HowTo is intended to take away some of the mystery of how Linux distros can use overlayfs mount option (or aufs can be used with minor tweak to what follows) to implement load-sfs and changes/persistence facilities. In particular I wish to show how such facility can be used with a full install despite some myths going around that such facility is only useful for frugal installs.
...
since most users on Puppy are using frugal installs and aufs rather than full installs and overlayfs, this howto will maybe only be of interest to a few in its current form. Easy as I say to modify for aufs and/or frugal install situations as long as you understand the principles of operation illustrated in this example (or make a test full install of Void Linux and try it... as shown!).

Scripters can of course easily then implement all of this with a user friendly gui frontend... I'm using overlayfs, but can be easily changed to use aufs instead (I will show the minor alterations for that later along with same for frugal installs, though both these alterations are trivial really. I used overlayfs because it has been officially adopted for Linux kernel and aufs wasn't available by default on the system I used).

As you can see at my old post, I had reserved space to write more about that (including for use on top of frugal installs), but I forgot all about it and never did! Actually I never used sfs-load facility other than quick tests so I probably didn't value the idea as much as I maybe should have.

[fredx181]

That's interesting, makes me wonder though if the overlayfs version of similar wouldn't also work with frugal installs since as I've said it is permissible to re-use filesystem layers in a new overlay. In the following I successfully loaded an sfs of gimp on top of a full install a few years ago, but may be advantage to using unionfs instead I don't know about since I've tried it:

I think your example in the old forum does basically the same, I used unionfs-fuse mainly because I had the script already using unionfs-fuse (included long time ago already in DebianDog (sfs-load for full install)) but made some changes (for KLV). And as you say, can be used in a full install too.

...makes me wonder though if the overlayfs version of similar wouldn't also work with frugal installs since as I've said it is permissible to re-use filesystem layers in a new overlay

Depends what you mean by permissible, AFAIK it's not possible to add a new layer (sfs) to the overlayfs stack (after boot), like aufs can (sfs-load 'on the fly' as it works with aufs on many puppies).

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
So this alternative method we may call a 'poor man's sfs-load', as the applications in the SFS are not really integrated with the system (e.g. do not show in the default Menu).
To really integrate in the system the SFS must be loaded by placing in the frugal install folder and reboot.
(some additional info, forgot to write in first post)

[wiak]

...it is permissible to re-use filesystem layers in a new overlay

Depends what you mean by permissible, AFAIK it's not possible to add a new layer (sfs) to the overlayfs stack (after boot), like aufs can (sfs-load 'on the fly' as it works with aufs on many puppies).

No I don't mean like aufs can add new layer to existing stack. Certainly overlayfs can't do that. What I meant is that you can take layers that were used in a previous overlayfs stack and use them in a new layer construction on an already running frugal installed system.

[rockedge]

Testing this today as thoroughly as possible

[fredx181]

Testing this today as thoroughly as possible

Ok, thanks, just some more info that I didn't write in first post:
- The loaded SFS doesn't leave any traces behind in the system (the mountpoints it creates in /mnt are gone after reboot), except (possible) configuration files in ~/.config or wherever a program stores it, but that's how it should be IMO.
- The "xterm" entry in the "Applications" Menu can be used to run command line programs (inside the SFS, that don't have a .desktop launcher, so not appearing in the Menu).

Nowadays SFS's are not used much, I think (specially not in KLV), so I think including a more safe way to load an SFS 'on the fly' is better, so there's anyway an option for those who want to use it.

[rockedge]

@fredx181 some tests underway.......

So far the SFS loading is working. I did end up with 2 apllication windows and working out how it works with multiple SFS's loaded.......

Screenshot-44.jpg (66.82 KiB) Viewed 1281 times

Also to note the 2 stray volume icons on the desktop. Logging out and in does clear them. I have also caused this to happen with other scripts in KLV and in those cases I don't really know why, and then of course, how to get rid of them!

I do like this SFS load method and seems to work well so far.

[rockedge]

KLV-Airedale-rc10.1 with two SFS packages loaded, TextMaker and gtk-pipe viewer both run flawlessly.

Notice that TextMaker which was loaded first, shows up on the gtk-pipe_viewer SFS-Load window. Test what will happen when the original TextMaker window is closed, if it will still be loaded until the second window is closed? The TextMaker icon is correct in the first yad window but is not detected and or displayed in the second yad window.

UPDATE: the test shows that closing the original TextMaker load window does umount and removes the SFS. The second SFS remains loaded and operational.

Screenshot-02.jpg (45.52 KiB) Viewed 1261 times

[wiak]

I'm finishing off a new test FR initrd under development at the moment.

The current 6.0.1 release has proved to be very stable, so I'm naming the new one version 7.0.0 rather than simply a new rcN release of 6.0.1. In particular, though not many code line changes are involved (well, there are a few...) there are some fundamental changes and any alteration of the critically important initrd/init comes with the risk of unforeseen (by me) consequencies. Nothing at all being changed that should effect major user utilities such as save2flash or remaster though.

The problem with any development of the FR initrd actually stems from its flexibility - some of the mechanisms it provides may not be used much, but they still have to be taken into account; for example, putting changes into alternative partition/directory than the bootfrom partition, putting the numbered sfs and/or numbered uncompressed-addons into alternative partition/directory. Any fundamental code change has to take all of these differing mechanisms into account, and such thought sometimes confuses my brain big-time! However, the new initrd seems to be functioning as designed in early tests. Nevertheless, it will take a lot of testing and probably by many people in different use scenarios to see if all is still well. I won't myself likely test all possibilities (there are simply too many). In the meantime therefore, my plan is to use the new initrd only in KLA-XFCE for a while, since I'm using that distro actively at the moment, and also via Ventoy. Whilst I've indicated I am generally more concerned for all to work well for normal frugal installs and for Qemu, I do now use Ventoy for quick tests so that encourages me to try and keep that mechanism working well with KL. I am not using SG2D at all - I find it messy, even though its use of a loopback.cfg makes it a bit easier to deal with via initrd code; pity Ventoy didn't also use loopback.cfg, but for some reason or other its designer did not utilize that method.

Since I'm mentioning Ventoy, I will point out a potential issue that can arise:

Ventoy has its own very intensive search routines to find isos on the system. Seems they can be anywhere. Problem arises if you have two identical isos located in different locations on the system. Ventoy will include them both in its boot menu. But... when you select the one you want to boot from, Ventoy does not export any information about which one or where it is (unlike SG2D which does export path, but not partition of the iso). The iso being booted thus has to itself work out where Ventoy booted it from. Maybe there is an easy way I don't know about, but currently that means KL initrd has to do its own search for the iso stipulated in its grub.cfg - PROBLEM is that if there are more than two isos of the save name/label stored on system, there is no guarantee the one it finds is the same one that Ventoy found that started the whole process off... KL initrd has no connection with Ventoy code so they do things their own way. Result can be that a different iso can end up booting than the one Ventoy originally booted to find its grub.cfg menu configuration. That can cause problems also related to what partition gets mounted to read the iso. Only way I can see to avoid this lack of sync between Ventoy and the distros it is booting is to make sure you don't have copies of save bootable isos on your system, if you see what I mean. If you don't see what I mean, you will find out one day! Aside, from that potential issue Ventoy seems to work well with KL, though whether Ventoy will keep being developed is another matter altogether...

Note that the code I have included inside the initrd to handle the likes of SG2D and Ventoy is separated off and not used at all for normal frugal install so has no impact on boot time or performance of normal frugal install case.

The reason I am making these code changes is to bring the likes of alternative location for changes save persistence and alternative location for NN addons brought into line with handling of normal bootfrom location - helps particular with how usefully they become available in filemanager and its side panel (similar to recent improvement I made to that for that bootfrom partition appearing now in Thunar sidepanel). Also, when iso being used, the contents or the partition it is booted from are now displayed in Thunar side panel, as are the inner-contents of the iso itself (read-only) - for that case will be found in /mnt/layers/iso for convenience (filemnt that iso wouldn't work since it is already mounted). In terms of number of lines of code, the initrd/init doesn't get any bigger really - it is just a little bit of re-organisation and only at a few key locations - overall efficiency/performance should be identical I think.

[Clarity]

I think I follow your observation of Ventoy, but have a couple questions for both understanding and research

Questions: I think you mean

1. if the same ISO filename exist in different folders on the Ventoy partition at its boot, it will list BOTH. Is that what you meant?

2. The Ventoy info available to the booting system is different than that provided to the system by SG2D?

On the first, without testing this, I kinda can see that. Yes, Ventoy is Indiscriminate in its searching its boot drive, but to my knowledge, it ONLY searches its own boot drive and NONE OTHER. I'll test and report back. In my past, I NEVER have done that as on each USB, no matter which ISO file booter I use, I have downloaded and kept ALL ISO files in a single folder; namely BOOTISOS on the partition. Ventoy does have a mechanism where you can store ISO files on a system drive, but that entails the running of one of its utility to create the 'reference file' for EACH ISO stored on a a system drive into a folder on the Ventoy boot drive. But, as I remember, those filenames show that they are on the system drive when viewed in the Ventoy Menu list before user ISO choice is made.

Searching for ISO files to present to the user for selection differ between the 2 boot methods you mention;

• Indiscriminate searching for ISO files on ONLY its boot disk is where Ventoy differs from SG2D.

• SG2D discriminates looking across the system for any folders named by the varied combinations of names that can be derived the lettering of 'bootisos' (upper/lower case). As such it is NOT restricted to only the boot folder as Ventoy is.

ON the second, I thank you for that information as your layout descriptions has help me see more of what the initrd sees when it boots any system whether via one of the "ISO file booters" or not.

Thanks...back soon on #1

Edit: Confirmed that Ventoy on its initial Menu does NOT provide an indicator to determine which partition folder the identical named ISO files reside. To determine their location on the partition, one needs to use "F3 key" to see which folder they resides and select the ISO file from the folder the user has placed.

Of course this identical can only happen by user choice. So far, ALL RECOMMENDATIONS has been to place and keep ALL ISO files in the BOOTISOS folder to avoid duplication, and confusions of this sort. Distro filemanagers do not allow duplicate filenames in the same folder so this cannot happen if one is using a Linux system.

I think I will continue to emphasize in my guidance where I instruct users in my directions to insure they keep all ISO files in a folder BOOTISOS for both good housekeeping and to insure duplicate filenames to not show up in the Ventoy Menu listing to avoid confusion.

SG2D does NOT suffer from this possibility as it list ISO file locations that are in BOOTISOS no matter where on the PC if finds them. Thus there is location information for every ISO file it shows. This is one of the differences between the 2 boot-helpers.

[wiak]

Yes, since KL isn't part of Ventoy, it does not know where Ventoy is booting from per se. Ventoy presents a menu choice for an iso and then, once that iso is selected, Ventoy mounts it, reads the grub.conf that is inside the iso, and presents that as the final boot menu. That results in KL booting. However, the iso that Ventoy temporarily mounted for its own use does not remain mounted for KL so KL has to find the iso all over again and mount it so that it can set up the overlay. So the problem is that KL has to search for the iso itself, all over again, and since it does not care about Ventoy (or how any other boot mechanism selects isos) it searches all partitions looking for it (in either /BOOTISOS location or in / location of all searched partitions. And of course if there happens to be more than one identically named KL isos the best KL can do is assume that the first one it finds is the one to continue booting with...

SG2D on the otherhand, can find isos all over the place in the locations you mention, but it tells KL via an exported grub2 variable the path to the iso (but, unfortunately, not the partition that path is in). So KL searches for the iso again - this time in order to determine the partition for that provided path. Once again though, if there happened to be two identically named isos on the system in path /BOOTISOS but same in different partitions, KL will assume the first one it finds is the one to boot. It will boot, but may have unexpected consequencies. For example if saving back to Sessions on bootfrom partition, the session saves will end up on different partition than user probably expected... the overall side-effect possibilities are tricky to predict so best simply to avoid any such duplication of same iso stored somewhere. It isn't Ventoy or SG2D you have to worry about, it is KL itself will select the first correctly named iso it finds... and who can blame it?

[Clarity]

Thanks for that understanding of initrd processing.

Summary

1. So in Ventoy case, KL receives an ISO filename, but not its path;

2. while in the SG2D's case KL receives a ISO file's path, but not the partition info.

3. without the presence of either ISO file-helper, booting ISO file on bare-metal/QEMU via KL's starts without any environmental info,

In all cases, KL knowing the ISO filename will search all system's partitions for 1st occurring match and will use that.

Is that a correct external understanding?

[rockedge]

Is that a correct external understanding?

Yes.

[fredx181]

@wiak @Clarity I can't see any connection with what you're writing about and the thread subject.
@rockedge Thanks, yes, loading multiple sfs's shouldn't overlap, I think it can be fixed, will look at it.

[Clarity]

Hi @fredx181 I agree that this got a little off-topic from your SFS discussion to how KL handles its booting. I will post any future of KL initrd processing elsewhere vs continuing here.

Thanks @rockedge. I will include this knowledge as I do when giving/assisting/writings in boot-helpers use by members.

[rockedge]

@fredx181 nice!
When I have both gtk-pipe-viewer and TextMaker loaded, with TM being loaded first, TM shows up in the gtk-pipe-viewer window and can be selected there and it will start. When the orginal TM window is closed and TM is removed, it still is in the second yad window but of course does not work because it is removed by closing the first window.

So far I have both SFS-Load systems installed. I am testing mixing loads and so far no problems. I am loading several SFS's using the fuse loader only and this is working. Then loading several SFS's just using the original SFS-Load and also no problems.

I can imagine trouble when mixing both methods and loading SFS's but so far I haven't had an issue.

Even hard stopped to see what happens.......seemed to get cleaned up and the fuse SFS load left no trace no problems.

[wiak]

@wiak @Clarity I can't see any connection with what you're writing about and the thread subject.
@rockedge Thanks, yes, loading multiple sfs's shouldn't overlap, I think it can be fixed, will look at it.

Oops, sorry, put my post in wrong thread. I had just noticed 'KL-Dev_Work' and thought that was the thread when actually it is a forum section. My mistake not realising the title indicated a separate thread.

[fredx181]

*** Updated load-sfs-fuse package ***

New (v1.2) attached at first post, changes and fixes:

- When loading more than one SFS, now they have separate applications menu (rather than 'overlapping' as it was with previous version)

- Fixed that a useless 'Volume' icon appeared on the Desktop (happened only sometimes, but annoying)

- Fixed that in some circumstances an SFS was not properly unloaded, because of accidentally forgetting to close an application before hitting the Applications menu 'Quit' button
Now it gives a message IF some process(es) still running and will kill these, required for proper unloading, see pic below.

Screenshot(2).png (100.83 KiB) Viewed 1127 times