wd_multi KL frugal installs howto

[wiak]

EDIT: See this post for quick SUMMARY of using wd_convert2multi (for installing multi-instances from existing frugal installs): https://forum.puppylinux.com/viewtopic. ... 401#p99401
See below posts for using w_multi (for installing multi-instances from isos)
EDIT: version 1.0.0 -rc4 of both wd_multi and wd_convert2multi are now available. Too late to post more about them than quick info/download/use instructions here: https://forum.puppylinux.com/viewtopic. ... 334#p99334
NOTE that the use of wd_multi_grubplug is a temporary demonstration mechanism only for using wd_multi with Puppy Linux or perhaps other forum distros. When I have more time later I'll build that capability into wd_multi and so on itself - that will involve wd_multi detecting which kind of distro it is processing and adjusting the grub stanzas appropriately. For the moment you need to disable the wd_multi_grubplug (via renaming) whilst processing KL/FR distros and enabling it (naming it exactly as wd_multi_grubplug) when processing recent Pup distros.

DISCLAIMER: wd_multi and wd_convert2multi scripts need to be run as root user and are thus potentially dangerous - any mistake in the scripts may remove parts of the filesystem you are running on; USE AT YOUR OWN RISK but works for me... Safer to run from KL/FirstRib RAM2 save on demand mode only (or Pupmode 13)

A good (safer) distro to run wd_multi from is, funnily enough, probably a wd_multi created instance running in wd_changes=RAM0 or wd_changes=RAM2 mode!!! Afterall, you may have lots of these available after using this script...
=======================================================

Following on from the "Sharing KL/FirstRib components between KL distros" this thread announces a practical build script that automates the construction as many frugal install KL distro instances as you wish to have at your fingertips.

Each instance (and you can have hundreds of such frugal installs, thousands even, in a few GiB of storage space) can be booted as an individual distro (exact grub stanza provided) with its own upper_changes folder so each can be customised into whatever shape and form you wish.

Short of time, so see this link for some details: https://forum.puppylinux.com/viewtopic. ... 151#p99151

The usage demonstration given in above link post was for a total of 300 individually bootable frugal installs (three different KL distro varieties) taking up a total space of only 2.3 GiB on my hard drive (i.e. under 8 MiB for each distro!).

Usage (all commands done as root user):

1. Create an empty directory on the Linux formatted partition you want to boot these from. For example:

Code: Select all

mkdir -p /KL_multi  # the folder name is actually irrelevant...

2. Download the wd_multi script into that empty directory using command:

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_multi && chmod +x wd_multi

3. Download or copy in any or several KL distro isos into that same /KL_multi folder (or whatever you called it)

4. In a terminal, at that directory, run the command:

Code: Select all

./wd_multi <n>

where n is the number of individual frugal install instances you want of each distro variant. If n argument is omitted, default is n=1.

So, for example:

Code: Select all

./wd_multi 100

would give you 100 individual bootable frugal installs for each iso type you are including.

Each frugal install instance directory includes an automatically created grub_config.txt file containing exact grub stanzas that you can put in your grub config to immediately thereafter boot that instance.

Please enjoy your potentially thousands of distros that fit in but a few GiB of disk space (or small usb stick)!!!

NOTE: I've only just briefly tested this, so please let me know if any issues. EDIT: Just used wd_multi 1000, using 3 isos, to make total of 3000 frugal installs (still just taking up total of 2.3GiB). My wife thinks I've gone crazy: "what are you going to do with them all?", she asks. Good question...

[wiak]

Later I'll make a variant of this specially for KLfull2frugal so you'll be able to have hundreds or thousands of frugal install 'instances' of your full installed mainstream distro (like Linux Mint, Zorin, whatever).

That's important for my usage since work matters require me to be in big distro quite often but this KLfull2frugal multi-instances approach will give me plenty of scope for experimenting even with the big distro without breaking the underlying full installed filesystem.

But I'm really enjoying wd_multi already because so easy to just drop in any and all KL iso variants and desktop types I want to play with. Using 'instances' takes away all danger of corrupting any important save folder, but notice that the upper_changes of a different 'instance' can also be loaded as an overlay by linking into the other instances frugal install directory. As an example to show how powerful this is conceptually:

Code: Select all

cd instance49
ln -s ../instance21/upper_changes 50uc21

Here, the read/write upper_changes from instance21 is being used (following a reboot) as a read-only rollback filesystem at layer 50 of instance49 distro... That's the power of FirstRib being able to use normal directories (uncompressed) as overlays. Obviously you have to make sure that other instance upper_changes is compatible with the instance you are booting into and make sure to avoid package manager database corruption, but since they all start from same base frugal install (for any given iso) it is perfectly possible to keep the pkg manager database clean; with care... Doesn't matter if you don't since can always just delete the current upper_changes or use a different instance!

[rockedge]

@wiak Using a QEMU VM, I just successfully ran the script on a virtual hard drive qcow2 formatted as ext4 with a Grub4Dos supplied boot loader.

Made a directory called /mnt/sda1/KLV-multi then placed in it the wd_multi script and added the KLV-Airedale-RT.iso I just made.
Then opened a terminal from inside the directory and used:

Code: Select all

./wd_multi 7

Screenshot.jpg (44.74 KiB) Viewed 6582 times

/mnt/sda1/KLV-multi/instance1 with a upper_changes is 62 M in total size.

Screenshot(11).jpg (44.25 KiB) Viewed 6578 times

[wiak]

@wiak I just successfully ran the script on a virtual hard drive qcow2 formatted as ext4 with a Grub4Dos supplied boot loader.

Made a directory called /mnt/sda1/KLV-multi then placed in it the wd_multi script and added the KLV-Airedale-RT.iso I just made.

That's good to hear, rockedge. I'm also using wd_multi instances all the time now. It was tricky writing that script, particularly integrating it with correct internal grub_config functionality. But despite me making it, I was actually amazed at the result - really powerful way to have multi distros individually changeable. I also like being able to simply use a different instance if I break something in one I'm using! The thought that I have tons of them makes me feel like a kid in a sweets shop!!!

[rockedge]

Frugal system /mnt/sda1/KLV-multi/instance2 has not been used yet though has menu.lst boot stanza's, is 2012 bytes in total size!

That is the total size of the frugal system install instance2 = 2012 bytes

Screenshot(14).jpg (48.45 KiB) Viewed 6545 times

[williwaw]

creating a link with a name preceded by a number is all it takes for a new upper_changes to be created, provided the renamed link exisits in a new frugal (which can be comprised of other links)?

creating a restore point should then be as easy as adding an additional question and if/then statement to save2flash.

if the user chooses to make a restore point, then a new frugal could be created with the original name after the existing frugal is renamed

[wiak]

Frugal system /mnt/sda1/KLV-multi/instance2 has not been used yet though has menu.lst boot stanza's, is 2012 bytes in total size!

That is the total size of the frugal system install instance2 = 2012 bytes

Yes, it all feels somehow magic/crazy... doesn't it... but it isn't. 2012 bytes... makes me laugh, but that's exactly what the alternative pristine 'frugal KL install' is... and can even share upper_changes with other instance or use an other instances upper_changes file as a rollback (just a link to it with filename anything you like as long as appropriate 2-digit layer number at front link name).

A simple way of swapping to a different instance prior to reboot is simply to rename the other instance to the one already configured in grub.cfg. For example I was booting instance1 so I just changed that folder name to instance1b and renamed instance2 to instance1, rebooted, and now I'm in instance2! Actually, I just realised I'm currently booted into an instance2 of KLV_sway running Firefox full screen - I really love tiling wm/compositors now - used to think they were cumbersome, but now that I understand the tiling tricks (including float window ability and resize) I find it amazing and low-resource usage. I blame Windows 95 for training my brain to want floating/stacked arrangement, which is nothing like as quick/efficient to use in practice.

I'm sure there are tons of 'tricks' others can think up with this 'big' wd_multi system of distros - must be really interesting experimenting with it in Qemu qcow2 arrangement.

The experience reminds me of a scripting system I devised long time back (1999...) when I had a script building a complete network of little user mode linux virtual machines, some of which I auto-configured during the build as routers and so on. I used that as a virtual network laboratory for my students of the time to experiment with data comms protocols and so on; had hundreds of auto-made virtual uml machines and the students had remote access (dialup back then...). It was great - I even did a conference presentation on it and a case study that was published on the user mode linux website back then. The similarity isn't the technology being used but just the magic-feeling nature of auto-making so many distro machines in such a small size; the uml instances all used a cow mechanism to a common base distro (was using a debian image then), so hundreds-of-internetworked-uml-machines build size was also tiny.
EDIT: Funny, just checked, the old case study from over 20 years ago (job I had for a year or two on moving from Scotland - can't say it was my favourite job though...) - the uml case study, is still out there (turns out it was 2001); second one down at: https://user-mode-linux.sourceforge.net ... udies.html
Fact is, I could develop wd_multi to include a plugin facility that could be used to configure IP subnets of these potentially thousands of machines such that some were routers and so on in different IP subnets (so just like the uml creation I once did) - in fact some of my old uml scripts would contain the required code for making/controlling the build and interconnection of all the IP subnets - involved some binary-manipulation tricks to automate versatile IP subnets; makes a great teaching lab: can even insert special components with delays and so on to simulate satellite link connections (VSATS) which is what I used to work with in real life research group working for European Space Agency contracts. Experimenting with such inter-network configurations is fascinating, especially when simulated sat links are involved since that introduces not only big delays but also link noise effects so really stresses out TCP/IP protocols so good then to test new algorithms to work way round these complex issues (that's what we were doing back then). But, really, uml was better for what I describe than Qemu-type situation since the way uml works it uses very little RAM despite tons of uml machines.

Anyway, it's great to have so many immediately inter-connectable machines and power! Next year I'll think about developing that to include inter-networking per the above description.

[wiak]

which can be comprised of other links?

We need to experiment with that, but off the top of my head I think that should be okay.

[wiak]

creating a restore point should then be as easy as adding an additional question and if/then statement to save2flash.

if the user chooses to make a restore point, then a new frugal could be created with the original name after the existing frugal is renamed

Yes, the wd_multi setup definitely provides great scope for alternative way of controlling rollbacks with restore points as you say.

[rockedge]

@wiak I have been thinking a while on how to apply KLV to be the base systems for a virtual machine Computer Cluster, simulating something similar to a Raspberry Pi cluster computer set up.

Now setting up the cluster components for QEMU machines to simulate cluster computing is a breeze. No idea if it's practical or useful, to have a cluster computer simulator......but that is beside the point!

EDIT: I symlinked the upper_changes and work directories from instance1 to instance2 and........it works!
Entire frugal install including the 177 M upper_changes persistence folder symlink'ed is 2104 bytes in total size!

[wiak]

I'm going to add a DISCLAIMER to the first post regarding use of wd_multi script. Of course there is always a disclaimer implied in all contributions and especially build scripts because not every eventuality can be tested and unforseen mistakes can be made. But a DISCLAIMER is necessary when any script is run as root user and wd_multi generally needs to be run as root user (I think... though haven't thought about it much).

Running any script as root user is of course potentially dangerous since if the script says mv something or rm something, for example, if an error in paths is made (either by error in the script or because script is run from an unexpected location...) then anything might get remove. Let's just say, such disasters happen to me frequently because I'm experimenting with previously untested new scripts.

I keep telling myself that I should only do such work when booted into RAM such and not saving any session changes back whilst such tests are in progress. Trouble it, I'm sometimes too lazy to reboot into a save-on-demand frugal install distro when I'm already, say, in my full Linux Mint install.

Anyway, I write this post for a reason - nothing to do with wd_multi script, which I always use in the same way and no issues have occurred, but if you run it from another directory than I've suggested then I have no idea what the result may be (you will be running it as root user afterall, so I suggest only running it on a frugal installed save-on-demand distro so any disasters occur only in RAM!!!

[wiak]

wd_multi is proving pretty handy for me. In fact I'm now always using it to install any KL iso since why not?... The multiple instances that result take up really no extra space but are really useful the way I work.

I am now working on a second variant that takes an existing frugal install (rather than an iso of a KL distro) and will make similar multiple instances out of that. i.e. for FR-based frugal installs that have not been packaged into iso form.

That variant will not only allow me to take some of my existing KL frugal installs and turn them into multi versions, but will be particularly useful for firstribit-type creations (what was weedogit).

Whilst none of the following is essential, they can improve the system a bit:

TIPS: If you've used wd_multi you will realise that the default actual files/dirs from the iso are all unpacked into instance0. It is useful to sometimes remaster the main root filesystem in instance0 prior to use by the other instances. That is doing the likes of:

1. unsquashfs 07whatever # assuming main rootfilesystem is at layer 07 (adjust command otherwise)
2.Then use ./mount_chroot_umount.sh script to update the resultant squashfs-root (unsquashed root filesystem).

Code: Select all

./mount_chroot_umount.sh squashfs-root

Exit out of that chroot when finished making your alterations. (If instead you used "./mount_chroot.sh squashfs-root" make sure you use "./umount_chroot.sh squashfs-root" afterwards to clean up mounted filesystems involved)
3. Finally use mksquashfs to squash it back up to sfs form again (making sure you use the same sfs filename as previously so the symlinks in the other instances find it correctly). It is probably worth using zstd 19 compression when resquashing since will the distro will end up more responsive for each and every instance...

Code: Select all

mksquashfs squashfs-root/ 07KLblahblahblah.sfs -comp zstd -Xcompression-level 19 -b 512K
rm -rf squashfs-root

A second tip is that you can use wd_multi script many times. Just remove any old isos and insert new ones and run the script again and the new ones will be processed alongside any previously done.

There are some variants to wd_multi script that I feel would be useful generally, but for now best to just either do some manual mods to the structure or write your own mini-scripts to fix any symlinks up depending what you are doing. Later it may, as I've said earlier, be appropriate to include simple plugin support in wd_multi (for example, for the case where you want to make instance0 a "pseudo-fullinstall" type of arrangement for easy upgrade/update, but it remains important not to break package manager database whatever is done).

[wiak]

I'll release wd_convert2multi shortly.

That's for taking an existing KL frugal install and making a KL multi out of it.

The existing frugal could of course be an existing instance0 of some previous wd_multi creation.

In practice wd_convert2multi will often be even more useful than wd_multi itself because the new instance0 can include a previously configured upper_changes folder, which means provides an (optional) inherent update/remaster facility.

That previous KL frugal install upper_changes moved into new multi build structure instance0 will be symlinked as a read-only 2-digit upper layered addon in ALL the created instances by default. That has powerful amazing usefulness since any upgrades later done to the new instance0 via its read/write upper_changes will automatically be used as that read-only numbered layer provided to all other multi instances. Trivially easy to upgrade the multi-structure therefore. Normally, instance0 doesn't need used at all except when updating/adding to its upper_changes folder. A simple multi form of structure remastering in other words.

Of course you can unlink that instance0 upper_changes include in any other instance just by deleting the numbered symlink to it...

[wiak]

The wd_convert2multi script to convert any already installed existing KL/FR distro into a wd multi type of construction is now complete and working to my satisfaction. Will publish very very soon once I've written instructions on usage. This is even more useful, I feel, than the original for iso files, wd_multi. Feels almost revolutionary in terms of scope to me on using it, and, I think and hope, very easy to use in practice.

Yes, you can convert to mulit-instance-form any and all KL/FR distros you may already have frugal installed on your system all in one go (via a simple cut or copy and paste operation in your preferred filemanager), or add other ones later to same /KL_frugals directory (or whatever you want to call the multi-instance holding folder). The convert process takes seconds. You can call that directory anything you like, in other words, and make as many such multiKL-structure-containing dirs as you want to on the root [/] directory of the boot partition being used.

One interesting test case was to make a FR multi install of a mainstream distro that I had earlier made into a FR booted frugal install (aka way weedogit made them). It was a test with a FirstRib-ed big Garuda Sway (which is a 2.5GiB distro extracted from its iso). So did a "wd_convert2multi 27" of that FR frugal install of Garuda Sway (which I had previously updated so had stuff in its upper_changes folder) and result is that I now have 27 FR Garuda Sway instances (still just taking up total disk space of 2.5GiB).

The actual core FR Garuda Sway files are all now in instance0 along with its upper_changes. All the other instances contain a 60instance0 symlink to that instance0/upper_changes folder so that gets included as a read-only layer (60) on booting any of the other instances (instance1 up to instance26 in this exemplar and each of these other instances have there own read-write upper_changes folder (at topmost layer of the overlay). Of course the 60instance0 symlink can be deleted in any instance you don't want that instance0/upper_changes to be used as a layer.

If you've used previous wd_multi then I'm sure you understand the above, and also see just how powerful a facility this FirstRib build-concept/structure is.

As I've said, I now always install all my KL/FR distros in this multi format (takes up next to no extra disk storage space so no reason not to) and generally thereafter leave instance0 in pristine state unless updating it. That way I can use any of the other instances individually with their own upper_changes without in any way effecting the base instance0. Means that if I accidentally ever 'break' an instance (as I do now and then since doing dev work as root user), I can simply use another instance or make a clone of another instance over what was the broken one!

No more destroying my base installs by accident whilst doing dev work and great facility, per what I discussed with rockedge, also for using in a qcow2 disk for multi-distro-instance booting as virtual machines too! Both new wd_multi (very minor change) and wd_convert2multi scripts (version 1.0.0 -rc3) should be available in under an hour from gitlab (I'll publish the wget fetch command on this forum per usual once I've finished writing the usage instructions - that might not be for around 12 hours since I'm tired so bedtime approaching soon for me here...). Despite my DISCLAIMER about the danger of running any build scripts as root user, don't hesitate to try this - you may be amazed at how useful it is... but, once released, read the usage instructions carefully (simple though):

Code: Select all

./wd_multi --help

./wd_convert2multi --help

Main thing you need to remember to do is that once you cut or copy the existing frugal into say /KL_frugals, you need to give each directory you wish to then convert2multi an extension of .MKL (standing for MultiKL). Then the new wd_convert2multi shell script (which looks for that special .MKL extension) works pretty much the same as wd_multi script (where extension was .iso for that one). Note using Cut rather than Copy on same partition is very quick since the actual files don't need moved at all... However, using Copy gives you a backup copy of your original frugal install if you want that. Thereafter, my usage is not to use instance0 at all - just use grub to boot any of the other instances you want - that way, as I said, instance0 remains in its original re-usable state. EDIT: Both wd_multi and wd_convert2multi scripts are up on firstrib gitlab now, so if you are confident to use them you can fetch them from there now. I haven't written the usage instructions yet, but will publish all details tomorrow. Scripts seem to both be working fine. But be careful and read usage instructions carefully and use at your own risk per usual...

[wiak]

By the way... Yes the following are ACTUAL Puppy Linux frugal installs (hundreds) produced with this FirstRib wd_multi utility. Not firstribbed or weedogged - these are normal Pups and booted with normal Pup grub stanzas and Puppy aufs or whatever it is using save folder layering. Certainly you can't use uncompressed filesystem layers since Puppy doesn't use these - but the multi instances all work as normal Pups via this same utility.

One reason I am taking sometime to publish full usage instructions for the new wd_convert2multi for converting existing frugal installs to multi-distro-instances format (along with wd_multi for isos) is that, like FR initrd itself, I have been careful to write the scripts in as 'generic' usable way as possible.

For this particular, most powerful FirstRib-created system, I continue to push my forum-friendly (Kennel) agenda; I don't want to leave such anyone behind when such a powerful multi-distro system is available via FirstRib. I've designed it such that, drums roll... be excited and don't further suggest FirstRib contributions/discussions should not be on this forum - this system WILL work with Puppy Linux (and possible the other distros featured on this forum. You can only use uncompressed layers with FirstRib-based distros as things stand, but that doesn't matter, YES you can make thousands of independently usable recent Puppy Linux distros with same instructions and scripts, wd_multi and wd_convert2multi, per the information provided in this thread (so if you haven't read any of this because you don't read FirstRib/KL section then you may be missing out on something big).

Right now I am making version one further update to the two scripts (so will be version 1.0.0 -rc4) that will contain simple plugin facility I discussed sometime earlier (without giving any details about its use). What the exemplar plugin I provide will contain is a simple replacement of working grub_config.txt code, but for Puppy Linux grub stanzas for this system. However, the current version on firstrib gitlab of wd_multi (per post one of this thread) does in fact also work at producing multi-instance Puppy Linux distros. At the moment, however, you need to modify the grub stanzas written out by the script since currently they are for FirstRib-based distros, but everything else is fine... For example:

Code: Select all

./wd_multi 200

will produce 200 Puppy Linux usable distro instances, taking up only the storage space of one such Puppy frugal install, and each instance from instance0 to instance199 will have its own save folder!!! The trick to get the individual save folder with Puppy Linux is in the grub config as illustrated below:

wd_multi (version -rc3) will output grub stanzas intended for FirstRib-based distros so of the form (assuming using a Puppy F96-CE iso here):

Code: Select all

menuentry "KL_frugals/F96-CE_4/instance35" {
  insmod ext2
  search --no-floppy --fs-uuid --set 7d34b88e-bc8e-4bb6-86cf-d619c3b545a9
  linux /KL_frugals/F96-CE_4/instance35/vmlinuz w_bootfrom=UUID=7d34b88e-bc8e-4bb6-86cf-d619c3b545a9=/KL_frugals/F96-CE_4/instance35 w_changes=RAM2
  initrd /KL_frugals/F96-CE_4/instance35/initrd.gz
}

wd_multi version 1.0.0 -rc4 that I am working on will allow use of a simple text file plugin that corrects the above linux kernel line so will be in Puppy correct boot format including save folder location. But manually you can arrange it as stands by simply looking at above and instead using for this example build, where I'm just happening to use distro instance35 (but really you can boot any of the whole 200 distro instances available ):

Code: Select all

menuentry "KL_frugals/F96-CE_4/instance35" {
  insmod ext2
  search --no-floppy --fs-uuid --set 7d34b88e-bc8e-4bb6-86cf-d619c3b545a9
  linux /KL_frugals/F96-CE_4/instance35/vmlinuz pmedia=ata pfix=nocopy psubdir=/KL_frugals/F96-CE_4/instance35/ psave=7d34b88e-bc8e-4bb6-86cf-d619c3b545a9:/KL_frugals/F96-CE_4/instance35/
  initrd /KL_frugals/F96-CE_4/instance35/initrd.gz
}

Note the 'psave' trick, which tells Puppy Linux where to put the save folder for this particular instance35 distro. There are other ways of doing that.

How much disk space do these 200 Puppy F96-CE distro's take up altogether you may ask... Answer is 537.7 MiB in TOTAL! And each of these 200 F96-CE Pups are individually bootable with their OWN save folder. You can have thousands of instances if you want or need in similar total storage space. Of course whatever you add to their individual save folders does increase overall size accordingly, but break one instance in your experimenting ever and just use another...

Tomorrow, probably, I'll release the wd_multi ver 1.0.0 -rc4 with a grub-related plugin for recent Puppy Linux (doubt the old Pups work with psave parameter - I wouldn't know since long time since I used Puppy, but that's another reason I'll release with Puppy-specific PLUGIN so Pup experts can modify the plugin to their hearts content. May well be that the likes of EasyDog, DebianDogs, and FatDog will work too - just modify the grub output (or current 1.0.0 -rc3 version) or plugin provided for Puppy along with -rc4.

[wiak]

EDIT: wd_multi and wd_convert2multi version 1.0.0 -rc4 are now available at firstrib gitlab site. To download use commands:

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_multi && chmod +x wd_multi

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_convert2multi && chmod +x wd_convert2multi

I haven't had time to test them much since adding in the Puppy Linux related grub plugin code.

For KL/FirstRib-based distros, just run as they are (but first check ./<relevant_script> --help for usage):

Code: Select all

./wd_multi --help

./wd_convert2multi --help

Whilst there is a special extra optional second 'levels' argument for wd_multi, that's for my experimental use so don't use it or you may damage your system. So just use the likes of:

Code: Select all

./wd_multi n

where n is the number of distro instances you want in your multi instances build

For example:

Code: Select all

./wd_multi 10

which would make a multi-instance build of 10 frugal installs each from any provided Kennel's isos
CHECK THE INSTANCEnumber grub_config.txt for grub stanza to use to boot it. NOTE WELL, if doing this with Puppy Linux distro(s), with the plugin attached to first post of thread enabled, I only tested with last grub2 UUID stanza type. I leave it up to Puppy Linux enthusiasts to check the plug details - you may well need to fix it for LABELS and other stanzas. I can't and won't be checking other arguments for Puppy usage (such as usb stick booting and so on). Adjust/modify that plugin as needed and also for other forum distro multi-distro usage attempts

If you have existing frugal installs you can use these instead of isos via the wd_convert2multi utility. Then, instead of copying Kennel isos into the likes of an empty /multi_distro build directory (e.g. /KL_frugals) you Cut or Copy the existing frugal install into there instead, and then rename that clone with extension .MKL prior to running the convert frugals to multi-instances command:

Code: Select all

./wd_convert2multi n

again where n is number of instances you want in the build.

For Puppy Linux or other forum distro you need to also enable the first post of this thread attached wd_multi_grubplug. For other forum distros than recent Puppy Linux it needs modified anyway according to grub needs of these other distros - specially concerned with arranging where save folder is to be located.Enabling it just means removing the PUPPYONLY.tar from the filename. DO NOT ENABLE IT FOR KL/FR-based distros or you'll get wrong grub_config.txt information in each KL/FR-based distro instance created!

I'll post better instructions tomorrow (maybe in separate threads for wd_multi and wd_convert2multi tomorrow. Too late here just now.

This is a powerful flexible facility with lots of potential uses and variations of use. Please let me know how it goes... and specially if any bugs, but also PLEASE be careful and follow the simple use-case instructions exactly. USE AT YOUR OWN RISK anyway and especially since need to be run as root user to make the builds I think.

[Clarity]

Looks like this will work for PUPs saving/finding the PUP's save-session on a partition labeled "Persistence" where all session-saves exist in its "/Sessions" folder.

Code: Select all

menuentry "KL_frugals/F96-CE_4/instance35" {
  insmod ext2
  search --no-floppy --fs-uuid --set 7d34b88e-bc8e-4bb6-86cf-d619c3b545a9
  linux /KL_frugals/F96-CE_4/instance35/vmlinuz psave=Persistence:/Sessions/KL_frugals/F96-CE_4/instance35/ pmedia=ata pfix=nocopy psubdir=/KL_frugals/F96-CE_4/instance35/ 
  initrd /KL_frugals/F96-CE_4/instance35/initrd.gz
}

If a modern PUP, insure the SAVESPEC file is in the boot path: It steers to the Sessions folder. Then, a psave parameter on the "linux" line is not needed

Code: Select all

menuentry "KL_frugals/F96-CE_4/instance35" {
  insmod ext2
  search --no-floppy --fs-uuid --set 7d34b88e-bc8e-4bb6-86cf-d619c3b545a9
  linux /KL_frugals/F96-CE_4/instance35/vmlinuz 
  pmedia=ata pfix=nocopy psubdir=/KL_frugals/F96-CE_4/instance35/ 
  initrd /KL_frugals/F96-CE_4/instance35/initrd.gz
}

[williwaw]

psave=Persistence:/..../..../..../Sessions/

which puppies can use this parameter?
which ones cant?

[Clarity]

The stanza was corrected, above

SAVESPEC sample

Code: Select all

SS_ID='Persistence'
SS_DIR='/Sessions'
SS_MEDIA='atahd'

All Sessions for ALL distros are kept in this common folder and their sessions found by modern PUPs at boot-time. This file is a steering agent for modern PUPs session management from pristine/reboots to shutdown. It is built into their architecture, maintained by @gyrog in WoofCE GIT development.

It can be tailored as necessary. It is discussed elsewhere in the forum, but useful knowledge here for PUPs.

[rockedge]

@wiak I have set up on a QEMU qcow2 virtual hard drive formatted with one partition as ext4, LABEL=QEMU. Grub4Dos boot loader

The VM HDD directory tree looks like:

Code: Select all

/KLV-multi/KLV-Airedale-RT
/KLV-multi/KLV-Airedale-RT/instance0
/KLV-multi/KLV-Airedale-RT/instance1 
/KLV-multi/KLV-Airedale-RT/instance2 
/KLV-multi/KLV-Airedale-RT/instance3 
/KLV-multi/KLV-Airedale-RT/instance4 
/KLV-multi/KLV-Airedale-RT/instance5
/KLV-multi/KLV-Airedale-RT/instance6

/KLV-multi/F96-CE_4
/KLV-multi/F96-CE_4/instance0
/KLV-multi/F96-CE_4/instance1
/KLV-multi/F96-CE_4/instance2
/KLV-multi/F96-CE_4/instance3
/KLV-multi/F96-CE_4/instance4
/KLV-multi/F96-CE_4/instance5
/KLV-multi/F96-CE_4/instance6

When I boot into /KLV-multi/KLV-Airedale-RT/instance1 an /upper_changes is created within the directory I booted from and persistence works as expected using the boot stanza's created by wd_grubconfig.

It is important that in the boot stanza's for all of the /KLV-multi/F96-CE_4 instances include on the kernel command line the path where the save folder will be.

Code: Select all

psubdir=/KLV-multi/F96-CE_4/instance1 psave=QEMU:/KLV-multi/F96-CE_4/instance1 pmedia=atahd

Otherwise because all the instance's system files are symlinked to instance0 and unless specified, the save folder created in any of the instances 1-6 will end up being created and saved to in the directory instance0

This happened to me and so I moved the save folder to the intended instance2 and then the persistence for that instance worked, using psave=LABEL: on the kernel command line seems to work well for this to supply the path for the save folder directly during system start.

[wiak]

If a modern PUP, insure the SAVESPEC file is in the boot path: It steers to the Sessions folder. Then, a psave parameter on the "linux" line is not needed

Yes, using a SAVESPEC file was an alternative mechanism I considered in this demo implementation, but I was hoping some older pups might work with psave argument on grub command line and, for consistency/elegance of approach, I personally tend to prefer psave grub mechanism than needing extra (SAVESPEC) text file in distro bootfrom folder.

You might notice that with KL/FR distros all such alternative locations are indicated in exactly the SAME manner and no SPEC files need to be constructed at all. For example:
w_bootfrom=UUID=blahblahblah1=/pathname1 w_changes=UUID=blahblahblah2=/pathname2 w_altNN=UUID=blahblahblah3=/pathname3

As I say elsewhere, by the way, the use of a wd_multi_grubplug plugin file is just a temporary mechanism to allow Puppy or other forum distro enthusiasts/experts to sort out grub booting parameters and SAVE2SPEC-type approach prior to me later making an integrated version that does grub configs differently depending on what distro-type wd_multi and wd_convert2multi establish they are using.

[wiak]

psave=Persistence:/..../..../..../Sessions/

which puppies can use this parameter?
which ones cant?

I have no idea myself which Pup distros include this and/or alternative SAVESPEC text file with same information mechanisms. I knew about both but don't know if one or the other is available more often in older pups. I prefer using psave since this is about grub kernel line functionality (function) so better/easier to use method for automation but SAVESPEC could be used instead when available. The only two Pup distros I quickly myself tried were upup and F96-CE and with success. The wd_multi and convert2multi work with all KL/FR distros of course and wd_convert2multi also allows use of instance0 uncompressed upper_changes (save folder) as an optional read-only filesystem layer in all instances; that can't be done with Pup distros since they use sfs layers only - the multi-instances facility still very useful I feel with Pup installs.

Yes, would be good if Puppy testers indicate which Pups can use psave and/or SAVESPEC way of pointing the location for Pup savefolder.

Probably best to discuss use with Puppy distros in Puppy area since I'm focusing only on KL/FR distros here and don't myself have time to test Pups too.

[wiak]

NOTE that the use of wd_multi_grubplug is a temporary quick demonstration mechanism only for using wd_multi with Puppy Linux or perhaps other forum distros. When I have more time later I'll build that capability into wd_multi and so on itself - that will involve wd_multi detecting which kind of distro it is processing and adjusting the grub stanzas appropriately. For the moment you need to disable the wd_multi_grubplug (via renaming) whilst processing KL/FR distros and enabling it (naming it exactly as wd_multi_grubplug) when processing recent Pup distros.

Main thing about the plugin temporary approach is that it should allow other forum distros to experiment and provide the correct grub syntax that will work with their distros to include how they indicate alternative locations of their save folders, which is required. I won't myself by investigating that, but if other forum distros want to use this multi facility I will take notice of the customised wd_multi_grubplug stanzas they discover work and publish here.

[wiak]

Sorry guys, I don't myself have time to do more at the moment, but regarding the new wd_convert2multi for turning existing frugal installs into multi-instances (including for KL/FR distros the OPTIONAL re-use of instance0 upper_changes savefolder as a read_only numbered layer in all other instances:

Code: Select all

./wd_multi --help

Code: Select all

./wd_convert2multi --help

Main thing you need to remember to do is that once you cut or copy the existing frugal into say /KL_frugals, you need to give each directory you wish to then convert2multi an extension of .MKL (standing for MultiKL). Then the new wd_convert2multi shell script (which looks for that special .MKL extension) works pretty much the same as wd_multi script (where extension was .iso for that one). Note using Cut rather than Copy on same partition is very quick since the actual files don't need moved at all... However, using Copy gives you a backup copy of your original frugal install if you want that. Thereafter, my usage is not to use instance0 at all - just use grub to boot any of the other instances you want - that way, as I said, instance0 remains in its original re-usable state.

So can:

1. Create empty directory at root of partition. Say: /KL_frugals (or whatever you want to call it)

1. Cut or Copy existing frugal install into there along with the wd_convert2multi executable script. When wanting to convert such frugal installs, you indicate it to the script by adding extension .MKL to the name of the copied in frugal install directory.
2. Finally, run command:[/b]

Code: Select all

w_convert2multi <n>

where n is number of instances you want

so, for example, ten instances (instance0 to instance9) would need build command:

Code: Select all

w_convert2multi 10

Remember NOT to include/enable wd_multi_grubplug when building such KL/FR instances. It is NOT NEEDED at all then (and must either be disabled or not included at all), but is needed if trying same with Pup distros. Later versions of the multi build scripts will include detecting distro type and adjusting grub stanzas automatically without any plug file needed.

Downloads for wd_multi and wd_convert2multi are, as I've said:

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_multi && chmod +x wd_multi

and

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_convert2multi && chmod +x wd_convert2multi

[Clarity]

psave=Persistence:/..../..../..../Sessions/

which puppies can use this parameter?
which ones cant?

"psave=..." has been a parm in Puppyland going back to the days of Woof Pups, IIRC.

SAVESPEC for WoofCE PUPs has been around since 2019, again IIRC.

Hope this helps

[rockedge]

@wiak How does the wd_multi_grubplug (wd_multi_grubplugPUPPYONLY.tar} get integrated? I can't seem to be able to get the KL_multi to create a Puppy Linux specific grubconfig.txt using it

I've been manually creating the boot stanza's but want to be one of the cool guys and have it done automatically.

[wiak]

For the moment you need to disable the wd_multi_grubplug (via renaming) whilst processing KL/FR distros and enabling it (naming it exactly as wd_multi_grubplug) when processing recent Pup distros.

Haven't had time to use wd_multi_grubplug (for Puppy distro multi instance builds) for a while, but should just work per above comment from first post of this thread. NOTE: how it has to be EXACTLY renamed as wd_multi_grubplug - any other name means it isn't being used during the wd_multi creation. You just need it to exist in the same directory as wd_multi script all going well and it should cause correct Puppy grub configs to be made during instances build...

Per first post instructions:

2. Download the wd_multi script into that empty directory using command:

CODE: SELECT ALL

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_multi && chmod +x wd_multi

AND also download the dummy tar wd_multi_grubplug from first post of this thread if for making multi instances of Puppy Linux itself.

Let me know if not working and I'll try to find some time to have a look at it again. Or just look inside the wd_multi script and you should see where it gets called up via an if statement (which checks if such a file exists, and if so uses it per the code in the plugin).

Assuming wd_multi_grubplug exists in same directory as wd_multi it comes into play during a multi instance build around line 57 of wd_multi script because of this simple code:

Code: Select all

if [ -e ${HERE}/"wd_multi_grubplug" ];then
  . ${HERE}/"wd_multi_grubplug"  # use grub PLUGIN for non-FR distros

Later on I plan to incorporate into the standalone wd_grubconfig script so the grub stanzas can be determined independently at any time, but yeah... will have to wait till around March.

Alternatively if you are converting an existing Puppy into a wd_multi the script you need (which similarly works with wd_multi_grubplug for Puppy builds is downloaded with:

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_convert2multi && chmod +x wd_convert2multi

and quick notes for using that convert existing frugal to multi mechanism are provided here (as first post of this thread says): viewtopic.php?p=99401#p99401

[wiak]

In my view, the most, or at least certainly one of the most convenient, important current developments in 'What's up in this Forum World' remains multi-instance installs (firstribit and also KL_full2frugal pretty useful too actually). But I don't write about all that in What's up in Puppy World, because KL isn't a Puppy so don't want to confuse any users about that matter.

Just posting right now because last night had a great example of how useful multi-instances are, and this didn't come from me personally as a need. Rather, I was sitting doing something of my own on my laptop when my 12 year old son suddenly told me he had just upgraded his whole KLV-Sway distro that he has faithfully stuck to as his favourite since I first published the KLV-Sway Wayland/Pipewire- basee iso now long ago. Personally I haven't been using it so long ago have forgotten the Sway-related keyboard commands.

Anyway, the upgrade worked fine apparently, but pipewire sound wasn't working. My son said that he expected he needed to reboot and that would re-activate pipewire-related runit services and all would be fine, BUT... his system being so old had a HUGE upper_changes folder and he didn't want to risk that ending up being broken so didn't want to risk save2flash followed by reboot...

His idea was that he would like to abort the upgrade (by not doing a save2flash), but instead to make a number of mult-instances using wd_convert2multi.sh script of his current huge frugal install PRIOR to a full upgrade of instance01 only such that after save2flash, all going well, he would find that audio system was not broken... and if broken he could revert to using, say instance02. So that is what we did (simply cut and pasted the original very big frugal install into an empty directory, renamed the pasted version with .MKL extension and simply ran command:

Code: Select all

./wd_convert2multi.sh 10

to make ten instances of the original, followed by the upgrade and reboot of instance01, which now automatically contained the original huge upper_changes but as a high-2-digit-numbered uncompressed directory (well actually just the name of the symlink was automatically something like 60instance00_upper_changes; I'm not on that system so can't check exactly right now).

The wd_convert2multi process produced an accurate grubconfig.txt file with the immediately usable new grub.cfg stanza and all worked well. Upgrading did not in fact break sound system at all - the reboot was indeed all that was needed, but he was able to do that without worrry since had nine other original-state instances to play with if necessary... (just leaves instance00 alone as the copy of the original frugal install). Sorry if my explanation sound confusing - simple in practice and I'm sure @geo_c understands what I'm saying since he is very familiar with using multi-instances, which I truly recommend we all do!!!

Just a reminder of the simplicity of making multi-instance of already existing KL frugal install:

1. make empty directory for containing the multi-instance structure (we used: mkdir -p KL_multi).
2. Cut the original frugal (we had /KLV-Sway) install directory and paste it into that empty subdirectory /KL_multi (I used thunar as root user for that) - but rename the resultant /KLV_multi/KLV-Sway into /KLV_multi/KLV-Sway.MKL (refer to ./wd_convert2multi --help to understand the .MKL); Since used same partition this doesn't involve any actual dir moving so almost an instant operation.
3. Copy executable script wd_convert2multi.sh from firstrib gitlab into KL_multi directory (now alongside that pasted in KLV-Sway.MKL).

or better, fetch latest version of wd_convert2multi.sh script (which I haven't need to change for ages) from firstrib gitlab site using command:

Code: Select all

wget -c https://gitlab.com/firstrib/firstrib/-/raw/master/latest/build_system/wd_convert2multi && chmod +x wd_convert2multi

making sure to put it, for above example, into /KL_multi directory

4. From terminal then opened at KL_multi dir simply enter command:

Code: Select all

./wd_convert2multi.sh 10

where 10 was the number of instances we made (can make any number you like in practice. For example change the 10 to whatever you want even 100 or thousands if you crazy enough - still won't take up much extra disk storage space AT ALL....!).
5. Then go into resultant KL_multi/KLV-Sway/instance01 (or whichever ones you want to boot...), find the instance related file grubconfig.txt (which the script automatically makes for each instance) and fix grub.cfg followed by reboot to choose the appropriate instance you want booted...

If you then mess up by doing new stuff in that instance...; rather than try to fix it back to the original, you can simply boot a different instance next time. Powerful Worry-Free Simplicity in Action.

[wiak]

...
I was sitting doing something of my own on my laptop when my 12 year old son suddenly told me he had just upgraded his whole KLV-Sway distro that he has faithfully stuck to as his favourite since I first published the KLV-Sway Wayland/Pipewire- basee iso now long ago.
...
His idea was that he would like to abort the upgrade (by not doing a save2flash), but instead to make a number of mult-instances using wd_convert2multi.sh script of his current huge frugal install PRIOR to a full upgrade of instance01 only such that after save2flash, all going well, he would find that audio system was not broken... and if broken he could revert to using, say instance02. So that is what we did ...

Oh my goodness. So tonight I was thinking to transfer his KLV_Sway install onto a different lighter laptop since he needs to take one to school when it restarts in a weeks time.

So I asked him what size his installation now was (original frugal install plus whatever junk he has, over the years, saved into upper_changes). "No idea", he said. A few minutes later, he said, "which one" meaning which of the multi-instances (he is currently using instance01 of the ten possible). I said to check instance00 since that was the original the multi-instances were all based on.

Well, shock, horror... 87 GB!!! "What on earth do you have on there I asked?", not caring what the answer was - he has gradually installed every framework for everything known to humanity over time - c'est la vie for the young brain. Only talking about his current KLV_Sway install, BTW - he does have some other KL distros on there too. Needless to say, I am now seriously reconsidering the idea of transferring his install over to another laptop, or maybe I will. Would be nice if I could convince him to erase most of the junk stuff he surely rarely looks at any more.

Admittedly, my own laptop hard drive partitions are all now pretty much chock full, though I do hoover them down a bit now and then to give me some space for some new frugal install related testing. In my case, firstribit required a lot of testing so many a main stream iso download and conversion involved, but no idea what my son fills his machine up with and unlikely just cache.

[geo_c]

Well, shock, horror... 87 GB!!! "What on earth do you have on there I asked?", not caring what the answer was - he has gradually installed every framework for everything known to humanity over time - c'est la vie for the young brain.

You should squash that upper_changes and archive it at least. One large massive OS would be nice to have on hand now and then.