why chroot?

[williwaw]

Of course, hacking via mount_chroot.sh is often quick and easy....

And when finished doing whatever in the chroot, you enter exit command till out of it and must then run:

./umount_chroot.sh firstrib_rootfs

why is it best to chroot into a firstrib_rootfs when building on a host instead of booting with 07firstrib_rootfs?

is booting with 07firstrib_rootfs the same as "pseudo full install mode"?

[rockedge]

To use the system in a pseudo full install mode rename the 07firstrib_rootfs to upper_changes and replace the 07firstrib_rootfs with a dummy rootfs directory like 07Dummy_rootfs. This places the entire file system in the upper_changes so all the changes are written directly to the uncompressed upper_changes. The upper_changes can be squashed at any time to a 07KLV_rootfs.sfs again and the dummy replaced and a new upper-changes will be created automatically.

Using the chroot mount scripts will allow changes, installations and configurations to happen to the firstrib_rootfs while it's running and the resulting file system can be squashed. It is similar to running in the pseudo full install mode but is usually all command line on a console.

[williwaw]

Using the chroot mount scripts will allow changes, installations and configurations to happen to the firstrib_rootfs while it's running and the resulting file system can be squashed. It is similar to running in the pseudo full install mode but is usually all command line on a console.

"can be squashed".... after the chroot is unmounted?

what I would like to do is to make the assembly of the build more modular.

1. mount or chroot a minimal install
2. add xorg, or a large browser, or my personalized collection of apps to a build then..

then save each of the large downloads as sfs.'s this way I would not have to suffer thru long builds that fail often on my connection. reassemble on builds using small plug files.

does psuedo seem like the best way? seems like making sfs out of upperchanges would work. dunno if chroot would be better.

[rockedge]

@williwaw with the pseudo full install method the system can be run and configured with whatever the desktop manager is, and those immediately get included in the squashed SFS so customization will remain persistent. This is possible also in the chroot method, but is tricker because it the KLV running on top of the OS underneath, using the host operating system's kernel and resources and X server. KLV-Bspwm is in pseudo full install mode running fully and all changes are written to the complete file system.

I like your idea of modulation and believe the method will work, including putting Xorg in an SFS and a desktop in another. You can use 99 layers.

[wiak]

Okay let's consider advantages and disadvantages of all three methods (but remember there is a fourth, often more useful method, which is to use portable apps of some sort or another, such as these from mike walsh - not all will work, but some will, and AppImages though big are good alternative):

1. Using mount_chroot followed later by umount_chroot to clean up. This is all done on host build system, not on the running new FirstRib distro.
If 07firstrib_rootfs is stored as an sfs file, first you must unsquash it so you can then chroot into it via script mount_chroot.sh. If it is just the build directory you can skip that step:
unsquashfs 07firstrib_rootfs.sfs # which will uncompress 07firstrib_rootfs.sfs to directory named squashfs-root. You can leave it with that name at this point if you wish or rename it 07squashfs-root (or 07anyname/ such as 07firstrib_rootfs). Let's assume you renamed it to 07firstrib_rootfs, so now:
Use command "./mount_chroot.sh 07firstrib_rootfs" to run commands inside thereafter that permanently modify the 07firstrib_rootfs via a chroot from running host system, followed (after entering 'exit' command to leave the chroot), by running "./umount_chroot.sh 07firstrib_rootfs", to clean up after using a chroot (which mounted various filesystems temporarily as what are called 'bind mounts'

2. Using what we refer to as "pseudo full-install", which is simply arranging things as follows:
mkdir -p 07anyname # as long as it starts with the two-digit number
unsquashfs 07firstrib_rootfs.sfs # which produces an unsquashed verion of the 07firstrib_rootfs.sfs in directory called "squashfs-root"
(at that stage you should disable the original sfs via: mv 07firstrib_rootfs.sfs Dfirstrib_rootfs.sfs (or remove it altogether since you have it in unsquashed form now anyway).
mv squashfs-root upper_changes # which on reboot becomes the read/write top layer for all new changes
mv 07firstrib_rootfs.sfs D07firstrib_rootfs.sfs # to disable it, or alternatively can remove it altogether since not really needed anymore

3. Using rollback upper_changes folders (renamed to start with 2-digit numbers to signify what layer they will be used at) to modularise additions.
For example, after using the system and installing some stuff on it which is saved to upper_changes persistence folder, later:

mv upper_changes 50upper_changes # note: doesn't need to be 50, just an example, where if I keep repeating this method the next rollback layer will be named 51upper_changes and the next after that as 52upper_changes and so on. You can optionally squash these up via mksquashfs if you wish.

Method 1 above (mount_chroot method):
a. You do not need to boot the new FirstRib system to use this method. It is done from the host build system by well-known chroot methodology, which provides internet via host internet, but does not see the host filesystem at all since uses 07firstrib_rootfs ( as / (root) directory of its hierarchy, so the package manager inside 07firstrib_rootfs.sfs is available to you during that mount_chroot.sh period (but nothing outside of 07firstrib_rootfs.sfs since in chroot jail during that period).
b. Anything you do in that chroot will be permanently written to the unsquashed 07firstrib_rootfs directory
c. Once finished making your changes, you have the choice of either leaving 07firstrib_rootfs as an uncompressed directory, and simply booting into it in that form, or to save space you can compress it using the likes of:
mksquashfs 07firstrib_rootfs/ 07firstrib_rootfs.sfs -comp zstd -Xcompression-level 19 -b 512K # assuming you want zstd compression, which doesn't highly compress but gives better performance
Advantages: No need to boot into new distro since can do at any time from another host system (usually the host you originally built from)
Disadvantages: Doesn't help modularise system - just a convenient way to add to it.

Method 2 above ("pseudo full-install" method):
Advantages: Just using the system automatically updates the underlying root filesystem.
Disadvantages: what was the original 07firstrib_rootfs needs to be kept as a much larger uncompressed directory all the time, which is renamed as upper_changes such that all changes get auto-saved into it; though save-on-demand RAM2 save2flash mode can still be used with it, so that's not a problem. Doesn't help modularise either since also just a convenient alternative way of adding/updating the main root filesystem constantly during use.

Method3 above (rollback upper_changes addons):
Rollbacks work fine and help modularise system, but there is a potential problem that limits their use for modularisation. The problem is package management database needs to be kept accurate and clean. Each rollback (numbered upper_changes folder, which can be squashed via mksquashfs) will contain updates to the package manager database and therefore you need to keep the whole sequence or you will break package manager database. So it is not like a portable app solution at all. Rollbacks are useful if you want to go back to previous state one level at a time: first you would delete the most recent unnumbered (main read/write layer) upper_changes folder, but you can go back further by also deleting (or renaming as unnumbered) the last highest numbered rollback NNupper_changes and so on and on... but can't suddenly delete an NNupper_changes somewhere in the middle since that would break package manager database consistency. We don't yet have utilities for merging rollback addons together since not been used much yet (only geo_c really used them) and no-one has yet found time to right useful merge scripts...

So if modularisation is your goal, then using some form of portable app is the way to do that. But... a new upper_changes read/write layer CAN be turned into a kind of portable app by removing its package manager database files so the package manager doesn't know it exists... For example, you might use official package manager to install say FIrefox and then simply remove the package manager database in upper_changes relating to that and theoretically you may well be later able to use the resulting directory as an addon (uncompressed dir or sfs) at any layer you like (since won't effect actual package manager database since none in it...) per example I gave here:

"Making firefox sfs for KL (invisible to apt/dpkg)"
viewtopic.php?p=84173#p84173

I give no guarantee for that firefox idea, but definitely worth investigating as 'cheap/simple' portable app for KL making method. Then again, plenty of good mike walsh or AppImage alternatives (though all of these likely to be much bigger addons than the firefox sfs addon I describe in above link).

In practice, method 1 and method 2 are very simple. If new to firstrib building I'd stick to that. Keep it simple. No need to learn every FR initrd possibility all at once - later you'll find out the tricks anyway, but nothing complicated needed to begin with or usually in fact. Rollbacks are simple enough, but portable app making is a trick (and a bit tricky...) and not something to bother about really.

[geo_c]

Okay let's consider advantages and disadvantages of all three methods (but remember there is a fourth, often more useful method, which is to use portable apps of some sort or another, such as these from mike walsh - not all will work, but some will, and AppImages though big are good alternative):

In practice, method 1 and method 2 are very simple. If new to firstrib building I'd stick to that. Keep it simple. No need to learn every FR initrd possibility all at once - later you'll find out the tricks anyway, but nothing complicated needed to begin with or usually in fact. Rollbacks are simple enough, but portable app making is a trick (and a bit tricky...) and not something to bother about really.

In the past two weeks I have built three KLV systems from the ground up using a combination of psuedo-full install, and roll-back changes which I then squash into an additional layer.

I have always made use of portable apps and appimages, and I see absolutely no disadvantages with them. Just about every @mikewalsh portable works in KLV, and appimages also fully work. As appimages I daily use LibreWolf and MuseScore4, and for portables I use daily Thunderbird, LibreOffice, Ungoogled Chromium, Palemoon, Gimp, and Zoom. They are fast and clean, easy to update, and can be booted fresh without a saved profile by simply moving or renaming the profile folders. So in essence, if you wanted to boot up a portable with a saved configuration, but none of the collected use history, you could set up a configured profile and save it as a master, then create a simple script to copy/load it to it's expected location and launch the portable/appimage. Configured but with no risk of picking up malware/adware scripts and executables from repeated use.

The method to build the the KLV systems was simple:
1) Use @wiak's psuedo-full install method described above. Install and configure a base system and resquash the rootfs.
2) Boot the new rootfs and make changes and install applications, Shutdown and squash the upper_changes layer, removing the uncompressed upper_changes.

My KLV systems look like this now (viewed from KLV-spectr) Layers 20-22 are added above 07rootfs:
click to enlarge

[williwaw]

@rockedge

When working in the chroot envrionment, does one need to unchroot before making a sfs of the rootfs?

Rollbacks are useful if you want to go back to previous state one level at a time: first you would delete the most recent unnumbered (main read/write layer) upper_changes folder, but you can go back further by also deleting (or renaming as unnumbered) the last highest numbered rollback NNupper_changes and so on and on... but can't suddenly delete an NNupper_changes somewhere in the middle since that would break package manager database consistency. We don't yet have utilities for merging rollback addons together since not been used much yet (only geo_c really used them) and no-one has yet found time to right useful merge scripts...

Thanks for the comprehensive explanation of the differences. I like the idea of removing the package managers data to make a "portable".
Perhaps if very few numbered_upper_changes.sfs were loaded when the portable is created, then there would be less issues? of course at the expense of duplicating dependencies in other numbered-upper_changes sfs's.

Is there a packinglist for 12KL_gtkdialogGTK3filemnt64.sfs somewhere?

@geo_c
why would you want to merge rollbacks?

[geo_c]

@geo_c
why would you want to merge rollbacks?

Well I would only be calling them rollbacks as kind of shorthand. In reality they are stages of developing a personal system.

In the old days, when @rockedge and @wiak were rolling out KLV-airedale-beta2 or something like that, I started using it and wanted to experiment. So I shutdown, renamed upper_changes to an XXupper_changes-[some-description] and trecked onward with new upper_changes.

After a few months I had 20 levels of numbered upper_changes, and wouldn't it be nice to simply merge all of those instead of starting over fresh, or copying them all by hand from machine to machine? (if only I'd known to squash them as files, then copying would have been much simpler)

Well, as it went, @rockedge and @wiak kept improving and rolling out new rootfs's, so I would carefully run a void system-update in my current upper_changes so it wouldn't overwrite the new rootfs's with older system files, then shutdown and copy the new system files underneath my XXupper_changes.
That worked well for a solid 6 months all the way to KLV-airedale-RT, then the long chain of upper_changes revealed something broken probably in the package management.

So I did start over, and this time I went above and beyond the old system all in one shot and adding two more levels of upper_changes which are squashed and quite small in size, as you can see from my above screenshot, files of 2MB, 780K, etc.

And at this point in time, I'm not looking to merge upper_changes.

Instead I've managed to use the psuedo-full install technique to get everything in there clean out-of-the-box, and I add a couple squashed upper_changes for things unique to particular machines, like wifi-connections, monitor resolutions, minor tweaks, etc.

Well the one big sfs layer is the 20-lip...sfs, because that has hplip printer drivers in it.

[rockedge]

When working in the chroot envrionment, does one need to unchroot before making a sfs of the rootfs?

Yes, very important or the squash will fail.

[williwaw]

the chroot envrionment can have issues! (I guess if it is not properly unmounted)
I understand the benefits of being able to run the package manager and commands specific existing in firstrib_rootfs.

Is it always neccesary to use chroot when working with your 07firstrib_rootfs
for instance, making edits to configurations or creating (or deleting) files?

What other operations do not require chrooting?

[wiak]

Is it always neccesary to use chroot when working with your 07firstrib_rootfs
for instance, making edits to configurations or creating (or deleting) files?

No, not at all. You can do general config work inside the uncompressed 07firstrib_rootfs from the host build system without any chroot. It's just a directory with files in it after all. You can therefore manually add or change files in it without doing chroot and when you reboot you changes will of course be there.

Using a chroot is mainly a way of running the actual system from a host build system so its unique package management works. For example if you don't chroot from the host build system (or alternatively actually boot the FirstRib/KL system) then anything you add is unknown to xbps package manager database (sometimes that fact is handy and for small utility apps, such as devised on this forum, xbps doesn't need to know anyway, though sometimes it is convenient to make an xbps package of a forum utility since then can be conveniently removed later via package management or installed again).

In practice, when using my own systems, I often leave 07firstrib_rootfs uncompressed or use it as upper_changes (pseudo install methodology) and mess around with its config files and more directly whilst in host build system. And nowadays, more and more I make tiny little numbered sfs addon directories (not bothering to squash them) that I try out on my system as layers only, and either leave them in that form or squash them up or manually merge them into 07firstrib_rootfs itself; often such utility apps are mainly held under /usr/local hierarchy with main scripts being accessible via /usr/local/bin, which is generally top priority searched by PATH.

For example: I recently made an addon to include rox filer on my own KL systems that didn't otherwise include it. All I did was made a normal folder called 11r_rox_something/, and then put a directory structure inside it starting at usr via command (with terminal opened at buildfrom directory):

Code: Select all

mkdir -p 11r_rox_something/usr/local/bin 11r_rox_something/usr/local/share/applications 11r_rox_something/usr/share/pixmaps 

# and whatever other dirs I wanted/needed for rox component parts; the rox components being extracted from some package or other - one I used was an rpm rockedge provided me from old Fedora repo) and copied rox binary and configs and .desktop into that wee structure. On boot it gets merged to layer 11 of the overlay (so appears as if installed).

This method/mechanism is very very flexible. We have the up to 99 layers, so might as well enjoy the advantages/flexibility they provide!

As for the likes of 10 or 12KL...gtkdialog_filemnt sfs, there is no list of package contents, but it is simple to check what is in it as a list since near enough all of it is in /usr/local/bin, so to see what app/utils it contains just mount the sfs (or use utility that does that. In Linux Mint I normally install filemnt to allow me to click open sfs and iso files, but don't really need filemnt: easy enough (and I do this often) to just, for example,

Code: Select all

mkdir mymnt

followed by command:

mount 12KLblahblahblah.sfs mymnt

and can then examine what is in the sfs, and copy from it (but not write to it since read-only mount) without needing alternative method of unsquashing it using unsquashfs command (you umount it at the end with command umount mymnt).

Advantage of instead unsquashing the sfs is that the result is both readable and writable and after modifying its contents you can use mksquashfs to squash it up again of course.

[williwaw]

......For example if you don't chroot from the host build system (or alternatively actually boot the FirstRib/KL system) then anything you add is unknown to xbps package manager database (sometimes that fact is handy....

good to know, and so noted.

On boot it gets merged to layer 11 of the overlay (so appears as if installed).

Is all the firstrib magic done when booting with your initrd.gz and vmlimz?

I am intrigued with this tutorial

https://docs.voidlinux.org/installation ... tfs-method

can I build a rootfs with other tools and name it 07_xxx and use it with your initrd.gz and vmlimz to "ribit"? Is there anything in the rootfs that needs to modified?

just a note to anyone following, mount_chroot.sh and/or umount_chroot.sh does not play well with EasyOS but works well in puppy.

[wiak]

......For example if you don't chroot from the host build system (or alternatively actually boot the FirstRib/KL system) then anything you add is unknown to xbps package manager database (sometimes that fact is handy....

good to know, and so noted.

On boot it gets merged to layer 11 of the overlay (so appears as if installed).

Is all the firstrib magic done when booting with your initrd.gz and vmlimz?

I am intrigued with this tutorial

https://docs.voidlinux.org/installation ... tfs-method

can I build a rootfs with other tools and name it 07_xxx and use it with your initrd.gz and vmlimz to "ribit"? Is there anything in the rootfs that needs to modified?

just a note to anyone following, mount_chroot.sh and/or umount_chroot.sh does not play well with EasyOS but works well in puppy.

Is all the firstrib magic done when booting with your initrd.gz and vmlimz? YES (though KL user-made system 'helper' utilities by the usual other regular KL build contributors utilise these FR initrd provided functionality/facilities.

So overall, YES, most often doesn't matter where you get the root filesystem from or how you make it; simply use the FR initrd.gz to boot it and all the user-made system 'helper' utilities will generally work with it (including the special KLfull2_frugal script that can take existing full installed distro such as Linux Mint, Zorin, or whatever, and simply use its root filesystem (without change) as a frugal install (and the full install still works as before - just a different boot menu stanza).

So, YES, you can build a root fs and name it, for example 07_xxx (or whatever layer you want it at above 01, which is generally reserved if using huge kernel for 01firmware and 00 reserved for 00modules). Of course, your root fs must be one that works and has a suitable init system such as runit, systemd, or sysVinit since FR initrd ends up by switching to the /sbin/init inside your root fs so that must of course exist and make logical sense as an init system... Most mainstream distro default available root filesystems work out of the box (since come preconfigured with runit, or systemd or some other init system and accessed usually via /sbin/init - very very occasionally they don't quite follow that convention and we either have to hack their root fs a little bit or slightly change operation of FR initrd (which can often conveniently and easily be done via provided w_init editable text code that FR initrd/init calls up as part of its normal operation).

A Puppy Linux root filesystem CAN also be used, but Puppy system files are closely tied into its own initrd, expecting 'PUPmodes' to be set and various DISTROSPEC files, so a little bit of work is involved avoiding its distro-system-script-specific parts, but not too difficult really. Mainstream distros tend to work, as I say, without any mods being required - all very consistent and universally usable immediately by the generic-capable FR initrd overlayfs code system. That general purpose nature of FR initrd (and also that of most mainstream repo root fs creations) is what makes FR/KL systems so flexible and powerful and relatively easy to construct. The centre to that flexibility is the FR initrd capabilities along with the simple build_firstrib_rootfs script (though as you ask, most any pre-made rootfs can be used as an alternative straight away with FR initrd along with suitable kernel/modules/firmware; huge-kernel/firmware/modules like those from Puppy Linux work well, though can also build using upstream official kernels/modules and firmware so up to the builder; lots of scope.

Back in the early FirstRIb days (around May 2019), there only was build_firstrib_rootfs script, so the system at that stage was only intended to be used via a chroot on an already running host, but was pretty useful even for that (and started only with Void Linux repo and xbps package manager using variant). Development was very intense and very quick at that stage and @rockedge was an early adopter (using Xnest in chroot configurations) and a few weeks later once the first FR initrd came out (that always used overlayfs and could use compressed or sfs addons up to 99 layers from the start), aside from my own exemplars, rockedge was first to try f_ plugin creations/builds and rest is history. fredx181 tried it out soon after and provided some useful ideas/helper-scripts, and rufwoof also helped test and offer some great ideas back then (on the old forum) but only for a little while.

Frankly, I never imagined FR type builds with KL extras would reach the ease of use stage it is at now such that we produce so many very different types of distro with it. Never imagined that scope for a moment - lucky design choices maybe; fortunately the original FR initrd logic has proved to be very reliable and flexible and expandable - it wasn't hacked over time but developed in a purposeful progression, which is why it hasn't been difficult to add to its facilities. Yet I never originally even intended ever making a FR initrd - just wanted the root fs builder for chroot use on different running host... glad I made it now though. (Been tons of 'flack' surrounding its development discussions using this forum though..., which I am not glad about).

In terms of community use, well, it is open source; I design and maintain its core FR initrd and core build_firstrib_rootfs for my family use (including our business use) but I made the system full of plugin capability, which is why it can provide a powerful community resource since, whilst I help with how to do it and so on, the end f_ build plugin distro creations belong entirely to whoever makes them. So FR is a build system that includes a flexible general purpose FR initrd for frugal install capabilities. Also the file w_init contains most of the FR overlayfs initrd functionality in an easy edited form so anyone can produce their own 'special' w_init for special purposes of its own - so it is really a user/community-plugin for the FR initrd itself. There is a default copy of w_init embedded inside FR initrd so an external one for quick editing is entirely optional; often I use an external w_init that is identical to the one inside the compressed FR initrd.gz - the code in the external one is always used instead of the internal one if it is provided (i.e. the w_init external plugin has priority).

[williwaw]

mount_chroot.sh reports chroot directory is.... when the script is run, and
umount_chroot.sh is presumably supposed to return Cleaning up mounts to chroot directory...
when the unmount script completes succesfully?

If Cleaning up mounts to chroot directory... does not return, what can I assume? what should I do?

I am not sure if umount_chroot.sh will work correctly with all hosts, But then again I thought that umount_chroot.sh may not always be neccesary? After all, when the chroot is exited properly, isnt that's all thats should be neccesary?

is running umount_chroot.sh a non required "best pratice" script or a neccesary part of the total process?

[wiak]

is running umount_chroot.sh a non required "best pratice" script or a neccesary part of the total process?

It is absolutely necessay. Just like a normal mount needs a umount to end so does the mount_chroot.sh command. It was split into these two parts for some reason or other I can't remember. But, as I've said elsewhere, nowadays I generally use the one script, mount_chroot_umount.sh that auto cleans the amount on exit (which mount_chroot.sh doesn't). Seems to work fine.

If using ./mount_chroot.sh <name_of_rootfs>

then it is important to also use same <name_of_rootfs> argument when using umount_chroot.sh of course or the mounts won't be cleaned up...

Actually, I didn't put any final message at end of that old umount_chroot.sh. I should have I suppose. You can always add an echo "Done" line at end if you wish, but that could be lying... If you want to be sure the umounting has been done you'd need to add check code (such as using appropriate if test mountpoint kind of commands), which I never got round to doing since I like to check manually (using single simple mount command) anyway since even back then I didn't trust AI completely...

No such problem with combined mount_chroot_umount.sh though no AI type test in there either as far as I remember; on the whole I recognise that users who build distros are pretty savvy about Linux - don't need to be guru-sorts per se, but don't need absolute hand-holding, which means build system scripts can be kept simpler (with these check-all-is-well-yourself caveats). Of course main reason is that time is short and tons of todo jobs that are high priority - little helper scripts tend to be very basic since of low priority in the scheme of things. Different when you are publishing so-called 'polished' distros that you hope anyone's granny can comfortably use; I'm not saying no grannies or grand-dads aren't building distros - at my age I can't make that claim, but building distros is low-level work that expects some expertise - it isn't a shiny GUI frontend world that keeps the user safe. Yes, nice when safety check code included, but, yeah, time is too short for such much of the TIME...