Under pre-release testing: new WDLGO systems

[wiak]

Announcing WeeDogL_GO systems (think lego).

See Blog for details:
viewtopic.php?p=23804#p23804

[wiak]

Not quite ready to release yet but going really well.

Difference between current and earlier tests is that I am no longer simply hacking root_fs and initrd together. Rather, I'm modifying build_firstrib_rootfs and build_weedog_initrd scripts so that most everything is done automatically. The only bit I leave for manual tinkering, during a script pause I have inserted for the purpose, is to allow builder to substitute different modules (slimmed down list, or for particular kernel needs) into the weedog_initrd and the firstrib_rootfs during the build_weedog_initrd phase. That arrangement makes for simple/flexible developer's creation of WDL_GO no package management system - takes just a couple of minutes to create the frugal bootable system.

Smallest possible is to use a huge kernel (i.e. one that has all boot modules included). Well guess what?... Puppy pretty much provides that in its huge kernels. Only thing unfortunatly missing is that by default Puppy does not include overlayfs driver in the kernel (only as a module). Hence I've testing a WDLGO vnopm (no package manager) tiny system using Puppy kernel. For that I'm currently using BionicPup32 kernel, and inside the initrd, all I need to add is the single overlayfs module into initrd/usr/lib/modules (easily done in seconds manually by cut and paste from BionicPup32 zdrvXXX.sfs).

The result is bootable on pretty much any computer, and its size is only 7MiB total... That's using standard gz compression, so could be even smaller with more compression, but fine as is. Next step would be to provide 02addon.sfs and so on, but even without that this basic WeeDogL_GO frugal install cmdline core system (or 7MiB bootable iso) connects out-of-the-box to internet via ethernet.

Anyway, I'm bumping up the two build scripts to version 3.0.0 to mark the introduction of these new WDLGO build options (vnopm and vmini). Will still work fine (and as before) to build rockedge's WeeDog_Void32 system and my own WDL_Arch64 openbox/tint2 distro. But I have a bit of further testing to do before release to make sure I've not introduced any build bugs along the way.

So I think first iso I release will be of that 7MiB no package manager busybox bare/simplistic commandline distro that uses BionicPup32 (vnopm because the filesystem layout is organised the Void Linux way where all binaries end up in /usr/bin and so on - which is pretty much same in Arch Linux BTW). Then I'll also publish a HowTo showing how easy it is to make this tiny distro using the usual two WeeDog build scripts. The relatively minor alterations I've made inside the build scripts also makes it very easy to produce the WDL_FossaPup64 (or any such wee-dogged Pup) in minutes (if you have the original FossaPup64 iso at hand) or, similarly, a WDL_Slackware, for example (using any Slackware rootfs renamed to firstrib_rootfs), or whatever... There is really no limit to this system, and f_build-plugins continue to work with ANY new root filesystem that is being constructed using build_firstrib_rootfs script (including the above vnopm that uses BionicPup32 kernel to save some space).

P.S. the booted WeeDogL_GO (WDLGO) "no package manager" BionicPup32-kernel simplest cmdline system reports RAM usage = 14MB
Booted to commandline in 5 seconds (this is a breath of fresh air). Being vnopm, no xbps package manager is installed on this compressed-form 7MiB system; hence it is so tiny, but still expandable via sfs addons.

Actually, this is a bit like the first WeeDog bootable system I made almost two years ago, since, rockedge may recall, I used a Puppy kernel plus overlayfs module for that early creation for similar huge kernel containing most drivers reason (along with additional drivers available in addon 00_modified_zdrv_firmware.sfs; that method still works. Though I don't need that firmware/modules sfs for WDLGO system here the idea was established back then).

wiak

[rockedge]

I'll be building some of these sooner than later....as the work on WeeDog32-Void is nearing a point of this OS being ready to fly from the nest.

I have also worked on the PLUG file that should be about ready to build the base system from the scripts. Crazy is the WDL technology is rapidly expanding and suddenly there are all these doorways open. So focus is the key.

[wiak]

Crazy is the WDL technology is rapidly expanding and suddenly there are all these doorways open. So focus is the key.

Yes, focus is the key component to get any particular release issued so don't be distracted!

Actually, however, 'WDL technology' core ingredients already included all these doorways - we just haven't had time to introduce or use them. One of the main reasons WDL system exists is to remove mysteries of how distros work, to allow anyone who wants to be a developer to get their hands dirty. Anyone who has actually used WDL from a build perspective/philisophy should have some understanding that First Rib WeeDog implements (and recommends) the following formula for constructing its frugal installable distros (via two parts: firstrib_rootfs and weedog initrd):

1. The First Rib root filesystem consists only of:

simple shell + options

which in more detail is:

(simple shell) + <any (number of) package manager(s)> + <any_wanted_convenience_default_inclusions> + <further_extras_via_plugin(s)>

where:

the number of package managers can be 0 or more.
the number of any_wanted_convenience_default_inclusions can be 0 or more
the plugin(s) can be 0 or more

For example, here are some distro variants FR WD can be used to build:

i. busybox alone system (a no package manager system) - this being vnopm system I discuss in posts above.
ii. busybox + xbps (Void package manager in static build form) - this being Void mini system (vmini).
iii. busybox + xbps + (wiakwifi + void_base + eudev + wpa_supplicant etc) + <optional f_plugin> (This being regular WeeDog_Void builds).

build_firstrib_rootfs build script could always do this (just needed to comment out, for example void_base, eudev, and wpa_supplicant etc to achieve vmini system).

The only simple changes I've been making to build_firstrib_rootfs build script for next release simple do the 'commenting out' for you (the build user), nothing more than that.

2. The WeeDog distro-agnostic initrd (which uses overlayfs for all its frugal install, save changes/rollback etc tricks and works with any or all of above FirstRib root filesystems, and, as has been shown, with most ANY other distro root_filesystem at all):

I haven't really changed build_weedog_initrd at all except to allow it to pause during build as a convenience to allow the system builder to change whatever kernel/module selections they wish to use.

Rather than using chunks of someone/big-repo underlying design, WeeDog is build from scratch using simple scripts and first principles (not so much the debian/ubuntu variants since I had no real choice but to use debootstrap, which downloads ready-made by upstream first filesystem). The WeeDog initrd is not copied from anyone else, standalone distro agnostic, and all created by one relatively short shell script; so its operation is visible for anyone to understand.

So the apparent 'rapidly advancing...doorways open' is actually just more time being available to use FirstRib WeeDog build system, and more 'focus' to actually get round to completing more of the near infinite creations that could be constructed via this build system. I always in fact said that a WeeDogLinux system could be constructed as big or tiny as you wanted it (within the above limits). The initrd has from its early designed days supported any sized build (including optional huge initrd, which can embed any root_filesystem or extra sfs files inside the initrd itself). So hopefully working with WeeDog build scripts also increases overall forum knowledge about these 'tricks'. They are not revolutionary (even layered filesystems have been implemented in ancient UNIX era) but hopefully WeeDog helps show that they are not mysteriously black-magic guru-only tricks either.

Moreover, the design of FirstRib WeeDog means it is always quickly upgradable to latest builds. No worry that it will suddenly not be developed - its future is pretty much self-sustaining by design (and especially when using upstream rolling distro flavours such as Void or Arch Linux).

[wiak]

Under pre-release testing: new WDLGO systems

Growing in design so taking me longer to release than I intended...

Progress:

The build_firstrib_rootfs/weedog_initrd Void-Linux based vnopm and vmini versions scripts described in above link, have been 'completed' but will still undergo further required testing by myself prior to first release. However, as I worked on these first versions, an immediate interest I have in a WDL_GO Ubuntu variant has caused me to suspend testing the above simple Void-based systems, hence a bit of delay for now.

The WDL_GO Ubuntu variant:

Not being Void-based, this build mechanism is considerably different and not as yet integrated into the newest FirstRib WeeDog build scripts. Since quite a bit is involved that is unrelated to current FR WD build scripts, I will probably release any build scripts for this particular 'strain' as separate entities - I don't want to excessively complicate the current FR/WD build script system so a separate build system feels appropriate for the special Debian/Ubuntu-based WDLGO versions. In practice, I plan to release a working version prior to build script implementation since it always takes more work to polish an auto-build system than provide a working build.

Summary: Currently working on WDLGO Ubuntu... (first using Bionic32-base, but also done some work on WDLGO Focal64 version - may do a Debian variant if all goes well). Not sure how long first release will take (may be a bootable iso, or as separate numbered sfs files - being as it is an expandable system in component sfs form). I have it all designed on paper... but still to assemble the initial base sfs (the 01firstrib_rootfs.sfs). Rather than manually just hacking that together (which would be quick) to save overall time I'm partially working on the build script for that in parallel (to keep my mind on track of how to do it), so takes longer to do it that way...

Once that is released I'll get back to further testing the newer FR/WD build scripts that provide the Void-based additions vnopm and vmini.

Note that WDLGO Ubuntu variant is nothing to do with my much later planned (autumn!) WDL_Ubuntu...64 full desktop 'clone' of current WDL_Arch64. Rather it is a much finer-grained component-oriented distro, whose first release will likely be commandline only (but should be (relatively) easily expanded to X desktop via addon/re-named upper_changes.sfs (able to be rollbacked) components.

Next time I report on these new WeeDogLinux project additions/developments will coincide with an actual release of some shape or size... May take a week, may take considerably longer since it is summer where I live and I'm busy otherwise generally (indeed, WDL developments were not planned till autumn arrives, but rockedge working on his WDL_Void32 kept getting me back in front of my computer since so interesting...).

Since this thread is in WeeDog Announcements area its informative only, but next report-with-release will be available for general-user comments via WeeDogLinux forum.

wiak

[wiak]

No release has appeared of ubuntu WDLGO system I was working on, sorry. I hit a roadblock (after much work) and decided yesterday to abandon the attempt whilst failing with an ubuntu focal WDLGO build attempt (frustrating...). However... this morning I tried one last thing since I remembered I did have some early dpkg-related success with simplest bionic WDLGO version, and that seems to be partly working. No idea why the focal version isn't the same, but I will continue and release the bionic version since seems to be usable afterall (albeit with dpkg/apt-related authentication and locale-related issues) ... Indeed, a few seconds ago I successfully installed a new package on the system via "apt install", which is what I was aiming for. Someone with expertise in apt configuration will no doubt be able to fix the authentication-related issues - dpkg database seems to work fine. Also, apt --fix-broken install is no longer giving me any error messages, so basically a success...

Not sure what the issue with focal is though - can't be much hopefully.

Note that I'm not using debootstrap, which of course would sort out all these issues anyway (WeeDog can already build a ubuntu system via debootstrap but I wanted something absolutely minimum, albeit incomplete - just enough to on simplest version to use dpkg alone (use to resolve any dependencies on their own), and on next version to use apt (since needed for dependency resolving).

Whether it is worth making this mini-Ubuntu depends, for me, on final overall size compared to a minimum debootstrap creation. Time will tell. The mini/crippled new creation is certainly educational to play with...
As things stand, the WDLGO bionic64 dpkg/apt-capable firstrib_rootfs.sfs (normal gz compression) is currently 92MiB (EDIT: this is rubbish. I made mistake checking the size, so much better than that per next post below. thank goodness...), so it is not minute, and needs firmware/modules sfs added to that (say another 35MiB for medium-sized version of that) and provides only the simplest of commandline systems, so around 130MiB all-up for basic apt-capable ubuntu bionic (32 or 64bit) commandline system - maybe a bit less since I haven't zero-sizes man and doc pages, or cut locales, and so on. Overall, I'm not convinced the whole effort with resulting likely size is worth it thus far though... Gone are the days when a whole debian-based X desktop could be made under 100MB...

The simplest dpkg-only version might be more useful since much smaller and can be used for one-off small dedicated commandline systems (but with further effort since not dependency resolving) - it can extract deb packages though, so that might be useful in itself. Both might have some use for usage in virtual machines.

Anyway, whether useful of not, I will likely release whatever works or partly works since fun at least to experiment with.

EDIT: I'm glad I didn't abandon this WDLGO ubuntu bionic system - I now have the apt part of it worked out (at least working...). I'll look into any remaining WDLGO ubuntu focal issues thereafter - still some work to do since will be producing iso (and of the much smaller dpkg-only variant). I think this ultra-minimised version could be very useful to help trim down dedicated debian/ubuntu systems more generally, as well as for chroot or virtual machine installs. Might also be useful for helping to design a more accurate apt-on-puppy implementation since easier to see what is required altogether in creating such. Personally I am an advocate of using upstream repo's official package manager rather than needing to translate package formats since that is both time-consuming and subject to inaccuracy - I don't mean to imply that PPM or pkg for that matter isn't useful - actually I think they are very useful but for a different use-case: being for Puppy-independently-compiled repos (much like those used in original Pups - mixing and matching various upstream distro-repos I'm not so sure about overall: hardly seems necessary anyway.

[wiak]

So, thank goodness I did not abandon this attempt. Had already decided to do so owing to problems getting it to work...

However, now basically have first WDLGO Ubuntu system working including dpkg/apt without issues, and full multi-user pam passwd working, and with wpasupplicant, so connected via wiakwifi via my wlan with no problems. Best news is my first measurement of the firstrib_rootfs was totally wrong (I must have tripled installed stuff!) - the actual size of the firstrib_rootfs.sfs is around just 30MiB when compressed via normal gz, and only 21MiB when compressed heavily with mksquashfs xz.

I'm currently booting it using the huge kernel from BionicPup64 and the (large sized) firmware/modules from the zdrv of that. The resulting firmware/modules sfs used is thus pretty large at around 56 MiB when heavily xz compressed. In practice, I'd look to use quite a tiny firmware/modules sfs to fit my own machine. But anyway, it looks to me now that I could indeed make a (VERY) simple X-desktop WeeDogLGO_UbuntuBionic64 of around 100MiB (and probably less if I severely limit the firmware/modules provided).

I'm not planning an X desktop version, but anyone can make that since fully dpkg/apt compatible anyway. I meant to report no more on this other than a release, but too excited with result not to. Still a bit to do on packaging this prior to upload, and yes the build is scripted... so shows exactly how to trim down to what is pretty much minimum Ubuntu/dpkg-apt/multiuser/wpasupplicant-capable system possible. Of course, an ethernet-only version will be even smaller... since wpasupplicant has quite a number of dependencies (aside from the wifi firmware required for that). Actually, I haven't bothered trying this WDLGO via ethernet connection yet (too lazy to look out ethernet cable and hence me including wpasupplicant in this first build...).

free reported around 30Mb RAM used immediately after boot (a bit more because of caching by the time I took the screenshot photo attached below).

Now I can definitely say: "Coming soon" though. But I intend making 32bit version of this Bionic release first since nice for old machine maybe or in a chroot or virtual machine.

EDIT: WDLGO Ubuntu FocalFossa version now working too (though can't state firstrib_rootfs size as yet till I re-run fixed build script).

wiak

[wiak]

I now know why my initial firstrib_rootfs measurement was considerably larger...

I had run apt update, which pulls down lots of var/lib/apt (and dpkg-related) and /var/cache/apt stuff of course (most recent pkg database stuff for all sources.list repos and so on).

I'm not sure how much, if any, of that can be zeroed out thereafter without damaging apt - I suspect most of it since apt worked fine on first run and didn't have all that junk... I'll consult with fredx181 since he will know about such matters. However, I won't run apt update prior to building release iso...

Currently working on final build of WDLGO_UbuntuFocal64full (with large firmware/modules and wifi connect support) for release as iso (still some later work to do on the build script). I had intended making WDLGO_UbuntuBionic32 first (much smaller since first version of that planned for ethernet connection only), but I need to download some files prior to completing that one so working with what I already have on my system. Expect first iso release within a day or two.