minimal modular woof-ce puppy

[wanderer]

hi all

in my never ending quest to be annoying
i have decided again to request a minimal modular woof-ce puppy

why

well i think it would be very useful for the puppy community
both users and developers

the template would be

1. a bootloader module - syslinux

2. a kernel module

3. a core module - busybox

4. sfs modules - everything else

can woof-ce be made to build this ?

i use both puppy and tinycore/dcore so i have no dog (pun intended) in this race
but it is something i (and i think others) would like to see

wanderer

[s243a]

I think the easiest way to do this would be to split the puppy into components based on info in:
/var/packages/DISTRO_PKGS_SPECS
/var/packages/woof-installed/packages
/var/packages/builtin_files/*

Prior to doing this one would also want to define pseudopackages for the various compenets of the orginal woofCE rootfs.

[wanderer]

hi s243a

thank you very much for your reply

i have very limited knowledge (essentially no working knowledge) of woof-ce
but i understand (i think) as you mention
that it is built using lists of the included parts
however it would take a guru to know
the minimal components necessary to build the core
so this is where i hit a dead end

wanderer

[s243a]

hi s243a

thank you very much for your reply

i have very limited knowledge (essentially no working knowledge) of woof-ce
but i understand (i think) as you mention
that it is built using lists of the included parts
however it would take a guru to know
the minimal components necessary to build the core
so this is where i hit a dead end

wanderer

Is there a particular puppy that you would like this done for? If so I might give it a shot.

[wanderer]

thanks again s243a

in my opinion the most important thing is to pick something
that is as easy as possible to work with
what puppy would fit this criterion ?

maybe something that is fully developed
and has someone who supports it and has knowledge of it

i use upupbb32

but you may have one you think is better suited

devuan ?

in my opinion the core is key
the other stuff could be added later unchanged

is there a part of woof-ce that just builds the puppy core
before other things are added to it
the skeleton ?

anyway thanks for all your work
in my opinion just this discussion alone is very useful

wanderer

[dimkr]

We're getting close to that.

In https://github.com/puppylinux-woof-CE/woof-CE/pull/1948 and https://github.com/puppylinux-woof-CE/woof-CE/pull/1984, I've implemented:

• An alternative boot scheme: no initrd; instead, the kernel mounts the boot partition directly, and /init is responsible for SFS loading using overlayfs/aufs

• Applications are compiled from source and trimmed during 3builddistro, and if it's a non-x86_64 build, this is done using emulation

• A minimal dpup based on Debian Bullseye, with zero .pet packages used during the build

• Integration of kernel-kit into the build process, so the kernel is always updated to the newest bugfix version

The compiled packages (except busybox and other 'core' things) can be moved to a separate SFS, and that would be very close to your proposition.

[wanderer]

thanks dimkr

that sounds like the perfect base

something that is "almost there"
supported by someone who very knowledgeable and involved

is that a possibility ?

[dimkr]

It's possible.

I've set up automated releases for the next slacko64 at https://github.com/puppylinux-woof-CE/woof-CE/releases, to make development easier.

I want to set up something similar for the minimal dpup, but with automatically triggered releases (once a week? month?) and an optional auto-update mechanism that updates your system to the latest release on GitHub, by replacing the SFSs and vmlinuz. It's a 'rolling release', barebones Puppy.

If all applications are moved to a separate SFS, puplets based on this dpup will be able to use the same update mechanism, and that will solve the security and stability issue of one-time puplets that don't get any fixes.

[wanderer]

wow

thanks dimkr

s243a is that something you might be interested in

wanderer

[s243a]

thanks again s243a

in my opinion the most important thing is to pick something
that is as easy as possible to work with
what puppy would fit this criterion ?

maybe something that is fully developed
and has someone who supports it and has knowledge of it

i use upupbb32

but you may have one you think is better suited

devuan ?

in my opinion the core is key
the other stuff could be added later unchanged

is there a part of woof-ce that just builds the puppy core
before other things are added to it
the skeleton ?

anyway thanks for all your work
in my opinion just this discussion alone is very useful

wanderer

Would @wiak's WLDGO bionic be a good reference for the core part of your system (related to which packages to include) minus anything that puppy doesn't have related to getting apt/dpkg to work? Such a system would be about 20mB to 40MB but could be made smaller with more utilization of busybox.

[wanderer]

hi s243a

yes wiaks system
would seem to be a very good base

but i do not really understand it

and thanks again for all this work

wanderer

[wanderer]

hi s243a and wiak

i feel wiaks system is definitely the way to go to make a minimal modular puppy

i of course have been following weedog since wiak started posting
but it is very sophisticated and hard for me to get a clear picture of it in my mind

could someone make an example core iso with the system
and provide a brief explanation of how it was built
so i can start getting a grip on things

no rush of course
i will keep reading

thanks

wanderer

[s243a]

hi s243a and wiak

i feel wiaks system is definitely the way to go to make a minimal modular puppy

i of course have been following weedog since wiak started posting
but it is very sophisticated and hard for me to get a clear picture of it in my mind

could someone make an example core iso with the system
and provide a brief explanation of how it was built
so i can start getting a grip on things

no rush of course
i will keep reading

thanks

wanderer

I also don't know how wiak's system works. Wiak though is planning to publish some documentation about the system and put the code on github. Hopefully, I'll find some time to learn about it. Right now, I'm just looking at it as kind of a starting point for which packages to include in a minimal debian based system.

Woof-CE can produce a minimal system. However, some rootfs aps are GUI dependent in woof-CE. Also, I see no reason why a full system can't be separated into components based on the package manager metadta and distro specs. What's missing is separating out specific components in the woof-CE rootfs and I've identified some of this separation in my woof-next fork.

I don't to try to rank the two systems against each other. Woof-CE produces a very polished distro and I'm starting to notice wiak's work more as development progresses. What I think is different about wiak's system is that it might be more functional to build a distro from the ground up via the package manager because it doesn't have the gui dependencies that a minimal puppy system might. Perhaps with Scotmann's package manager (i.e. pkg) we can do the same with a minimal with CE distro as we can with wiak's system. I don't know but currently I don't believe it was a goal of woof-CE to build a distro in the 20MB to 40MB range whereas I think this is one of the things that wiak wanted to try.

[s243a]

hi s243a and wiak

i feel wiaks system is definitely the way to go to make a minimal modular puppy

i of course have been following weedog since wiak started posting
but it is very sophisticated and hard for me to get a clear picture of it in my mind

could someone make an example core iso with the system
and provide a brief explanation of how it was built
so i can start getting a grip on things

no rush of course
i will keep reading

thanks

wanderer

I also don't know how wiak's system works. Wiak though is planning to publish some documentation about the system and put the code on github. Hopefully, I'll find some time to learn about it. Right now, I'm just looking at it as kind of a starting point for which packages to include in a minimal debian based system.

Woof-CE can produce a minimal system. However, some rootfs aps are GUI dependent in woof-CE. Also, I see no reason why a full system can't be separated into components based on the package manager metadta and distro specs. What's missing is separating out specific components in the woof-CE rootfs and I've identified some of this separation in my woof-next fork.

I don't to try to rank the two systems against each other. Woof-CE produces a very polished distro and I'm starting to notice wiak's work more as development progresses. What I think is different about wiak's system is that it might be more functional to build a distro from the ground up via the package manager because it doesn't have the gui dependencies that a minimal puppy system might. Perhaps with Scotmann's package manager (i.e. pkg) we can do the same with a minimal with CE distro as we can with wiak's system. I don't know but currently I don't believe it was a goal of woof-CE to build a distro in the 20MB to 40MB range whereas I think this is one of the things that wiak wanted to try.

Just as a side note, if one of the goals of Puppy was to produce a very small system (say 20MB to 40MB) then one way to acomplish this would be to add metadta to /var/packages/DISTRO_PKGS_SPECS, which gave some idea of what type a system to include a line of DISTRO_PKGS_SPECS, within. For instance, in debain repos some packages are marked as "Essential". iN puppy/woof-CE, we could modify DISTRO_PKGS_SPECS to have the same kind of flag/metadata, and there could be a build option in woof-CE to only include the essential packages from within DISTRO_PKGS_SPECS.

[wanderer]

hi s243a

another way to find the minimal packages for a core
is to look at tinycore core
just open the cpio archive
or boot it and look inside

they also have the root filesystem in their repository
that they use to build the core

their core is all busybox ash
so no gui

then the next thing is to look at their gui tcz/sfs
they have a whole selection of window managers

and so on for all the other apps

this is what i did
and it is all there
and easy to sort out
i just was not able to translate it into puppy
since i know so little about the various components

wanderer

[wanderer]

hi s243a

another thing to look at is dcore-stretch
both the dcore core and dcoreplus (has a gui)

dcore is able to download and bundle apps
from debian stretch into sfs/sce

its also all there in the core cpio image
just boot and look inside

i have them in my corepup repository

wanderer

[wanderer]

hi s243a

yes after some more thought i think that is the way to go

just translate the tinycore/dcore cores and sfs/tcz/sce
into the puppy build lists

you could even use a lot of the actual pieces from the cores
to avoid the gui problem

what do you think

wanderer

[s243a]

hi s243a

yes after some more thought i think that is the way to go

just translate the tinycore/dcore cores and sfs/tcz/sce
into the puppy build lists

you could even use a lot of the actual pieces from the cores
to avoid the gui problem

what do you think

wanderer

If dcore has the file /var/lib/dpkg/status

then we can easily see what packages went into building it. Otherwise, you'll have to look around and find the spec somewhere or alternatively reverse engineer it by looking at the iso to see which files are included.

[wanderer]

hi s243a

i just took a look at dcore and dcoreplus
and realized something that should have been obvious to me before

not only do we have the packages in the core
but we have all the scripts that are completely different

its not that we couldnt find correlates for all the components
its that we would need to rewrite everything

i think dimkr has offered all of us a great opportunity
he is working toward this end from the inside of woof-ce
which makes more sense

lets try to learn his system

wanderer

[wanderer]

hi dimkr

i hope you are still reading this and havnt given up on us

can you please tell us more about your system and how you are doing things

is there an iso we can look at

how do we run your woof ce to make an iso

thanks again

wanderer

[wanderer]

hi dimkr

i did not realize that you are the famed iguleder

we are not worthy

welcome back

as i said please tell us more about your system

thanks again

wanderer

[dimkr]

hi dimkr

i hope you are still reading this and havnt given up on us

can you please tell us more about your system and how you are doing things

is there an iso we can look at

how do we run your woof ce to make an iso

thanks again

wanderer

Yes, of course.

All my work is in https://github.com/puppylinux-woof-CE/woof-CE/pull/1948.

To get the relevant woof-CE tree: git clone https://github.com/puppylinux-woof-CE/woof-CE -b c201-v2

There are two configurations (x86_64, debian, bullseye-cros, and arm, debian, bullseye-cros).

You might need to install extra build-time dependencies, see https://github.com/puppylinux-woof-CE/w ... llseye.yml. This is the series of steps needed to produce bootable images, and it includes a short series of tests that boot Puppy in QEMU and makes sure the browser is able to start, etc'. Also, the kernel is built using kernel-kit, and gets automatically updated to the latest bugfix version (right now both configurations use 5.4.x).

The ARM configuration produces an image with the Chrome OS partition layout, that boots and works on the Asus C201 Chromebook. I still haven't tried to port it to other models. It's a bootable 2 GB image that can be written to a flash drive. Inside this image, there's a 16 GB image (yes, 16 GB image inside a 2 GB image - it's a sparse file) that can be written to the 16 GB SSD these laptops come with, and that installs Puppy persistently.

The x86_64 configuration produces two images, one with the Chrome OS partition layout (like the ARM one, same idea) that I still haven't tested on x86 Chromebooks, and one that uses extlinux and supports BIOS boot (the "legacy" image) and works with regular PCs. The "legacy" image is used for automated testing, because QEMU simulates regular PCs.

All variants of this Puppy are "weird" - these are not "full" installations, nor "frugal" installation. Instead, a tool called frugalify (https://github.com/dimkr/frugalify) is used as PID 1, and this tool sets up a union file system of the /save directory, and SFS files at /. It's similar to a frugal installation, but without savefiles (at least, for now) and without an initrd.

I've set up automated releases in a fork of woof-CE, which I use for testing. You can get "legacy" x86_64 images at https://github.com/dimkr/woof-CE/releases - for example, dpupos-8.0-ext4-2gb-x86_64.img.gz at https://github.com/dimkr/woof-CE/releas ... x86_64-v70. gunzip, dd to a flash drive and you're good to go.

This Puppy calls itself "dpup OS", because I didn't want to steal the name "BullseyePup" from whoever wants to build a traditional Puppy based on Debian Bullseye, because I want it to be a good, generic template to builds other things from, and because it's a replacement for Chrome OS, so it has to end with "OS".

Right now, these builds come with a small range of applications that covers the basics (Geany, l3afpad, LXTask, mtPaint, Firefox, etc'), but I might move the applications to a separate SFS. Still need to think how to do this, because Firefox is not built from source, so this SFS splitting mechanism must be able to deal with Debian packages as well.

[s243a]

Hello @dimkr ,

you system sounds interesting but I don't understand it. Can you explain the more the motivation for some of the design choices like separate sfs files for each package and the sparse files. I notice that the 2GB sparse file is only 296MB when compressed but maybe it isn't 2GB when uncompressed since it is a sparse file. I'm wondering if the spare files are meant to be test images for qemu or something. I used spare files to back up a partition before.

[dimkr]

separate sfs files for each package

Maybe not for each package, but for groups of packages. But the idea is that if the system consists of multiple SFS files, a puplet can do something like removing the SFS that contains the browser, and put a different SFS containing a different browser instead. Also, separation into multiple SFSs will allow more efficient updates, using binary delta.

the sparse files.

Many Chromebooks have a 16 or 32 GB SSD. A Puppy image, with matching size and the Chrome OS layout, can be written as-is to the Chromebook's SSD as means of installing Puppy.

Because I want to support both SSD and flash drive boot, I produce 2 GB images that can be flashed to a flash drive of any size that's at least 2 GB. When the Chromebook is running Puppy from a flash drive, it can be used to flash the 16 GB image on the flash drive to the SSD.

Why is the use of sparse files important here? Because that's what allows a 16 GB file to be inside a 2 GB image. Puppy itself is, say, < 300 MB, so the 2 GB image is ~1.5 GB of zeroes, and Puppy, and the 16 GB image is 15+ GB of zeroes and Puppy. Because sparse files only take the physical space needed to store their non-zero part, the 16 GB image takes only ~300 MB when saved as a sparse file, so there's enough room inside the 2 GB image for the 16 GB image.

Without the use of sparse files, the user would have to download the 16 GB image when booting off the 2 GB image, store those 16 GB somewhere, then flash to the SSD. Inconvenient!

I'm wondering if the spare files are meant to be test images for qemu or something.

As I said, automated tests are done using the x86_64 BIOS image (= not the x86_64 image for Chromebooks). The use of sparse files is irrelevant here and has only one purpose, which is easy installation on Chromebooks.

[wanderer]

hi dimkr

your system is very unique and sophisticated
very beautiful
but because of my limited abilities
unfortunately beyond my ability to fully grasp

i am looking for a puppy version of tinycore or dcore
where there is a small busybox based core built by woof-ce
and everything else is added as an sfs file
can your system build this

wanderer

[wanderer]

hi s243a and dimkr

can we just stop the woof-ce build process at a certain point after it has built the root filesystem

wanderer

[s243a]

hi s243a and dimkr

can we just stop the woof-ce build process at a certain point after it has built the root filesystem

wanderer

Just modify /var/packages/DISTRO_PKGS_SPECS and change what you don't want from "yes" to to "no".

[s243a]

hi dimkr

your system is very unique and sophisticated
very beautiful
but because of my limited abilities
unfortunately beyond my ability to fully grasp

i am looking for a puppy version of tinycore or dcore
where there is a small busybox based core built by woof-ce
and everything else is added as an sfs file
can your system build this

wanderer

It sounds like @dimkr is planning on breaking the base sfs file of puppy into smaller pieces (a greater number of sfs files). These smaller pieces could be used to build your tinycore like system. Furthermore, if puppy ever adds the option to load sfs files via symlinks (maybe a configuration option), then puppy could act even more like tinycore should the user chose.

[wanderer]

hi s243a and dimkr

dont you think that just stopping the build at a certain point is the best way
since it wont change anything in woof-ce
and we can just use the latest woof-ce and stop that build where we want to

wanderer

[dimkr]

dont you think that just stopping the build at a certain point is the best way

Define 'certain point'.

The only point where you can stop woof-CE and get a bootable image is the end of the process. What you're suggesting is analogous to stopping the assembly of a car before parts of the engine or the brakes are added, in order to make the car lighter or more 'modular'.