Puppy Linux Discussion Forum

wanderer wrote: Sun Mar 21, 2021 3:41 pm

does woof-ce download the binaries from the target distro and make them into pets
can the pets be then made into modules (sfs files)
eliminating any need for an intermediate step

IT's worth looking at the Readme here:

Open a terminal in the woof-out_* directory.

Download package database files

./0setup

OPTIONAL: Tweak common PET package selection. You can edit the variable PKGS_SPECS_TABLE in file DISTRO_PKGS_SPECS-* to choose the packages that you want in your build.

Download packages

./1download

About 500MB drive space is required, but this may vary enormously depending on the package selection.

Build the cut-down generic Puppy-packages

./2createpackages

https://github.com/puppylinux-woof-CE/w ... ne-scripts

So if I recall correctly, what the ./2createpackages script does is extract each package into a directory. One could convert these extracted directories into sfs files, however, for many of these packages there are various hacks applied which are analogous to an install or configure script. One might want to incorporate some of these hacks into a sfs module.

Here are some of the hacks:
rootfs-hacks.sh

Note that the directories aren't necessarily a single package from the binary compatible distribution. The modules can also be from one of the puppy repos or they could be built from source. Also some of these packages might be an amalgamation of more than one package from the binary compatible distribution. For instance in upupGG+D util-linux is defined as follows:

yes|util-linux|util-linux,fdisk,mount,uuid-runtime,bsdutils,libuuid1,libblkid1,libfdisk1,libmount1,libsmartcols1,libfdisk-dev,libmount-dev,libsmartcols-dev,uuid-dev,libblkid-dev,libmount-dev|exe,dev,doc,nls| #ADDED 20150717-08 20150730 20150808

in /var/packages/DISTRO_PKGS_SPECS. The third filed in the above line defines all the debian and ubuntu packages that are amalgamated into a single built-in package called util-linux.

the repositories are important because
people will be sharing their work by exchanging modules
as well as making it very fast to build an iso
and eliminating the need to download modules more than once

It would certainly speed up the build process.

also the modules will not be merged just placed unchanged into a folder in the iso
and then linked either by unionfs or symlinks
so there is no processing other than simply copying them to the folder to make the iso

DISTRO_PKGS_SPECS, has some inherent modulation where a single built-in package may be comprised of more than one "distro compatable package". This reduces the number of choices that a distro builder has to make in deciding what to include in the distro. If there was a puppy repo of modules, that had this level of amalgamation or more, then the amalgamation would speed up download time by having fewer tcp connections to make and greater compression efficiency. Also unitis with greater coursness are easier to understand and think about as a whole. The level of granularity vs amalgamation could be debated ad-infinitum.

wanderer wrote: Sun Mar 21, 2021 10:33 am

like a build time in 1 minute or so
on their own computer

That quite an ambitious goal! However, One could try making an archive of common modules that would be a single download. If this significantly reduced the download time and the number of modules wasn't too great then I could imagine a rapid build process.

the only thing needed to start
is for some guru to get woof-ce to make the first 2 isos
(a initrd only iso and a base x main.sfs file)
and then more and more modules can be added over time

i think s243a is doing that now

i cannot do it because i dont understand woof-ce well enough
to have it build the isos or the modules

lets give this a try and see if it proves to be useful
nothing to lose

william

In the near term, I want to further modularization fossapup64. Perhaps 2-5 times the level of modularization as currently exists in fossapup. Near-term I want to package this as a chroot, sandbox or container for experimentation testing and also for the purposes of running specific applications (e.g. a browser) perhaps in an older system. I would also like to experiment with @taersh lazy pup ideas as is embodied in the sdrv (PM taersh for this sdrv and the related init).

Once taersh builds a Hirsute Hippo with the modified init, I can look at applying these ideas to an ISO. However, I'm not sure if this is a good idea or not. As I note that we need to distinguish between modules from a design perspective and final implementation layers. Aufs has more overhead than overlayfs so a greater number of layers might be more applicable to an overlayfs or symlink approach. We won't know unless we test this. Also note that if we break the final distro into more layers then the file size will be bigger due to things like folder duplication (the actual fs entry that says a folder exists) and compression efficiency.

hi s243a

when i am talking about a build time of 1 minute
i am talking about from the local repository on your computer
not with the download time included

because of the number of modules
that will probably be included in a normal build
i think only the symlink option is feasible

looking at tinycore as a model i think is the best approach
since they have already done it successfully

also

you are doing an enormous amount of work

in my opinion
making just the 2 initial components
the initrd and the base x main.sfs isos
is enough of a start

i see this as a very lengthy process
that hopefully the community
including myself
will be able to help with

so please do not exhaust yourself

and once again
thanks for doing all the work

you are the guru

i am following along
and hopefully will be able to be of some use soon

william

hi s243a

since you mention there are a lot of post download hacks in woof-ce

would it be more straightforward
to simply let the the woof-ce build process
progress normally to the end and build an iso

and then break the iso apart manually (with a script)

that way you would not have to get involved
with all the convoluted details of the build

and could use woof-ce without any modifications

you could then make the components you isolated
into modules for our system

the script would only have to have a list of the components/apps
you wanted to isolate (copy over)

you also probably could use the built in modularity functions in woof-ce
like making the drvs to help

you could also do this with any iso that was already built
new or old of any system

also if you wanted to
you would only need to download the iso
not the entire build system

i did this a long time ago
and it worked fine
but i did not have enough knowledge of the components/apps
to get much beyond the basic x desktop and dillo browser stage

since this is something even i can work on
i am going to start doing that now
and will report back

william

wanderer wrote: Sun Mar 21, 2021 11:19 pm

would it be more straightforward
to simply let the the woof-ce build process
progress normally to the end and build an iso

and then break the iso apart manually (with a script)

that way you would not have to get involved
with all the convoluted details of the build

Indeed, which is what I suggested previously as one of many mechanisms that you could employ, though I didn't refer to one remastering technique being module extraction via post original iso created script (though that is what I automatically do to create Puppy dpkg_apt_PAM addon from WDLGO_UbuntuFocal64 iso creation - i.e. my build system first builds WDLGO_UbuntuFocal64 and then finishes by extracting the addon sfs module for use by Puppy FocalPup64):

wiak wrote: Sat Mar 20, 2021 9:00 am

You can also employ simple remastering techniques as part of the process, bearing in mind that remastering steps can themselves be scripted.

The 'difficult' part is indeed determining and auto-extracting all the parts a 'module' depends on. If the upstream repo Puppy is depending on is Debian-based that could include simply checking (via the script you are making) the dependencies listed in the associated deb package control file, EDIT: but that means recursively since dependencies of dependencies - it can be easier at the woof-CE build stage since dependencies should already be getting resolved then...

thank you very much wiak

the fact that you think its a good idea (and have already suggested it) is super important

i hope you will continue to provide words of wisdom as this goes along

any ideas will be greatly appreciated

william

But note my last minute EDIT comment in above post wanderer. Such extraction is not always 'easy', but it is a doable and a good way indeed of avoiding woof-CE complexity itself - probably very doable actually, and a good approach/way-forward.

hi wiak

yes not easy
but certainly not anymore difficult than rewriting woof-ce

also it is global
and can be used with any iso from any system

and i actually have already done it (up to a point)

so thanks again

william

hi s243a

would you consider using the system
where modules are manually broken out of isos
by scripts
rather than trying to rewrite woof-ce

there seems to be overwhelming advantages to it

and it think it would fit your advanced needs
like chroot stuff etc

william

hi all

so this is what i am doing

i am going to manually move stuff
from the normal fossapup64 iso into modules

and make a minimal modular system from them

i am going to use tinycore as a guide
since they have already divided things out
and solved the basic problems like symlinks etc

fossapup64 already is already very modular
and this should help a lot

my goal is to create the base isos
initrd only and
initrd plus base x main.sfs iso

since fossapup64 already has essentially a base x sfs file
i will use that
and break things out of that

when i have the base isos done
i will post them in the corepup repository
with a link here

i will report on my progress as i go along

william

wanderer wrote: Mon Mar 22, 2021 12:46 am

hi s243a

would you consider using the system
where modules are manually broken out of isos
by scripts
rather than trying to rewrite woof-ce

there seems to be overwhelming advantages to it

and it think it would fit your advanced needs
like chroot stuff etc

william

I will likely try it both ways.

wanderer wrote: Mon Mar 22, 2021 12:47 am

hi all

so this is what i am doing

i am going to manually move stuff
from the normal fossapup64 iso into modules

and make a minimal modular system from them

i am going to use tinycore as a guide
since they have already divided things out
and solved the basic problems like symlinks etc

fossapup64 already is already very modular
and this should help a lot

my goal is to create the base isos
initrd only and
initrd plus base x main.sfs iso

since fossapup64 already has essentially a base x sfs file
i will use that
and break things out of that

when i have the base isos done
i will post them in the corepup repository
with a link here

i will report on my progress as i go along

william

It's a good exercise and if you run into issues one thing you might try is replacing a given module with the dcore equivalent. Because tinycore uses symlinks to load packages dcore might have done things like compiled packages in a way to not use relative linking. Relative linking sometimes causes problems when you move parts of debian/ubuntu packages around. I ran into this issue with some java packages.

s243a wrote: Mon Mar 22, 2021 2:36 am

It's a good exercise and if you run into issues one thing you might try is replacing a given module with the dcore equivalent.

Exactly, by using appropriate dcore you can also get a lot of inherent compatibility with what standard FossaPup64 uses. Can possibly remove a lot of the work otherwise involved and dcore fits in well with wanderer's own knowledge re tinycore more specifically. Great thing about dcore is that these are not specific tinycore repo creations but derive from upstream Debian/Ubuntu repos. As you may know, I have also had an interest in dcore in the past (2017 as login mcewanw), and still do, though haven't time to do anything with it these days myself:

http://www.murga-linux.com/puppy/viewtopic.php?p=946635

Such compatibility was equally important to me when I produced that dpkg_apt addon (though that addon came as an after-thought rather than ever being part of my intention - I simply realised it was likely to work well with FossaPup, with some tweaks here and there).

Good point about the symlinks situation.

hi s243a and wiak

will keep the symlinks thing in mind
when i move things

and possible replacement with dcore/tinycore extensions for stuff
also dcore can build whole app packages (sce) which also may be useful as modules

william

Another issue that you may need to consider is:

Most distros that use a package manager utilise some sort of database to keep track of what has been installed. That relationship between the package manager database and the addon module is difficult to maintain.

I've thought about this some time back because of course WeeDogLinux is also inherently capable of being used in a very modular fashion since, by design, it currently allows for up to 100 sfs module layers (no adrv, ydrv and so on in WeeDogLinux - rather can be any number sfs from 00 up to 99). On the whole I've concentrated in making the WDL build system as simple and modular as possible (which woof-CE certainly isn't) but modularising the distro itself is not the same thing (woof-CE doing a reasonable job improving modularisation of the resultant distro recently).

For example, for a Debian-based distro that uses official Debian package manager dpkg/apt, the dpkg database becomes difficult (if not impossible) to maintain via addon sfs modules.

My feeling regarding above issue is that it is better in this circumstance to regard the 'modules' as portable app type addons that do not register at all with the package manager database (after all, they are read-only filesystems so you cannot remove them via the package manager anyway...). But that viewpoint is also a limitation in that the package manager no longer really manages all packages on the system and you have to be careful to avoid conflicts between the sfs modules and any packages that might get installed by the package manager. Effectively, you end up with two different package managers (that know nothing about each other usually): the system's usual package manager, and the package manager that handles the sfs module add-ons. Actually my dpkg-apt package manager is a special case since it is a normal 'package manager' itself, with its own dpkg database, that keeps track of everything installed via dpkg_apt, but it knows NOTHING about what is already installed in Puppy... and hence the slimmer the Puppy core used with it, the better... at least in terms of avoiding duplication.

hi wiak

i agree

the module system is essentially portable
so you dont really need a package manager
just a list of whats loaded or not

tinycore and dcore provide this function in their app managers

william

wanderer wrote: Sat Mar 13, 2021 11:09 pm

hi 666philb and everyone

i would like to try to make a more minimal modular form of fossapup64 from woof-ce

that is get woof-ce to build just

1. an iso with only the initrd and an empty main.sfs file
2. an iso with iso 1 + a main sfs file that only contains basic x jwm and an xterm

so that one can start with the very basics and add applications as sfs files

thanks

william

you could try remastering a very small puppy as a base; eg: turbo puppy (extreme).

Post can be found here.
Since the download links aren't working, try the one on archive.org, here.

It is to say this won't be a (excuse the pun) "walk in the park", but with some rudimentary knowledge in shell scripting, some virtual machine and patience, you could get what you want

wanderer wrote: Mon Mar 22, 2021 7:40 am

hi wiak

i agree

the module system is essentially portable
so you dont really need a package manager
just a list of whats loaded or not

tinycore and dcore provide this function in their app managers

william

Yes, but dcore does, as far as I remember, keep track of its sce package dependencies. I'm suggesting (and I take it you agree) that such complexity is not itself required in what you are producing here. Rather just design the modules carefully so they don't mess each other up (and any already installed stuff, which should be as little as possible, the small core you want, to avoid such issues). Nice simple system in other words - not sophisticated, or perfect in the sense of keeping track of packages, but has the advantage of no need to worry too much about that - nice for hacking and experimental work too.

hi wanthinker

thanks for your help

yes very small puppies are good candidates

the nice thing about this system is that you can use it with any iso from any system

over time we should get a list of the components for each build
that can be used for each level and each app

william

hi all

so i am now working on reducing fossapup64 to the 2 base isos

as expected if you remove the main files from the iso
the initrd will not be able to find them
and will drop to the shell in the initrd

so thats the 1st iso - initrd only

now i will work to reduce what is in the main.sfs file to base x only

i think the best way to start is to (as 666phib advised)
remove stuff from the distro pkg specs template
and see if i can get it down that way
and still have it make an iso

william

In a previous post/thread, I wrote about some issue that one might want to consider when deciding how busybox fits into a modularization scheme. I wrote my first draft script.

I originally thought about basing it on a script I wrote for woof-next that creates a wrapper script for tinycore busybox by placing all the busybox util symlinks in a directory called /bin/bb_utils. These symlinks either pointed to the suid or non-suid version of busybox depending on which binary they were part of. The wrapper script simply calls the symlink.

I ended it almost completely rewriting it because puppy doesn't split busybox like this so the wrapper script isn't needed but I did keep the same target directory for the busybox symlinks. The reason being so that the symlinks won't conflict with any of the full utilities. I also included some of the symlinks in their normal location if an only if the source directory (i.e. the puppy we are modularization) had such a symlink. As far as I can tell these symlinks all reside in the /bin folder. I presume utilities which normally are located in other folders are replaced by the full utilities. Anyway, I created a squash file of the output that one can download if they like:

puppy_fossapup64_9.5_busybox.sfs (download)

Not all options of the above script are tested and more capabilities will be added.

hi s243a

thank you for all your hard work

interesting busybox stuff
the busybox i came across in fossapup64 seems to be statically compiled
probably in the initrd

on my end
i have set up a system to minimize modularize the main.sfs file in fossapup64
which is the next step in setting up the modular system

its a hybrid of using woof -ce and manual manipulation
working well
could use it with any woof-ce puppy

also
very interesting to compare tinycore with fossapup64
to see where we could be

well back to playing

william

no way thruhammer

im on a roll

here is what i will have in the main.sfs

x-server - xorg or xvesa (dont know which is best) any ideas anyone ?

jwm - i will use my super cut down corepup-jwm which has only one config file corepup.jwmrc

and an xterm (dont know which one) any ideas anyone

the xterm will use busybox (from the initrd)
ash and vi
and (dhcp and wget ?) any ideas anyone

to provide a shell filemanager editor and wired demon and download manager

its good to use the command line

minimalism

william

hi all

i am slowly taking out uneeded packages
from the fossapup64 main.sfs
by no-ing out entries in the distro-packages-specs file

however this is a slow process
because i have to test each deletion individually
to see if it causes the x server to fail

if anyone has an idea what can be removed and what must stay
it would help a great deal

i only want an x-server
i will then put a window manager (corepup-jwm) and an xterm in (or i could leave rxvt)

i dont need any other sfs file at this time
no devx adrv fdrv etc

any help is appreciated

william

wanderer wrote: Wed Mar 24, 2021 4:23 am

hi all

i am slowly taking out uneeded packages
from the fossapup64 main.sfs
by no-ing out entries in the distro-packages-specs file

however this is a slow process
because i have to test each deletion individually
to see if it causes the x server to fail

if anyone has an idea what can be removed and what must stay
it would help a great deal

i only want an x-server
i will then put a window manager (corepup-jwm) and an xterm in (or i could leave rxvt)

i dont need any other sfs file at this time
no devx adrv fdrv etc

any help is appreciated

william

One could write a script to tell you what you cannot remove based on dependency resolution. pkg is supposed to have an option "--list-deps" to do this but when I tried it it diddn't work so I'd need to investigate it further.

thanks s243a

will continue the process of trial and error

post if you find an easier way

william

...

wanderer wrote: Wed Mar 24, 2021 8:02 am

thanks s243a

will continue the process of trial and error

post if you find an easier way

william

Hopefully you are doing some notes of what worked on removal.
Perhaps you could write a script then to automate these tasks for later editions?

hi taersh

yes i am keeping track of everything i am doing

eventually we will know what needs to be included in each module
and can write scripts to do all this automatically

the distro-packages-specs file is the key to the sfs builds
but there are other sub systems that need to be considered

this would be easy and fast if a woof-ce guru
would simply tell us what needs to be marked in/out
but since this is not happening
its all trial and error
and is going to take time

william

hi taersh and s243a

there are 2 ways to build our modular system from puppy

1. let woof-ce build it the normal way
this has many disadvantages (too many to list in this post)

2. use the puppy initrd as the core
let woof-ce build the sfs files from puppy packages
and symlink the sfs files to the initrd core
(we could also use a unionfs or both unionfs and symlinks together)

the second one has many advantages (too many to list in this post)

and i am going to explore this second variation as well as the first

it will be all puppy
just have a different boot process

william