s243a's Possible Plans for woof-next

[s243a]

Today, I started looking at my previous fork of woof next,
My latest code?
Old Murga-linux thread
Original Branch by Jamesbond
Official Fork into the Woof-CE project (i.e. zwn)

I've previously used the code to create:
http://wikka.puppylinux.com/Tiny_Puduan
Related threads:
tiny_puduan_ascii-PreAlpha11.5 (made via a woof-next fork)
alternative puppy build system

Woof-next was perhaps an effort by jamesbond to build an easier to maintain build system and served as a stepping stone for the fatdog64 build systems, which has code that originated from the woof-next branch. At one point on the murga-linux forum there was much interest in the woof-next branch because of the perceived difficulty of modifying Woof-CE for other use cases such as say a distro without a gui or one that is not debian or Slackware based.

Whether or not these criticisms are valid the design of woof-next is inherently more flexible because the build system resembles a domain-specific language used to build distributions.

When I built Tiny Puduan, I learned interesting things such as the differences between the tinycoure version of busybox and the puppylinux version and I was able to break up the rootfs of puppylinux into more modulre components. Unfortunately, I did so on code that wasn't the latest both because I used the main version for the woof-CE branch and also based some of it on the root-fs file system that mistfire selected in TazPup.

This illustrates the difficulty for an outsider trying to develop and maintain an alternative build system for a project that has a team of people working on it with likely more linux experience. My current idea to get around this challenge is to be more agnostic as to whether the build system is independent or derivative of the original Woof-CE project.

By this I mean that while I can have root-fs components incorporated into woof-next project, these should be optional so that One, could alternatively use an existing distro-ISO or build system as the starting point of the build and have woof-next add or remove components as needed to finish the ISO.

This way if woof-next development falls behind woof-CE then simply use woof-CE or alternatively an ISO generated from woof-CE as the starting point and remove/add what is needed. Scotmann's package manager (i.e. pkg) is a greater tool to remove unwanted software because it doesn't have the gui dependencies that the ppm has and doesn't care whether a package is user-installed or built-in provide the right options are supplied (e.g. the HIDE_BUILTINS environmental variable).

When adding packages to some base (e.g. a root-fs) one can either use the build tools included within woof-CE or use a package manager in a chroot environment. One simply needs a directive to change the install method, and then the rest of the specification should be mostly independent of the install method. One could even use puppy specific functions like petget and remove_builtin (check spelling) commands to strip away the puppy components as needed.

[s243a]

The reason that I actually started looking at this project again was because I wanted to experiment with apt on upupGG+D, both for the build-source functionality and also to see if it would have better success installing libreoffice. I do like the puppy specific package managers such as the official one (i.e. ppm for puppy package manager) and also Scotman's package magnmanager (i.e. PKG) but there is more development hours related to both the testing of apt/dpkg and also testing that packages are properly installed via these tools.

The option to be able to install alternative or parallel (via syncing) package managers gives the user more flexibility and can help with troubleshooting. When building a distro my typical preference would be to use one of puppy's two official package managers but I do also like to experiment with the Debian ones. In my case here, I'm not even talking about using build tools to construct an alternative to upupGG+D that has apt. Instead, I just want to write a script to sync up the databases of the two package managers and some of the code to do this can be found in the woof-next project and more specifically in my fork.

Specifically, I want to try this for libreoffice because both package and ppm aren't handling dependencies very well here due to multiple alternative packages (e.g. different versions of the jdk/jre) and I've seen both PKG and the ppm be inconsistent about which version they select. For example, they might install a different "headless java" from the main java.

I don't have high hopes that apt will solve my libreoffice problems but there has been enough people enquiring about how to use apt with puppy that I thought I would provide a more detailed solution.

[s243a]

Another reason, that I started looking at woof-next again is because of wander's request to build an absolute minimal puppy core with woof-ce. See post:
viewtopic.php?p=9818#p9818

This can be done as follows:

Reduce the specified packages in woof-CE to the minimal few that I want and then build using woof-CE.

After doing this we will strip the build down further. To see how first look at how I separated puppy components in woof-next. For example, I have:
puppycore_Xstartup
puppycore_icons
puppycore_locale
puppycore_net
puppycore_noarch]
puppycore_petget
puppycore_rox
puppycore_sysinit
puppycore_users
puppycore_utils
puppycore_xdg

What one can do is create a package list for each one of these packages, and insert this metadata into the puppy that you want to strip. Once the metadata is inserted you can then use a tool like PKG to uninstall this pseudo-package. Each of these pseudo packages are components of puppylinux root-fs albeit not as up to data as the official branch.

[s243a]

I've decided that I like the Weedog WLGO concept ( by @wiak ) of making a very minimal rootfs that can be used for one of three purposes:
1. Form the base of a new distro,
2. Upgrade an existing distro or three
3. Used as an add-on to provide new functionality to puppy.

See post:
viewtopic.php?p=16498#p16498

It occurred to me that I could create a small distro like this via woof-next. At some point I can lean about wee dog so that I can learn the pros and cons of each approach.

Anyway, my idea here is to use as little of the puppy rootfs as possible and to provide no Xorg/Xserver functionality. The reason being is that the Xorg components don't often need to be upgraded in order to get a browser working. As a consequence said system would be a good base for a minimal upgrade in many cases.

[rockedge]

I'm in agreement on the concept.
I have a nice Fossapup64-WDL_GO going with a full LAMP (Apache being interchangeable with Hiawatha) and Zoneminder install I was able to accomplish with the APT package manager and PAM capabilities of Ubuntu/Debian. This is a proof of concept and testing for @wiak for refinement purposes.

Let me know when you are ready and I'll test out what you come up with.

a script to sync up the databases of the two package managers

This I am also very interested in. At the moment I have the Fossa-WDL_GO running with 3 package managers and each one works!
PPM and Pkg are in sync..but what if there was scripting that also let APT be in sync and all 3 know what is installed and where????........

Great stuff @s243a , keep up the good work

[wanderer]

hi s243a

i of course am very interested in your project

and i think it will be very very very useful
both to the users and developers of puppy

if puppy was modular
one would only need to work on one module at a time
greatly simplifying the task of adding and updating things

and of course it would also allow puppy users to easily create their own particular version
with just what they want

in addition all the other great puppy addons
like portable apps
and collections of desktops etc
could be easily worked on added and removed

i think it will expand the available apps
and encourage others to contribute

another great benefit is that
if building a puppy from woof-ce
was simple and fast (because it was minimal and modular)
it would encourage people to learn and use woof-ce

i will follow this thread (or whatever thread you use) with great interest
and will test (if its simple enough for me to understand)

and thank you for doing all this great work

wanderer

[wanderer]

hi s243a wiak rockedge and everyone

is it possible to write a small script
like woof-next

that builds 4 components
from puppy woof-ce
and assembles them into an iso

1. a bootloader module - syslinux

2. a kernel module

3. a core module - busybox

4. a sfs gui module - jwm-rox

these 4 modules could be built and kept in a repository
so they do not have to be rebuilt each time

other sfs modules like firefox and geany could be added as desired
and also kept in the repository
so they would not have to be rebuilt each time

the modules could be updated as needed individually

then non gurus could assemble their own versions
and maybe learn a little bit

just asking

wanderer

[rockedge]

@wanderer
it would be worth trying the concept out. It would be "lego-ish" and one could start off with really minimal blank slate like what is possible with Void Linux but it would be a Puppy Linux.

[s243a]

@wanderer
it would be worth trying the concept out. It would be "lego-ish" and one could start off with really minimal blank slate like what is possible with Void Linux but it would be a Puppy Linux.

I think that @wiak, has a better grasp on the lego-ish style approach. That said, I can look at the output of what wiak provides to get an idea of what packages to included and provide a woof-next spec as an alternative way to produce a similar distro.

I'm actually more inclined to merge layers rather than add layers due to the overhead of aufs. This isn't an issue with the tinycore symlink approach. The following article suggests that overlayfs has less overhead:
Good Riddance to AUFS

so maybe with overlayfs more layers makes sense. Since Wiaks W_LGO/weedog approach uses overlayfs perhaps it is better suited to more layers.

My latest experiment is to use woof-next to build a pure cli system. I've included midnight commander, more full versions of the core utilities then puppy, and also some dev stuff like strace, gcc-core and perl, it is larger than wiaks W_LGO. It is perhaps about 80MB whereas wiaks w_lgo is in the 20-40MB range.

Note, that in this experiment the libs are likely newer than the wiak W_LGO version because I'm using the kali-rolling 32bit-binaries. No hacking tools are are included. I'm just trying to demonstrate what one can do with woof-next and I will likely try a different base next time (e.g. debian sid) rather then trying to release a polished version of puppy. My focus is on the method rather than the output.

I would actually like some more flexibility here so maybe I'll use this cli spec as a starting point and then write a script to produce a more minimal version or alternatively, add a way to separate things into separate layers, which I can later merge if I desire.

My previous experiment was upgrading Puli/Xenial64 with wiaks W_LGO. In this current experiment my first thought with Kali was to upgrade attack-pup because I was curious what was included in this old puppy. However, since the libs of kali-rolling might be too new for attack-pup (will verify later), I'll try upgrading upupGG+D (AKA grovypup) instead, and I'll create an sfs that has there separate root filesystems and utilizes hardlinks to save space. The systems are:
1. A pure cli system (Kali-rolling)
2. A pure upupGG+D
3. upupGG+D with core libs upgraded by merging with system#1 (i.e. kali-rolling cli system).

Each of these could be separated into a separated sfs file. However, if the merged system is in ram then it will load faster and won't depend on storage media once running. For many modern systems the size of puppy is small relative to the amount of ram so the size of said system shouldn't be an issue for many modern computers. However, in ram limited systems the base sfs file doesn't have to be loaded into ram.

Now again my intent here isn't a commitment to any specific version of linux as the base system. This is just an an example system, and said system will give me apt/dpkg support in 2 our of the 3 root file systems. The separate systems provide options for chroot systems, containers and midleboxes. A middlebox can be used to help ensure that network information isn't leaked from a network limited system (e.g. tor only).

I use upupGG+D on one of my computers so adding a separate system to the upupGG+D sfs file could be of some use to me provided the computer that I'm running it on has sufficient ram. If not, I can just run the sfs in a seperate container or chroot and not use it as my boot system. I don't actually know how to use kali to do any types of attacks. The only related tool that I use is nmap to see what is going on on my network. nmap and many other tools are available in most linux distros. I think the main advantage of the kali repos is if you want access to freeware or proprietary tools. I'm more interested in open source tools because I want to understand how things work.

[s243a]

hi s243a wiak rockedge and everyone

is it possible to write a small script
like woof-next

that builds 4 components
from puppy woof-ce
and assembles them into an iso

1. a bootloader module - syslinux

2. a kernel module

3. a core module - busybox

4. a sfs gui module - jwm-rox

these 4 modules could be built and kept in a repository
so they do not have to be rebuilt each time

other sfs modules like firefox and geany could be added as desired
and also kept in the repository
so they would not have to be rebuilt each time

the modules could be updated as needed individually

then non gurus could assemble their own versions
and maybe learn a little bit

just asking

wanderer

I'll give some thought to this. I think for the core-module you might like what wiak is doing with his W_LGO/firstrib/weedog stuff. I haven't looked too closely at it yet but @wiak has given much more thought to this stuff then me. Note that even tinycore includes more than just busybox and I think this is in part because it doesn't use a static version of busybox. While static libs are nice from a system robustness perspective I think that at times they may result in a larger sytem rather than a smaller system.

Regarding the bootloader/kernal and not mentioned the startup system (e.g. initrd), I'm not that strong on this stuff so I can't focus much on it at this point. I mostly at this point use what is provided in build scripts first developed by others but I think that a more flexible initrd system might be nice. Of course full installs don't use a separate initrd file and there are large differences in the initrd between puppy, tinycore and weedog.

I'll add the item #4 to my todo list (i.e. "4. a sfs gui module - jwm-rox" ), however, since I'm focusing mostly on upgrades of older puppies at this point item #4 isn't yet high on my priority list but is something that I want to work towards.

[wanderer]

just a note to remind that there is no need to mount sfs files with a union filesystem
they can be loop mounted and symlinked instead like tinycore

this eliminates the problem with union filesystem overhead with multiple layers

so a distro that uses a union filesystem
can be made to use symlinks with just a minor change in the loading code

you can even use a union filesystem and a symlink system together

so you can have both worlds if you want

this is very important with a modular system

wanderer

[s243a]

hi s243a wiak rockedge and everyone

is it possible to write a small script
like woof-next

that builds 4 components
from puppy woof-ce
and assembles them into an iso

1. a bootloader module - syslinux

2. a kernel module

3. a core module - busybox

4. a sfs gui module - jwm-rox

these 4 modules could be built and kept in a repository
so they do not have to be rebuilt each time

...

....

I'll add the item #4 to my todo list (i.e. "4. a sfs gui module - jwm-rox" ), however, since I'm focusing mostly on upgrades of older puppies at this point item #4 isn't yet high on my priority list but is something that I want to work towards.

While #4 isn't high on my priority list, I'm indirectly working on it to achieve other ends. For instance, I modfied the remove_builtin, so that it can work on an extracted sfs file, rather than the currently running puppy. It can can also process a list of files to remove, rather then listing each file as seperate arguments. This latter feature is useful if you want to add comments. Here is the preliminary script (, which seems to work):

The modifications were in part based on the process_pkglist() from woof-next. The modified function for removing packages is called process_remove_builtin_list() and is found on line#34 of my above script.

My test case is to remove core libs and cli tools from upupGG+D (grovyPup). In my case the filespec to remove is "removesfs" the package to remove is on the left of the line then on the right of the line I have a comment with the related line from /bar/packages/DISTRO_PKGS_SPECS. Here is the complete removal spec:

Not to completely @wanderer 's objective #4 here you'll likely want to remove other packages and also will likely want to remove the rootfs files from woof-CE.

Note: To use my remove_builtin.sh script you'll have to either export the variable CHROOT_DIR or modify line #12"

Code: Select all

CHROOT_DIR="${CHROOT_DIR:-/mnt/+aufs+devsave+test_save+ext2/Kali/kalipup_CLI.tar.gz.extracted/kalipup/puppy_upupgg+d_20.10.sfs.extracted}"

I also recommend testing this on a system that isn't too critical for your daily use given that we are removing files.

Here is an example of how one might run my script:

Code: Select all

bash -x ./remove_builtin.sh --pkg-list uninstallsfs 2>&1 | tee remove_builtin.log

[wanderer]

hi s243a

thanks very much for looking into this
like i said i think it will be very useful for the puppy community
i am following along as best i can

dont work to hard though
you guys have enough to do
as long as this concept is being explored
it should slowly come to fruition

wanderer