Corepup - a minimalist modular system

[user1111]

Mine now is equally modular. grub4dos bootloader, minimal base system initramfs + busybox, the rest in sfs's that are selectively symlinked into that. I only insert the sfs's into initrd and then build that initrd into vmlinuz so that I just have a single file (10MB vmlinuz) that when booted has framebuffer/cli/wifi/ssh/vnc functionality.

Fundamentally whilst very modular that could be added to, I have no need to add modules, as the boot into a framebuffer (graphical) cli that includes mc, vnc and ssh ... is enough. With that I can vnc into and share ... whatever. For instance I boot that system on my laptop and can vnc into my desktop (vnc server) that is running Fatdog and that is connected to the TV. Simultaneously I can also vnc into that server from my phone, so all three are seeing/sharing the exact same desktop. If I set the server to boot Windows/whatever in a kvm/qemu then again all of the devices see/share/control the Fatdog and Windows sessions. So its more appropriate to add modules/programs on the server side, not the client.

But yes, if I wanted to add say a diary program to the setup, a sfs for that and mounting/sym-linking it in would have that module 'loaded', in a similar style to tinycore. As could the intird and net.sfs/mc.sfs also be separated out of the vmlinuz and loaded in a similar 'modular' style.

[wanderer]

hi rufwoof

sounds great

do you have an iso for pugdog
can i substitute the extlinux bootloader and put it on a usb

wanderer

[user1111]

I build the kernel using make localyesconfig;make -j4 so its highly specific to my hardware (all firmware/modules are built into the kernel), as such its not portable, I can't even move it to another local machine as it wont boot due to differences in hardware. So sorry no shareable iso. The closest is Fatdog's Bulldog to get you to cli mode, but that involves all of the broader modules/firmware set (sfs), that is around 80MB in size alone IIRC. Fatdog includes kernel boot parameter choices where you can drop out to cli (Bulldog) at a range of points, early, late ...etc. so for sym linking sfs's you could select a appropriate point and add the code to simply mount each sfs and cp -ars the contents to /

The clever element in tinycore IMO is how its so portable, boots across a wide range of hardware with seemingly relatively small levels of firmware/modules. IIRC that's because it uses xvesa as the common denominator. A core of Xvesa + sym linking of sfs's ... was (is) a great/clever idea. What I'm doing is reducing that further down to just a framebuffer cli boot using busybox as the 'userland' and init ... on top of which sym linking sfs's can be added to build up the 'system' functionality.

[user1111]

Just adding a kvm/qemu sfs to my Pugdog sfs's collection. With qemu you can either boot directly from within X so that the booted system appears as another window, or you can boot it to serve vnc, so you can vnc into that, in which case the server doesn't even have to have X running (headless/cli). qemu supports booting iso's directly or image files, a partition, or even just giving it a vmlinuz/initrd.

So a small 'core' system, could serve/open-up having multiple systems all running concurrently. In this image I've actually booted Fatdog, and then qemu'd another Fatdog session, as well as Tinycore and a OpenBSD sessions.

s.jpg (121.02 KiB) Viewed 1592 times

using my laptop (2 core 4GB ram).

I could have added my Fatdog server to that, i.e. vnc'd into it, which in turn could have a qemu of say Windows and/or DOS sessions running.

Multi-boot (concurrently)

cwm is a nice window manager in that respect ... or a large window/resolution that you can pan around. cwm is awkward at first (learning curve/not-intuitive), but has borderless windows and where everything can be controlled using the keyboard, and is simple enough that just a few controls have to be learnt in order to use it reasonably well.

ssh/sshfs is great for the sharing side of things, to for instance make it easy to port a file from say the tinycore session to the openbsd session. Rox filer is nice in that respect for its ease of drag/drop between sshfs'd folders. Being able to full-screen/zoom individual windows (systems) is also pretty much a necessity.

On the practical front however ... how many different choices do you actually need before its just duplicating functionality. Most of what I use X for is to simply run a gui browser for when I need more than the likes of text searches (googler/lynx) and/or reading multi-media type content. Much of whatever else I do can more often be served purely textually i.e. I ssh into hashbang from where within tmux I can email (mutt), irc (irssi and weechat sessions), diary (calcurse) ...etc. and drop into/leave that running 24/7 (so drop back into where it was last left, maybe using a different device such as my phone).

I allocated around 1.5GB to 2GB of ram to each of the qemu sessions, where each has the cpu to be the same as the host. Another nice thing with qemu is you can set specific cpu's to 'use', and even set more cores than is actually available on the host. Posting this from my standard Fatdog system (host) using chrome and ram is pretty much totally used up and around 1.2GB of swap is being used ... so when I do switch to another system there is some delay whilst things are swapped around. But the lag is relatively small, a few seconds type lag.

[wanderer]

hi rufwoof

your system certainly meets the criteria for a minimalist modular system
(and then some)
and is of great interest
please continue to post and explain how it works and is built
it is a little complex for me
but i am slowly getting it

some questions

no unionfs system is used only symlinks --- right ?
everything is symlinked into the initrd --- right ?

how did you build your initrd
where did you get the components
i understand that you build your kernel from scratch - right ?
is the file layout standard linux --- / /bin /sbin etc
as i understand it you are running most things in qemu -- right ?

anyway when you get the time and inclination please post
in the meantime i will review what you have posted

thanks

wanderer

[user1111]

no unionfs system is used only symlinks --- right ?

No overlayfs/aufs/unionfs ... just bulk standard kernel

everything is symlinked into the initrd --- right ?

Now yes, thanks to you, before I was extracting tar files into / (within initrd).

how did you build your initrd

It's very minimal, just enough to get to busybox cli. I'll have to strip out private bits (ssh keys etc) and upload my latest version

where did you get the components

Compiled from source, or as I use Fatdog as the built-tool and the same kernel version I can pinch bins and libs from Fatdog and drop them in. I do that for mc. Just a handful of programs, busybox, fnvnc, openssl (required by ..) openssh, wireless tools (iwconfig/wifi stuff) ... and otherwise just files/folders/scripts. All compiled statically, preferably musl and I upx the binaries (so no libs involve, like the kernel has no external modules/firmware, the required bits are all built into the kernel).

i understand that you build your kernel from scratch - right ?

Yes. Boot fatdog, load the devx sfs, grab the latest kernel source from kernel.org (I'm tracking the 5.4 kernel), extract that source code, cd into it and
make localyesconfig
make -j4
(I have 4 cores on my build system, if I compiled on my 2 core laptop I'd use make -j2 instead).
That creates a bzImage ... which can just be renamed to vmlinuz and you have your kernel. At least that's how I started off, and later I revisited the configuration (make menuconfig ... and change whatever settings and save, ... and then run make -j4 again to build that) and changed things so that it builds with a framebuffer ...etc. under something like devices, graphics ... in make menuconfig (forget offhand exactly where).

is the file layout standard linux --- / /bin /sbin etc

Yes

as i understand it you are running most things in qemu -- right ?

I compiled and added in fbvnc into my build (a framebuffer based version of a vnc client) and boot with a vga=ask kernel boot parameter (actually I've coded the choice I use) and select a VESA choice. Once that's booted I wifi net connect and then can vnc into my desktop system (that is running fatdog and has x11vnc (server) running) using fbvnc ... and up pops the gui desktop.

So the kernel (vmlinuz) boots and the kernel hands over to my /sbin/system-init which looks like

Code: Select all

#!/bin/ash

[ $$ -eq 1 ] || exit 1 ### bail out unless we're PID 1

KERNEL_POLL_MSECS=${KERNEL_POLL_MSECS:-5000}	# default kernel polling value if unset

# mount core filesystems
/bin/mount -t proc proc /proc # mount /proc first so /proc/self/exe works from now
mount -t sysfs sysfs /sys
if ! mount -t devtmpfs devtmpfs /dev 2> /null; then		# /dev/null may not exist yet
	# if no devtmpfs, use tmpfs and use mdev instead
	mount -t tmpfs tmpfs /dev -o mode=755 # mode 755 is what devtmpfs uses
	mdev -s
fi

[ -f /null ] && rm /null	# Clear up the null file created above
[ ! -e /dev/shm ] && mkdir -p /dev/shm
[ ! -e /dev/pts ] && mkdir -p /dev/pts
[ "$TZ" ] && hwclock -t # set kernel timezone

# debugging and error logging comment out next two lines for greater info during bootup
! grep -q showerr /proc/cmdline && exec 2> /dev/initrd.err
grep -q debuginitrd /proc/cmdline && set -x

# enable polling
if ! grep -q block.events_dfl_poll_msecs /proc/cmdline; then # if not explicitly set
	echo $KERNEL_POLL_MSECS > /sys/module/block/parameters/events_dfl_poll_msecs
fi

loadkmap < /lib/boot/keymaps/uk # keymap hard coded to use uk (GB)

ifup lo  # required by ssh

# Hand PID 1 over to busybox init (that references /etc/inittab)
exec /sbin/init 

where that last line hands over to /sbin/init ... that is a sym link into busybox init.

Busybox init then does its stuff and uses either its own integral /etc/inittab, or if one exists in /etc/inittab that is used instead. The busybox integral version looks like
https://git.busybox.net/busybox/tree/examples/inittab

No rc.d or anything like that coded/used, just a (graphical/framebuffer) cli within initrd that is then used to read/copy whatever the vnc server serves (gui desktop) to the framebuffer (display).

More recently, rather than using tarballs, I've added /opt/sfs folder into which I've copied mc.sfs and net.sfs files rather than having to mount and load them from externally. Once mounted (mkdir /mnt/mc;mount /opt/sfs/mc.sfs /mnt/mc), I sym link them into / (cd /; cp -ars /mnt/mc/* .)

Fundamentally the initrd is just busybox and all of the userland commands that contains are sym linked to that. Once booted then sfs's are mounted (mc.sfs and net.sfs in my case), and sym linked into / ... which then has a pretty functional cli based system, with wifi networking, openssh (scp, sftp, ssh, sshfs) and mc (file manager and ncurses style (easy to use) text editor). With that available you can pretty much go in any direction you like, but with access to a gui desktop via vnc and all that that offers, I have no need to add to that 'base system' (I have however recently added qemu.sfs so it can run a qemu session directly, such as booting any iso's, other vmlinuz/initrd's, image files, partition ...etc.), so it could boot multiple instances of other systems (Puppy, tinycore) and vnc into those (multiple desktops/concurrent-systems). But that's not really a necessity, as I could just boot one of the iso's directly and install/use qemu within that. Accordingly I've not copied qemu.sfs into /opt/sfs .. but instead leave it 'outside' so available to be mounted/sym-linked should I so choose.

Very much similar to tinycore in many respects, except their kernel is more portable, whilst mine is highly machine specific. Far easier to just use tinycore as the 'base system' rather than going through all of the above build process. When you're booting a system to just vnc into a server, a client system that is just handing raw screen reading from the server to throw to the display, that can be pretty much anything, security isn't really a issue (mem copy what is thrown across tcp onto the display). If the server is a iso read only booted system as mine is, then that's moderately secure, and also means that web sites see that system as the source system, not the actual source system. So any files on the tinycore system are secure, unreachable unless that system is otherwise opened up. You get to choose which files you might transfer to/from the server such as via scp, or using sshfs/mc (mounting and using a file manager).

[fredx181]

Recompiling the kernel to add in squashfs filesystem support and creating sfs's of the net and mc tarball files contents (net.sfs and mc.sfs), I can now boot and mount those
mkdir /mnt/net
mkdir /mnt/mc
mount net.sfs /mnt/net
mount mc.sfs /mnt/mc

... and then once booted I can 'load' those using symlink based copying ...

cd /
cp -ars /mnt/net/* .
cp -ars /mnt/mc/* .

Thanks for the concept of loading SFS using symlinks, I've been experimenting with it:
viewtopic.php?p=11354#p11354
That loadmodule script does a lot more (e.g. update-icon-cache, update-desktop-database, error checking etc...), but basically it's this way;

To load SFS:

Code: Select all

# load SFS
SFS=/path/to/mySFS.sfs     # change to the real path to SFS file
#SFS=/mnt/sdb2/gimp.sfs  # for example
BASESFS=$(basename $SFS)

mkdir /mnt/$BASESFS
mount -o loop $SFS /mnt/$BASESFS
FILES=$(find /mnt/$BASESFS | sed "s|/mnt/$BASESFS||")
while read line; do
if [ ! -e $line ]; then

# create symlinks in the system, do NOT overwrite existing symlinks
ln -s /mnt/$BASESFS/$line $line 2> /dev/null   
fi
done <<< "$FILES"

To unload SFS: EDIT 2020-12-02, FIXED slow unloading

Code: Select all

# unload SFS, may take some time
SFS=/path/to/mySFS.sfs     # change to the real path to SFS file
#SFS=/mnt/sdb2/gimp.sfs  # for example
BASESFS=$(basename $SFS)

FILES=$(find /mnt/$BASESFS | sed "s|/mnt/$BASESFS||")

# unmount SFS, check for broken symlinks and remove them ...
umount /mnt/$BASESFS && rmdir /mnt/$BASESFS  
while read line; do
if [ ! -e $line ]; then
# 2020-12-02 speedup unloading, check first if symlink exists before removing
# if not doing that, rm outputs lots of error messages, which makes it very slow
[ -L $line ] && rm $line
fi
done <<< "$FILES"

The "cp -ars" works, but doesn't symlink folders, the above does and makes it easier (I think) to do a clean unload.
But I guess it can be improved and/or simplified.

Fred

[wanderer]

hi all

take a look at dcore stretch
if some of the gurus would focus their genius there
great things would be accomplished

symlinks to a ramdrive
makes bundles of debian apps and dependencies

wanderer

[fredx181]

Fixed the slow unloading of SFS, just as fast as loading now, edited unload code here: viewtopic.php?p=11355#p11355
And re-attached loadmodule script here: viewtopic.php?p=11354#p11354

@wanderer perhaps a bit off-topic, sorry.

Fred

[wanderer]

thank you very much fredx

very much on topic

also using your portable firefox in dcore (soon to be corepup-9)

and thanks again for that

wanderer

[fredx181]

thank you very much fredx

very much on topic

also using your portable firefox in dcore (soon to be corepup-9)

and thanks again for that

wanderer

Well, thank you wanderer !
The posts above about loading SFS using symlinks are perhaps a bit off-topic, as Tinycore does it already that way.
Btw, you have any idea how they do it ? (the codes above may work OK, but maybe the way Tinycore does it is smarter)

Fred

[wanderer]

hi fredx

tinycore and dcore load sfs files
called respectively tcz and sce files
by loop mounting them
and then using cp -ais to symlink the files

wanderer

[wanderer]

hi fredx

the relevant file in dcore to load the sce (sfs) files
is in etc/initd/sce-load

wanderer

[wiak]

Hopefully, tinycore symlink method (or per Fred's implementation) will also provide optimum solution to using sfs load/unload when employing underlying overlayfs layers rather than aufs. Seems likely from above reports by Fred. Very useful, as are some other of the mechanisms rufwoof is employing to produce his own ultra-small and secure system. Interesting times indeed.

wiak

[wanderer]

hi wiak rufwoof fredx and everyone

makes me happy to see some are using
symlinks to ramdrives in their work
a very useful system

as i have posted before the next corepup-9
will be based on dcore-stretch
which is in the corepup repository on the first post

this system has many interesting features
it uses extensions symlinked to a ramdrive
and is able to make extension bundles from debian stretch debs
which contain both the app and all its dependencies

it is very modular and can be efficiently minimized

i would like to see puppy do this
as i think it would make things much easier to learn use and develop

i assume that puppy still uses a base init file
and its components are stored as pet files
so it should be reasonably straightforward
to make a small puppy based version

however my level of expertise is very limited
so i will concentrate on more modest goals

1. getting puppy pets and sfs files to load
2. looking at how the debs are downloaded and packaged as extensions
3. further exploring the symlink system

etc

i encourage others to take a look at this system
as i feel it would be a very interesting and enjoyable experience

see you soon

wanderer

[williwaw]

this system has many interesting features
it uses extensions symlinked to a ramdrive
and is able to make extension bundles from debian stretch debs
which contain both the app and all its dependencies

Extensions contain the application and all the needed dependencies? similar to a flatpac or an appimage?

but use links to load and unload? Could you describe in more detail how the links/unlinks work to loop mount?

[wanderer]

hi williwaw

yes the system makes app bundles of debs
the app and all its dependencies
like flatpaks but only for dcore-stretch
it does not try to cover other distros

they symlink system is very simple
an sce (compressed file)
is mounted with a loop device (mount -o loop)
and then the copy command (cp -ais) is used to symlink it into the filesystem
this puts symlinks for everything in the appropriate directories
so that commands can find them

wanderer

[williwaw]

are the extensions recompressed with configuration changes to replace the original? or are they static and changes handled in a different manner?

[s243a]

I thought of a different use-case for the symlink package loading. One thing that I might do use use a savefile for a sandbox. To keep these savefiles small I could symlink to content in an existing savefile.

[wanderer]

hi williwaw

the extensions are just mounted
they remain compressed as files
and are just read
changes and configurations are written in home

things can be backed up as a compressed file
but i never use that
as writing to home is more flexible

hi s243a

yes the symlink system is almost infinitely flexible
you can symlink just about anything to anything

i use it to symlink an uncompressed app directory stored in home
that way i do not have to make a compressed file extension
and can modify the app without decompressing and recompressing it

much like fredx portable firefox

wanderer

[user1111]

Brainstorming/rambling ...

If you had a minimal kernel/system, just cli and that supported aufs layering, then you could kvm/qemu that 'guest' as part of a overlayfs host system ... and share/copy/link/whatever part/all of that guest systems directory/folder tree into the host. Or maybe even chroot into that guest systems folder tree. Whilst the host system (bulk standard overlayfs incorporated kernel) might be updated/changed (such as later kernel dot releases that incorporated security fixes) the guest systems kernel/structure could remain relatively static. i.e. a structure whereby the guest system manages any aufs layering of sfs's with those changes being reflected into the host.

Loads of caveats, as I said just rambling.

[williwaw]

i use it to symlink an uncompressed app directory stored in home
that way i do not have to make a compressed file extension
and can modify the app without decompressing and recompressing it

Hi wanderer

what benefit does this have over just running the app normally?

[wanderer]

hi williwaw

if the app is uncompressed in its own folder
then you can modify it without having to compress and uncompress it
for example if you need to add dependencies or configuration files
you can then compress it if you want to use it as an normal sce

this is also good for examining and modifying an existing sce or tcz
that you have uncompressed and want to work on

it also does not have to go through the normal loading process
so you can directly symlink it with cp -ais
and unlink and relink it by changing the name of the folder

wanderer

[wanderer]

hi rufwoof

very complex stuff

why use a union file system at all

cant you do it with symlinks

wanderer

[williwaw]

it also does not have to go through the normal loading process
so you can directly symlink it with cp -ais

I made a directory with cp -ais that contains sub directories with links. In this case it is for a portable browser, and I can run this browser by navigating to and clicking on the link to the executable.

How is this better than doing the same to the actual executable in the original directory, or simply making a link anywhere I choose that points to the executable in the original?

[wanderer]

hi williwaw

keeping everything that pertains to an app in one uncompressed folder in home
is useful for running multiple instances of and app and testing and modifying things

when this is not desired one can always make it into an sce (compressed)
and run it normally

fredx firefox is a perfect example

just an alternative way of doing things when these abilities are needed

wanderer

[williwaw]

hi williwaw

keeping everything that pertains to an app in one uncompressed folder in home
is useful for running multiple instances of and app and testing and modifying things

when this is not desired one can always make it into an sce (compressed)
and run it normally

fredx firefox is a perfect example

just an alternative way of doing things when these abilities are needed

wanderer

the merits of compressed vs uncompressed I understand.

when and why do you use cp -ais ?
thanks

[wanderer]

hi williwaw

i use cp -ais to symlink the app into the root filesystem
so that i know that all the links are in the right directories
this way the app does not have to be modified to only use the folder its contained in

wanderer

[fredx181]

hi williwaw

i use cp -ais to symlink the app into the root filesystem
so that i know that all the links are in the right directories
this way the app does not have to be modified to only use the folder its contained in

wanderer

Hi wanderer, are you sure its "cp -ais" with the "i" ? From cp --help :

Code: Select all

-i, --interactive            prompt before overwrite

So then with every file to copy that already exist in the system, it asks if you want to overwrite.
I think it should be "cp -ars" or "cp -arsn" (to not overwrite existing), but correct me if I'm wrong.

Fred

[wanderer]

hi fredx

i think the i in cp -ais
causes it to not overwrite existing links
it never prompts to my knowledge

it is what tinycore and dcore use
see the file
/etc/initd/sce-load

i have never used any of the other command modifiers

but this is something to look into

wanderer