Kernel sources - built-in or not?

[dimkr]

I'm thinking, maybe it's best to add the kernel sources SFS to the ext4 images. This will make life easier for those who install proprietary drivers (for example, broadcom-sta-dkms will just work).

However, it's about 150 MB, and that's going to slow down the download for everyone, including users that don't build any out of tree drivers.

Despite the disadvantages, I lean towards adding the kernel sources SFS to the images, in the name of a better out of the box experience. devx and nlsx make Vanilla Dpup useful for development and users who prefer a language other than English, and they're preinstalled for the same reason.

(https://github.com/puppylinux-woof-CE/woof-CE/pull/2867 will fix the issues with wl.ko that breaks across Vanilla Dpup updates, but it's still nice to have the latest kernel sources SFS, instead of having to download it manually.)

What do you think? @retiredt00 @pemasu

[rockedge]

seems like the advantages of including the kernel sources SFS outweighs the added 150 M size. Add the SFS in to the image and let some usage hours sink in to see how it goes.

[dimkr]

I implemented this already, and I want to give people a chance to provide some feedback (the plan is to cherry-pick into 9.1.10 and 9.0.31):

https://github.com/puppylinux-woof-CE/w ... 67296cR387

https://github.com/puppylinux-woof-CE/w ... 1207-R1208

[mikewalsh]

I tend to agree about the inclusion of this stuff. Let's face it, it doesn't make the OS so much bigger, and the days of Puppy having to be as tiny as possible due to lack of resources is largely past. Look how many of us are now running dual- or quad-core CPUs, with 8/16/even 32 GB RAM.

And at least 2 of our number are now kitted-out with 64 GB..!

If adding this permanently into the mix makes DPup a bit bigger, I think that's a small price to pay for the added "ease-of-use" that comes with it. Go for it.

Mike.

[retiredt00]

Since somehow I think I contributed in this let me add my 2c
I think most of the people having a voice here are (aspiring) hackers and tinkerers so is likely to be biased for the inclusion.
However, an 800MB image without Office, Video, Music and all the other goodies puppy usually comes with, is hardly in the spirit and more important the reputation of puppy.

Kernel source suits me fine but I would think that with the inclusion of all development tools and libs as well as the kernel source, you may want to clearly label the IMGs as “-developer’s edition” or something.
Maybe Devedolers_Dpup instead of Vanilla_Dpup is appropriate for these builds.

[dimkr]

Kernel source suits me fine but I would think that with the inclusion of all development tools and libs as well as the kernel source, you may want to clearly label the IMGs as “-developer’s edition” or something.

The distinction between "development" and "non-development" stuff is super problematic.

To install many non-free drivers, you must have a compiler present, and a compiler needs headers and many libraries that have no use elsewhere. This is a common use case, although C programming is probably less common.

Also, in many Puppy releases, some applications don't work unless you have the devx loaded: for example, some calculators depend on GMP (https://gmplib.org/), a library used by GCC for calculations with big numbers. Calculators have nothing to do with development, but even these simple applications get broken because of this artificial dichotomy. This is why Vanilla Dpup has the devx built-in: this translates into a 'just works' user experience, in many cases, and I think the same logic applies to the kernel sources SFS, the second ingredient needed for easy installation of non-free drivers.

[pemasu]

First. I agree with mavrothal, that disabling subversion is not a good idea.
Second. If it is possible to include kernel source sfs subversion check to the updater, it would be enough for most users. Like a question during the update: kernel is now version 6.1.99, do you want to download kernel source sfs 6.1.99.
Third. I believe there are lot of users who does not need kernel source sfs, or does not know how to compile kernel module, or even does not know why their wifi or hardware does not work oob.
And last. For me, adding the kernel source to the build is extra bonus, but really, I would just like to know if the subversion of the kernel has changed, because then I would know what is possible reason that previously compiled out of the tree kernel module did not work...if it does not work. I just would like to have the right kernel source at hand when that happens.
This is new situation, because the kernel compile is automated and reflects the newest subversion.

[mikewalsh]

@retiredt00 :-

One thing needs to be borne in mind here. Yes, I'm definitely a 'tinkerer'; NOT a 'hacker' though, and almost never compile anything. However, many things required for a smooth, trouble-free 'user' experience are also included in these other, 'extra' packages, so it's daft NOT to include them.

Please remember something else, too. The computing 'scene' has changed out of all recognition to the days when Puppy was first launched. It is totally unrealistic to expect Puppy to be built in the same minute, minimalist, ultra-small fashion as it was almost 17 years ago.

Very, very few people are still trying to run the kind of hardware that was 'current' all those years back. Most of it is almost unusable, given how weak hardware was in those days, but one thing never changes, either now or then; as soon as hardware manufacturers come up with improvements to make the user experience better, most software developers almost immediately turn round and re-code their packages to mandatorily require this extra hardware in order to use their software. It's a game of cat-and-mouse, and software developers/packagers all seem to be under the impression that those extra resources have been developed exclusively for their benefit.....thereby enabling coding to become more & more 'lazy'.

Much of Puppy's code has had to grow over the years to accommodate the extra demands being placed upon it by almost any software you care to name.

Mike.

[retiredt00]

@dimkr
It would appear that you are trying to fix small things with a big hammer
If a puppy sfs includes a program that needs a library in devx that is a building issue. This library should be included in the puppy sfs, not the devx.
Regarding the calculator paradigm, if I remember correctly all puppy calculators work without devx. When some user wanted to install a GIMP-dependent calculator with 100MB dependencies then the package manager did the dirty job for the specific user, not the builder, taxing everybody else, preemptively.

@mikewalsh
I was making the point that puppy’s reputation is a small, fast ISO that includes all needed applications and utilities for most uses.
What @dimkr is proposing is a sizeable IMG with a lot of development tools, libraries and sources, minimal apps and a browser configured to do most things. A “development chromebook”-like thing.
My point has nothing to do with hardware requirements. It is not the devx or kernel source SFSs that will stop old hardware from running a modern puppy. Network/communication/gaming applications will do that.
That is why I suggested a change in the name (for the IMG versions) and not the omission of the content in question.
I would also like to clarify that for me “small footprint” does not mean “for old hardware”. It means “well thought out and carefully designed, lightweight and fast” LinuxOS.
It is the niche that puppy managed to curve for itself through the years and would be useful (if not necessary) to keep this distinction.

BTW would be hard to believe that the 300MB+ in question (devx and kernel source) are needed or are even sufficient, to satisfy the dependencies of any application out there. Besides if someone wants to add Gnome or KDE on it, now it comes with apt!

@both
Apologies for the long and combo message

[dimkr]

I would also like to clarify that for me “small footprint” does not mean “for old hardware”. It means “well thought out and carefully designed, lightweight and fast” LinuxOS.

Sometimes, small is slow and big is fast. The best example I have is VAAPI drivers - they're big, but without them, the browser decodes videos on the CPU and that's slow and power hungry. Another example is zstd: the compression ratio is slightly worse than xz (making Vanilla Dpup bigger), but decompression speeds are so much better: even faster than gzip's.

I'm trying to emphasize usability and out-of-the-box experience. Users who want something smaller can always remove parts of Vanilla Dpup - it's modular. For example, you can delete docx, nlsx and adrv to remove most applications and get a "barebones" system. You can also delete devx and the kernel sources SFS, but some users will need them to install out-of-tree drivers.

[ozsouth]

Rolling devx, sources (& headers?) into a dpup, the result will be a kind of 'debian lite'.
I've often wished debian would make a lite version, so this could be a real winner.

[retiredt00]

I'm trying to emphasize usability and out-of-the-box experience.

It would appear that we have different ideas of what out-of-the-box experience means
In my mind it means I install effortlessly and I need nothing else. The linux (desktop) world appreciation for MX Linux or Mint (and puppy of yesteryears) would suggest that this might be a more general sentiment.
You appear to think that what out-of-the-box experience means a minimal system but with a solid foundation for easy user-initiated configuration.
I believe that this is very valid option too.
I just suggest that it should be labeled accordingly to direct expectations.
“Official Puppy” is coming in many flavors, so a name-distinction based on the distinct logic/approach rather than the package compatibility is perfectly fine I would think.
That’s why I suggested developers_Dpup instead of vanilla, but other names may be better (Foundation/Bedrock/Infra/Basis etc - Bedrock sounds good!)

But to come to the point as far as the kernel source in the IMG is concerned, I'm fine with it.

[OscarTalks]

I do not wish to appear to be complaining about anything, but since debate has been invited I will chip in with a few thoughts, maybe even playing "Devil's Advocate" a little bit.

First of all, I am fine with having kernel sources built in, although I wonder what percentage of all users will actually be using it. Furthermore, anyone who is using it (to compile driver modules for example) is presumably experienced enough to know how to load and unload a separate kernel sources .sfs for the brief moments when it is needed to be in the system.

If the thinking is that size does not matter so much, but functionality (including compiling and development) matters a lot more than size, I would argue that it is much more important to make sure that the built-in devx contains ALL of the DEV packages which correspond to all of the runtime library packages which are included in the main puppy .sfs

General compiling is a more common activity than kernel module compiling. I have built a few packages in VanillaDpup including ffmpeg and I have to spend a lot of time using PPM to find and add the DEV packages of compulsory and optional dependencies. With the optional ones, I would miss out on the functionality of runtime libraries which are actually present in system if I was not aware of the issue at build time.

[dimkr]

In Ubuntu or Debian, only a tiny portion of the kernel sources (just the headers and the build system) plus some "devx" tools (GCC, make, etc') is needed to install out-of-tree drivers.

I want to reduce the kernel SFS size, so it contains just the bare minimum - I hope to start working on this later today or tomorrow.

[dimkr]

I've managed to reduce the kernel sources SFS to 16 MB, and Broadcom WiFi drivers built just fine.

EDIT: not just Broadcom drivers - https://github.com/puppylinux-woof-CE/w ... 1039918024

[dimkr]

9.1.10 is out with the small kernel sources SFS and DKMS support. Only 17 MB bigger than 9.1.9, but now it's possible to apt install non-free drivers and they're automatically rebuilt on every kernel update.

If you're updating from 9.1.9, you might need to:

Code: Select all

rm -rf /mnt/home/upper/save/usr/src
echo 3 > /proc/sys/vm/drop_caches
apt install --reinstall broadcom-sta-dkms                                          (or, the driver you're using)

[pemasu]

Code: Select all

apt install broadcom-sta-dkms 
The following additional packages will be installed:
dctrl-tools distro-info-data dkms lsb-release
.....
Building initial module for 5.10.101
Done.
wl.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.10.101/updates/dkms/
depmod......
DKMS: install completed.

Are those additional packages installations relevant. I thought that dkms was already installed ?
Wl.ko kernel driver module compiled successfully and also works fine.

[dimkr]

Are those additional packages installations relevant. I thought that dkms was already installed ?

Yes, DKMS is not included by default.

Not all users need it, it takes time for it to check if all out-of-tree drivers are up-to-date (on my super slow laptop and a worn drive, it slows down the boot by about 0.4s IIRC), and you'll get it if you install an out-of-tree driver that depends on it.

[retiredt00]

The inclusion of kbuild in9.1.10 is very handy!!!
Indeed with these extra 16MB you can install proprietary/out-of-tree modules by a single click
One more innovation in this puppyland!
(that I should admit that the last few years was consumed in repackaging other distros, sometimes by just running them in a layered file system)

However, in the case of the wl driver is not so simple because you need the (wl-depended) connection to install it.
The good news is that if you have the deb file content you can compile the driver by a simple “make” command in this new enhanced environment. Kudos!

Talking about deb files, you may want to include gdebi and make it handle deb packages instead of petget that ignores dependencies.

Again, thank you for the good work

[dimkr]

However, in the case of the wl driver is not so simple because you need the (wl-depended) connection to install it.

That's true, but it's a one-time thing. The best thing about having built-in kernel sources and DKMS support is the freedom to update the kernel, without having to go through this annoying process manually, every time.

Talking about deb files, you may want to include gdebi and make it handle deb packages instead of petget that ignores dependencies.

On it!

https://github.com/puppylinux-woof-CE/woof-CE/pull/2884
https://github.com/puppylinux-woof-CE/woof-CE/pull/2887

These two PRs will make Vanilla Dpup super friendly to existing Debian or Ubuntu users, who are already familiar with apt.

[rockedge]

Wow I never saw this until now! Looks very promising and will be great tool to have included. It's all about the dependencies being handled well.

Now I would like to try this out on a Fossapup and/or a Bionicpup

[retiredt00]

Since the 9.1.10 downloads do not include the full kernel-kit sfs, can the one from the same date 9.0.X be used?
The 9.0.32 version in this case.
Would be useful if one would like to recompile the kernel for example.
Thanks.

[dimkr]

Since the 9.1.10 downloads do not include the full kernel-kit sfs, can the one from the same date 9.0.X be used?

No, they're incompatible. The 9.0.x one puts files under /lib, instead of /usr/lib.

You can find the whole kernel sources SFS here:

https://github.com/puppylinux-woof-CE/w ... el-kit.yml

https://github.com/puppylinux-woof-CE/w ... #artifacts

(or more specifically, inside https://github.com/puppylinux-woof-CE/w ... /169236646)

[retiredt00]

You can find the whole kernel sources SFS here:

https://github.com/puppylinux-woof-CE/w ... el-kit.yml

https://github.com/puppylinux-woof-CE/w ... #artifacts

(or more specifically, inside https://github.com/puppylinux-woof-CE/w ... /169236646)

I am afraid I can not.
The last link is broken while the other two are too cryptic for me to pinpoint the 5.10.101 kernel source
You might want to consider a more obvious naming or provide a kernel-kit sfs for the 9.1.X builds too.
Thanks again.

[dimkr]

You might want to consider a more obvious naming or provide a kernel-kit sfs for the 9.1.X builds too.

The 9.1.11 release will have the kernel sources SFS, alongside the images.