Using debian kernel configuration to build Puppy kernel [CLOSED]

[jamesbond]

This question is more for @dimkr, but others with interest, please feel free to join.

Dima, you advocate for using the Debian kernel configuration to build the Puppy kernels, modified for Puppy's specific configuration (e.g. like aufs), because in your opinion Debian kernels are built with larger support of hardware out there. I think there is no argument about this. The only downside that you noted is that the kernel would be bigger.

I've just tried to build a new kernel this way: Linux 6.1.90 (patched with aufs), with bookworm configuration, and compare it with the kernel build of the same kernel sources using Fatdog configuration. The resulting debian-configured kernel is a bit bigger (9MB) vs Fatdog-configured kernel (8MB), but this is acceptable.

What surprises me, however, is the size of the kernel modules. The uncompressed kernel modules (the one you get then you run "make modules_install") is 4.6GB, compared to 265MB for the Fatdog-configured build.

I notice that the debian folks don't configure every kernel options. They only define the "important" config items, and for the rest, they leave it to the default as set by the kernel developers. This probably ends up enabling all sort of stuff, e.g. even compiling drivers for embedded devices which are only available for a certain SoC platforms; drivers which aren't relevant for x86 platform.

So my question is:
1. Are you seeing this too?
2. Do you consider it as a problem, or are you happy with that extra 4.6GB uncompressed modules?

In addition, you said that for your vanilla dpup, you used a tweaked version of the debian kernel config to build its kernel. With these tweaks:

3. Are you tweaking it to reduce the size, or are you doing it just to support the Puppy-like features in dpup?
4. I would like to see the dpup kernel config if it is available somewhere to compare notes.

Thanks.

[dimkr]

The dpup kernel is built by applying a small eelta (https://github.com/puppylinux-woof-CE/w ... s/bookworm), with the Debian configuration as the base.

It's a regular kernel-kit run but it begins with download of the Debian kernel source rather than a kernel.org tarball, see https://github.com/puppylinux-woof-CE/w ... kernel.yml.

Regarding size, the size increase is very likely to come from debugging symbols. Without them, the size difference compared to a 'Puppy kernel' is small and even smaller after compression.

zcat /proc/config.gz will give you the whole configuration file used to build the dpup kernel, and you can use ncdu to see where the extra GBs come from.

Firmware is binary (less compressible) and dpup uses Debian's firmware packages without any trimming, so that's where the biggest size increase comes from.

[jamesbond]

Thanks, I will check it out.

I don't compare the firmware size yet, but I will. Which firmware package do you use? firmware-free or do you use firmware-nonfree as well?

[dimkr]

Which firmware package do you use?

https://github.com/puppylinux-woof-CE/w ... kworm#L280

[jamesbond]

The dpup kernel is built by applying a small eelta (https://github.com/puppylinux-woof-CE/w ... s/bookworm), with the Debian configuration as the base.

Ok, I see that the delta is just to make certain drivers built-in (usb, sata, mmc, squashfs).
But where does the Debian configuration comes from?

It's a regular kernel-kit run but it begins with download of the Debian kernel source rather than a kernel.org tarball, see https://github.com/puppylinux-woof-CE/w ... kernel.yml.

Which one? I don't see the source listed in this link. I'm using this one: https://packages.debian.org/bookworm/linux-source-6.1, but I see that the "orig" tarball is a standard Linux kernel. The patches and the config fragments are in the the other "debian" tarball.

Are you using this one too, or are you using something else?

Regarding size, the size increase is very likely to come from debugging symbols. Without them, the size difference compared to a 'Puppy kernel' is small and even smaller after compression.

Yeah, true enough, I've just checked, Fatdog-config actually produces more modules than the debian-config one, so I need to find and disable that debug symbols. I wonder though, how did the debug symbols get enabled. Perhaps because I was just cat-ting the config fragment instead of using merge_kconfig.sh. I'll try that next.

EDIT: Actually, using merge_kconfig.sh or just cat-ting the fragments together and running make oldconfig results in identical configuration. So that's not it.

Which firmware package do you use?

https://github.com/puppylinux-woof-CE/w ... kworm#L280

Okay, so that's quite a few bunches there. Thanks. I use linux-firmware-git, but I think there are some non-free drivers which aren't there, so it's worth checking it out.

cheers!

[dimkr]

Ok, I see that the delta is just to make certain drivers built-in (usb, sata, mmc, squashfs).

Yep, it's bus/device/partition/file system/NLS options to allow it to boot plus some performance, RAM consumption and size related settings (best examples are squashfs decompressors + no huge pages by default + entire test menu disabled).

But where does the Debian configuration comes from?

/usr/src/linux-config-*

Which one? I don't see the source listed in this link. I'm using this one: https://packages.debian.org/bookworm/linux-source-6.1, but I see that the "orig" tarball is a standard Linux kernel. The patches and the config fragments are in the the other "debian" tarball.

The .orig tarball and the
patches that accompany a .deb package are used to build it.

The kernel source package is a weird .deb because it's a package that contains source code. This is confusing.

The .deb installs a source tarball that's built from .orig tarball plus the patches, and that's what the build pipeline rebuilds after applying the Puppy tweaks.

[jamesbond]

Regarding size, the size increase is very likely to come from debugging symbols. Without them, the size difference compared to a 'Puppy kernel' is small and even smaller after compression.

Yes, this is it. By default apparently debian enables DEBUG_INFO. Once I set CONFIG_DEBUG_INFO_NONE the size shrinks from 4.6GB to about 331MB.

Yep, it's bus/device/partition/file system/NLS options to allow it to boot plus some performance, RAM consumption and size related settings (best examples are squashfs decompressors + no huge pages by default + entire test menu disabled).

Good stuff. I saw all that but I missed the disable the runtime testing part.

/usr/src/linux-config-*

But there does this comes from? From https://packages.debian.org/bookworm/linux-source-6.1, there are 3 things I can download:
- the "orig" tarball (contains vanilla unpatched kernel)
- the "debian" tarball (contains patches + config fragments)
- the ".deb" package, which, in the inside, only contains linux-source-6.1.tar.xz, which I believe contains the patched kernel ("orig" + patches from "debian" applied).

However I don't see the linux-config-* on either of them.

The .orig tarball and the
patches that accompany a .deb package are used to build it.

The kernel source package is a weird .deb because it's a package that contains source code. This is confusing.

The .deb installs a source tarball that's built from .orig tarball plus the patches, and that's what the build pipeline rebuilds after applying the Puppy tweaks.

Okay thanks, but I still don't get it. Is this the same .deb packages that I referred (from https://packages.debian.org/bookworm/linux-source-6.1 ), or is this something else?

Thanks.

[dimkr]

But there does this comes from? From https://packages.debian.org/bookworm/linux-source-6.1, there are 3 things I can download:

From linux-config-6.1.

https://packages.debian.org/bookworm/am ... 1/filelist

Okay thanks, but I still don't get it. Is this the same .deb packages that I referred (from https://packages.debian.org/bookworm/linux-source-6.1 ), or is this something else?

Nope, the kernel source is indeed taken from the tarball inside the linux-source-6.1 .deb package.

[jamesbond]

Great, thanks
That clarifies everything.

-------

To summarise (for everyone else): the manual process to build Debian-based Puppy kernel ("debian-based" means "debian-patched" and "using debian kernel config as the base").

1. Linux kernel source from here: https://packages.debian.org/bookworm/linux-source-6.1
Get the .deb, not the sources "orig" or "debian"; inside the .deb there is linux-source-6.1.tar.xz, which contains already-patched Linux source.

2. Linux kernel config from here: https://packages.debian.org/bookworm/am ... config-6.1
Get the .deb, not the sources "orig" or "debian"; inside the .deb there are a few config files for desktop, cloud and RT kernels.
Let's assume we use one those: config.amd64_none_amd64 (after we un-xz it).

3. Delta kernel config to make Puppy boots: https://github.com/puppylinux-woof-CE/w ... s/bookworm
(Let's call this file as config.woof.bookworm)

4. To prepare the configuration, merge the config from (2) and (3)
E.g. in the Linux kernel source directory, do this:
./scripts/kconfig/merge_config.sh config.amd64_none_amd64 config.woof.bookworm

5. Then run make oldconfig and proceed to build as usual.

[dimkr]

5. Then run make oldconfig and proceed to build as usual.

One thing I would add: after the build, verify that the diff between the Debian configuration and the configuration file used to build the kernel contains only the Puppy delta plus dependencies (i.e. if a driver changed to =y depends on something, it should become =y as well):

Code: Select all

./scripts/diffconfig -m config.amd64_none_amd64 .config > config-diff
diff -up config.woof.bookworm config-diff

If the build environment is not Debian 12 (things like GCC or binutils are older/newer) or something is missing, some kernel features are auto-disabled and this can have a cascading effect.

[jamesbond]

Code: Select all

./scripts/diffconfig -m config.amd64_none_amd64 .config > config-diff
diff -up config.woof.bookworm config-diff

Ah, so that's how you genereated config.woof.bookworm in the first place. I was wondering how you got the diff. I actually did that manually (did a manual diff -u and then used geany to edit and collect the changes ... hardwork!) to collect my own changes. The diffconfig definitely helps.

If the build environment is not Debian 12 (things like GCC or binutils are older/newer) or something is missing, some kernel features are auto-disabled and this can have a cascading effect.

Yes, that makes sense, thanks!

[dimkr]

I actually did that manually

I also cleaned up the diff manually, I kept the top-level configuration changes I made myself and removed changes to dependencies (merge_kconfig.sh takes care of dependencies). This makes the delta more 'portable' across kernel versions