Discussion, talk and tips
https://forum.puppylinux.com/
When a new puppy is made, e.g BookwormPup64_10.0.8, or a new KL is made, e.g. KLV-vmHost-rc4, does the kernel used undergo any modifications?
Are kernels modified a lot or are they the standard from someplace else?
I'm just curious how this is done.
Thanks
Are kernels modified a lot or are they the standard from someplace else?
The source code for the kernel's comes from the same places. Like -> https://git.kernel.org/
or here -> https://www.kernel.org/category/releases.html
During the build process the are extensive options to configure the kernel to be built that are in a configuration file. There are different approaches as what to configure depending on the kernel's intended uses. Some are made for Android, some for embedded devices like digital cameras (any devices using Linux) and some are for general use. For our uses I suppose we try to generate kernels that can work well in a Distro across a broad range of possibilities in desktop PC's and laptops.
does the kernel used undergo any modifications?
For full real time kernels a patch must be applied to the kernel source code to add the code needed for fully PREEMPT real time kernels.
Here are the patches -> https://cdn.kernel.org/pub/linux/kernel/projects/rt/
Welcome to the dark and mysterious art of Kernel construction 
A very small example below. A "standard" config can be 9000-10000+ lines long.
Code: Select all
# Linux/x86 6.6.0-rock Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=90400
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23400
CONFIG_LD_IS_BFD=y
CONFIG_LD_VERSION=23400
CONFIG_LLD_VERSION=0
CONFIG_CC_CAN_LINK=y
CONFIG_CC_CAN_LINK_STATIC=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
CONFIG_PAHOLE_VERSION=0
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y
#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
# CONFIG_WERROR is not set
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_BUILD_SALT=""
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_HAVE_KERNEL_ZSTD=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_KERNEL_XZ=y
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
# CONFIG_KERNEL_ZSTD is not set
CONFIG_DEFAULT_INIT=""
CONFIG_DEFAULT_HOSTNAME="fossapup"
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
# CONFIG_WATCH_QUEUE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
# CONFIG_USELIB is not set
# CONFIG_AUDIT is not set
CONFIG_HAVE_ARCH_AUDITSYSCALL=yThanks for this information @rockedge
That is helpful -- and complicated.
From my encounter with wifi firmware or other device firmware,
is that firmware added to the kernel or is the version of the kernel already set to what devices it will be used on?
Thanks again for the information.
AFAIK BookwormPup64 is built from the Debian kernel source without additional patches, only ~130 kernel configuration changes (see https://github.com/puppylinux-woof-CE/w ... s/bookworm). I set up this build pipeline, and AFAIK BookwormPup64 uses it. Also AFAIK, BookwormPup64 uses Debian's firmware packages, which should be compatible with this kernel version.
Other distros in this forum do other things, some mix a kernel from one source with firmware from another source, or have patched kernels.
If your particular hardware doesn't work with BookwormPup64, tells what it is and what firmware was missing so this can be fixed.
I have a new laptop that Bookworm only gives one resolution for the display.
I'll address this over on the BookwormPup64_10.0.8 Topic after I determine what firmware it needs for this.
The peculiar thing was that another KLV that has a 6.10.10 kernel couldn't connect to wifi. But BookwormPup could even though it uses an older 6.1.106 kernel.
So I found the wifi kernel files from BookwormPup and tried them in the KLV, which worked.
I was just trying to understand how kernels addressed these things.
Thanks
Configuring the kernel for good hardware support is not easy, especially without a process that prevents you from repeating the same mistakes every time you configure a kernel. Kernels people publish in this forum vary a lot because each is tested against different hardware, and there's no shared base configuration file everybody is using as the starting point.
The tie between the kernel version and the firmware packages makes the problem worse because some take firmware from a distro with an old kernel, so it doesn't include everything needed by the more recent kernel.
If something works for you in Debian 12, it's almost guaranteed to work with the BookwormPup64 kernel, because it uses the same versions of everything with almost identical configuration.
If it's a new laptop that doesn't work well with Debian 12, maybe it's too new to be supported by Debian 12.
The tie between the kernel version and the firmware packages makes the problem worse because some take firmware from a distro with an old kernel, so it doesn't include everything needed by the more recent kernel.
Huge problem is the woof-CE kernel-kit will no longer reliably assemble a firmware fdrv SFS. What ever the problem is like no reliable filter process for sorting through the very large complete firmware collection to create the 60-150 M sized fdrv/01firmware SFS's instead of 300-400 M the firmware collection is. The bottom line is the kernel-kit during manual runs only will complete the firmware portion 5% of the time. The kernel will compile successfully but only that 5% of the time does the fdrv get completed.
I can take and make an fdrv or 01firmware directly from the extracted Void Linux kernel. Which when used in KLV is the Void Linux built kernel extracted into a modular form. We have no dedicated mechanism to filter the firmware down to the target sizes of 01firmware or fdrv.
The Void Linux kernel's firmware is around 400 M and like mentioned with no well working filtering program to create a modular version, so I will use the firmware SFS's coming from Puppy Linux that already exist, either converted to usrmerge or used directly from a kernel version that is in the same series. This practice seems just fine. As long as the 01firmware/fdrv is in the same kernel series as the vmlinuz and modules this seems to work okay.
I have regularly taken @dimkr's kernels from VanillaDpup and used them or just the firmware SFS's.
Great thing is in KLV one can use the modular huge kernel or during the original rootfs construction just build in the actual Void Linux kernel and manually modify the skeleton initrd.gz to boot the system.
We can make available a Void Linux most recent version kernel firmware setup when size is of no matter and downloading 2 G images and looking to frugally install these is in one's range of know-how. I have tested with this large 01firmware and it covers a lot territory and yet to find something missing. But that's too big, even the VanillaDpup fdrv covers a lot of hardware but also is really big.
I have experimented with using the woof-CE kernel-kit firmware filtering script on the large complete firmware packages, coming from various sources with some success but this would be one area of development that needs some work. Constructing a configurable firmware filtering system that can manage to assemble variants of firmware SFS's
Over recent years, I have offered folk my kernel kit (modified 2018 version), but I really believe it is in it's twilight, as misconfiguration, particularly missing vital additions, is becoming too easy to do. I compile & test mine on a Dell i5 laptop, & my recent Celeron N4500 laptop is where I'm using my latest kernels. Whilst it does allow me some flexibility, I'm not confident (I'm over 70) in my ability to do this properly for too much longer. (that's why everything I make comes with a 'use at own risk' disclaimer).
Also, teams of astute IT folk make kernels for major distros, so much less likely to miss things.
That said, Puppy has been a tinkerer's distro, & many folk like minimalisation, especially for rescuing old pcs from trash. My fdrvs are small to accommodate this, but it will always mean some folk will be missing firmware, so I'm now adding a link to a big firmware collection.
There is a certain satisfaction in making something 'useless' work, but new pcs are a different proposition.
I was using dmesg to compare different startups of some different os's.
I think now I understand that some things like wifi try to use kernel firmware built in first, and if it can't find it, it uses some firmware that is added.
When it uses firmware that is added from a directory, the dmesg says that that firmware is "Loaded".
When it uses firmware that comes from the kernel, the dmesg just mentions the firmware it is using.
So those are two separate things.
I see the advantage that if the kernel doesn't have the needed firmware, then placing the firmware in a folder can make it work.
Am I correct with my understanding for this?
Thanks
JusGellin wrote: ↑Sat Nov 16, 2024 4:21 pmI was using dmesg to compare different startups of some different os's.
I think now I understand that some things like wifi try to use kernel firmware built in first, and if it can't find it, it uses some firmware that is added.
When it uses firmware that is added from a directory, the dmesg says that that firmware is "Loaded".
When it uses firmware that comes from the kernel, the dmesg just mentions the firmware it is using.So those are two separate things.
I see the advantage that if the kernel doesn't have the needed firmware, then placing the firmware in a folder can make it work.Am I correct with my understanding for this?
Thanks
Not really, the overlayfs ( or aufs) creates a single file system which is used. The system is using firmware from /usr/lib/firmware. There isn't any hunting for firmware, or any file.
You could have valid firmware in the 01fdrv.sfs and also in someother.sfs(not a good idea) and another contained in your persistence layer( read/write). As each layer is overlaid, only the top version is accessible to the system. As the persistance is the highest layer, it wil be the version that is available. The other versions effectively don't exist.
To use an analogy, if you have ever done any photo editing using layers, the end resulting image shows the top most visible portion of the merged layers.
@TerryH
Thanks, that's a good point.
But I was looking to see where the driver came from for two different laptops, using the same KLV-vmHost.rc4 (kernel 6.10.10.1) from Ventoy.
One laptop fails to run wifi.
dmesg indicates it loads :
iwlwifi 000:00:14.3: loaded firmware version 89.e9cec78e.0 so-a0-gf-a0-89.ucode op_mode iwlmvm
But then later on it fails:
iwlwifi 0000:000:14.3: Failed to run INIT ucode:-5
When I search for so-a0-gf-a0-89.ucode, it locates the firmware file at /usr/lib/firmware/iwlwfi-so-a0-gf-a0-89.ucode
The other laptop runs wifi when booted up:
dmesg indicates it uses:
rtw_8821ce 0000:02:00.0: Firmware version 24.11.0, H2C version 12
No fail occurs and wifi works.
When I search for rtw_8821ce, it locates the firmware file at
/usr/lib/modules/6.10.10/kernel/drivers/net/wireless/realtek/rtw88/rt88_8821ce.ko
There isn't any rtw88/rt88_8821ce in /usr/lib/firmware this time.
So I take it to mean that this came from the 6.10.10 kernel for this one.
Is that correct? Or am I missing something?
This is helping me to look deeper to try to understand this better.
Thanks for your help on this.
JusGellin wrote: ↑Sat Nov 16, 2024 11:15 pm@TerryH
Thanks, that's a good point.But I was looking to see where the driver came from for two different laptops, using the same KLV-vmHost.rc4 (kernel 6.10.10.1) from Ventoy.
One laptop fails to run wifi.
dmesg indicates it loads :
iwlwifi 000:00:14.3: loaded firmware version 89.e9cec78e.0 so-a0-gf-a0-89.ucode op_mode iwlmvm
But then later on it fails:
iwlwifi 0000:000:14.3: Failed to run INIT ucode:-5
When I search for so-a0-gf-a0-89.ucode, it locates the firmware file at/usr/lib/firmware/iwlwfi-so-a0-gf-a0-89.ucodeThe other laptop runs wifi when booted up:
dmesg indicates it uses:
rtw_8821ce 0000:02:00.0: Firmware version 24.11.0, H2C version 12
No fail occurs and wifi works.
When I search for rtw_8821ce, it locates the firmware file at
/usr/lib/modules/6.10.10/kernel/drivers/net/wireless/realtek/rtw88/rt88_8821ce.ko
There isn't any rtw88/rt88_8821ce in /usr/lib/firmware this time.
So I take it to mean that this came from the 6.10.10 kernel for this one.Is that correct? Or am I missing something?
This is helping me to look deeper to try to understand this better.
Thanks for your help on this.
i'm now out of my depth, as the first part of your dmesg referring to iwlwifi is for intel based wifi card. The firmware iwlwfi-so-a0-gf-a0-89.ucode is found, required by the kernel module iwlmvm, but the later failure to load I can't really advise on. Just guessing, but maybe an imcompatible/missing flag set when the kernel was compiled.
The second part referencing rtw_8821ce is for a realtek wifi card. I'm now totally lost on what wifi card is being used. *** (See edit bellow)
Hope you can get further assistance here, I'm definitely interested to see if others more knowledgeable can advise you further.
Edit: OK I now see now that you are talking about 2 different laptops here. I have limited knowledge about these cards, so won't attempt to offer advice for this card.
The laptop is Hp Pavilion x360 convertible 11m-ap0xxx
The nework device is Realtek Semiconductor Co., Ltd RTL8821CE 802.11ac PCIe Wireless Network Adaptor
Thanks
@JusGellin - re- your iwlwifi pc, would you please run the following in a terminal & post output:
lspci -nn | grep etwork
Also, from the /boot folder, would you please attach the config file there.
This is all that lspci gives
This is using void linux - KVM-vmHost-rc4 - maybe because of busybox?
Code: Select all
bash-5.2# lspci -nn | grep etwork
bash-5.2# lspci
00:08.0 Class 0880: 8086:a74f
00:15.1 Class 0c80: 8086:51e9
00:1f.0 Class 0601: 8086:519d
00:04.0 Class 1180: 8086:a71d
00:14.3 Class 0280: 8086:51f1
00:16.0 Class 0780: 8086:51e0
01:00.0 Class 0108: 144d:a80f
00:1f.5 Class 0c80: 8086:51a4
00:1f.3 Class 0403: 8086:51ca
00:00.0 Class 0600: 8086:a708
00:12.0 Class 0700: 8086:51fc
00:15.0 Class 0c80: 8086:51e8
00:06.0 Class 0604: 8086:a74d
00:14.2 Class 0500: 8086:51ef
00:02.0 Class 0300: 8086:a7ac
00:14.0 Class 0c03: 8086:51ed
00:1f.4 Class 0c05: 8086:51a3
bash-5.2#Did you want the grub.cfg?
/boot only has empty directory called kernel.
Let me know if you would like something else.
Thanks
I just noticed something looking at dmesg.
Before, I saw that iwlwifi said it failed.
But in between was another message:
PNVM data is missing, please install iwlwifi-so-a0-gf.a0.pnvm
I'll have to look for this to try to make it work.
Before I saw that BookwormPup64_10.0.8 was able to bring up wifi.
So I tried its firmware drivers on this which worked.
Perhaps I only need to add iwlwifi-so-a0-gf.a0.pnvm.
I wonder if BookwormPup has that so I could just install that.
Then @TerryH what you said may be right if I just add the pnvm file.
@JusGellin - sorry, seems KLV doesn't have the files I was looking for in the usual paths I was expecting. As I have never used it, I'll have to leave this to rockedge.
One thing you could try though, is to see if wifi is turned off - in a terminal run: rfkill unblock all
That's not firmware. It's the kernel module for the 8821ce driver. You can determine what firmware is used by a module with the "modinfo" command.
Yes let us not forget the most important 00modules or zdrv.........firmware is a companion and deeply intertwined with the kernel modules.
The firmware that came with KLV-vmHost-rc4, for the iwlwifi-so-a0-gf-a0.89.ucode has a date of 05/19/24.
This is from 01firmware-6-9.1_1.sfs
The iwlwifi-so-a0-gf.a0.pnvm is missing from this.
But I was able to make it work using iwlwifi-so-a0-gf.a0.pnvm from BookwormPup64_10.0.8 which is dated, 5/1/23. So that is ok, then.
Does it do the same thing like firmware when it is used?
For that laptop that works, I could see it used that from the kernel.
It doesn't have modinfo command.
I found that it used that using find for what dmesg said it was using.
Thanks
@JusGellin - I think the above shows that KLV is not a usual-case generic puppy. Bookworm is more generic in its operation, so more folk on the forum could help with that. Rockedge (& Wiak?) would be most help with KLV.
At this time all of the KLV's can use Puppy Linux huge kernels that are usrmerge or converted to usrmerge and the firmware fdrv's are interchangeable as they are with Puppy Linux.
Most of the Void Linux kernels I extracted are similar in assembly to the huge Puppy Linux kernels only the initrd.gz is different.
For KLV I assembled both types. All it takes for a KLV to run Puppy Linux kernels is to swap in the FirstRib skeleton initrd.gz. This generic minimal version will boot all of the Puppy Linux huge kernels that are in the usrmerge format.
The Void Linux extracted kernel like 6.10.10_1 gets a slightly modified skeleton initrd.gz that during the extraction process has modules and firmware inserted into it to be able to boot and construct the system to hand over too.
As far as kernel choices go, for example, KLV can run well with the RT kernels. These are all made as huge kernels for Puppy Linux then converted to fit into KLV format.
So they are mostly the exact same kernels only named and arranged differently to work in the FirstRib type layered and merged file systems that KLV uses.
Hope that clarifies the kernel config somewhat 
Due to my inexperience on this and in my terminology I'm still wondering about something.
This pertains to BookwormPup64_10.0.8 with kernel 6.1.106, running on my new laptop that has Intel Raptor Lake PCH CNVi technology.
For this to have workable wifi, it requires firmware to be added.
So at /usr/lib/firmware there are two files needed:
iwlwifi-so-a0-gf-a0-72.ucode
iwlwifi-so-a0-gf-a0.pnvm
Then it works (It already had these two)
So since there is no driver for the graphics part which uses Intel Raptor Lake-U technology,
Should there be a similar way to make the graphics work since there is no driver being loaded?
That is, firmware at /usr/lib/firmware.
Should this be somewhere, I just can't find it?
Just hoping for some ideas. I would like to make Bookworm work for the graphic part as well.
Thanks
Yes the graphic device firmware works the same way and will work if you can identify, find and copy the correct firmware.
I have a new firmware SFS at 378 M but it should solve all of your target goals.
I can make this available and you would be also testing it......
Download -> 01firmware.sfs
rename it to work on Bookworm
@rockedge
I tried it but it doesn't seem to work correctly.
I tried to use it several times to make sure I was doing it right.
For the last time I tried it, I first did a frugal install of a new BW on the internal drive.
It comes up like it did before with wifi connecting but the graphics resolution not working like before.
When I look on this original install I can see /usr/lib/firmware gets populated.
The next time it came up I replace fdrv with yours renamed to fdrv.
Then I rebooted.
When it came up, wifi no longer worked and display couldn't do resolutions.
If I look at /usr/lib/firmware it is now empty except for 3 files:
regulatory.db
regulatory.db.p7s
skl_hda_dsp_generic-tplg.bin
If I once again replace fdrv now with its original and reboot,
once again /usr/lib/firmware gets populated and wifi works again.
I think I'm doing it right.
Thanks
If I copy the contents of the new sfs into /usr/lib/firmware and reboot, wifi works again, but display resolution doesn't work.
@JusGellin
Remember the name of the fdrv SFS has to be exact or it will not load.
Make sure the 371 M version is named exactly like the fdrv SFS that works and make sure there is only the one fdrv SFS present,
