Successful Compile of a huge 5.16.12, Will Not Boot<SOLVED>

[rockedge]

Last night while working on KLV I decided to put the blade server through some paces and in the meanwhile compile a kernel.

So I set up with the latest "testing" branch kernel-kit and during the configuration stage, enabled built in overlayfs, added support for parallel and serial ports and virtualization, PREEMPT is selected and some clock speed adjustments made. The AUFS patches are applied. I enabled quite a few new features in this kernel. Using a Puppy Linux Bionic64 with a 4.19.82-rt30 full real time kernel booted to do the compiling, I only half way expected it to build successfully.........

Well it did! Finished normally and in around 35 minutes. So with this new,really stuffed full of features kernel, I went to try to boot it in KLV-Airedale-beta3 but failed with kernel panic. Tried to boot the 5.16.12 in Puppy Linux Bionic64, also kernel panic. Tried in a Fossapup64 and kernel panic. The error messages indicate problems with file system layering, though not 100% sure if this is really the only problem. I noticed also overall there are MANY new features added in the kernel configuration.

I might make it available to see if anyone is interested in seeing if it will boot for them on some other hardware.

[stemsee]

Probably its the rt patch causing the problem. Recompile without the patch and see if it's ok. I know @Dry Falls compiled 5.16 recently. Were overlayfs and aufs built_in=y or modules =m ?

[rockedge]

@stemsee I did not use any real time patches on the initial build and overlayfs = y with the AUFS 5.16 patches applied (via kernel-kit)

I am rebuilding it right now with the same configuration and using the new feature for PREEMPT that does not require the RT patches.

UPDATE: It works!!!! With all the same enabled settings I recompiled the kernel in a Fossapup64 running a 5.4.70-rt40 full real time kernel. It took 40 minutes to compile and swapped it into the Fossapup64 with change_kernels that built it. Boots cleanly and is idling well and performance so far looks good.

Will try it out in KLV-Airedale-beta3 next.

[Vverve]

Hey rockedge
Bump into log in ?!, wanted to test your 5.16.12
bump into aufs 5.16 too,with peebee kernels , is this the end of pmedia=ataflash ?
or just a transition state ? am i mssing overlays knowledge? yep..
i dont catch all of aufs ,amongst a lot more..

[Vverve]

Well it did! Finished normally and in around 35 minutes.

Wow...my one and only woof-ce kernel compiling success took almost 2.5 hours ...
I got heat from the grid here...i should trade an electric baseboard for a server for winters..

[rockedge]

@Vverve I will fix the web site page! For now look under "Kernels" here ->
https://rockedge.org/kernels/

[BarryK]

I am very interested to know that kernels greater than 5.10.93, greater than 5.15.16 and greater than 5.16.2 work for you. See my post:

https://bkhome.org/news/202202/kernel-51610-broken.html

Yes, 5.15.16 works for me. Haven't tried 5.15.17+. This is for EasyOS.

Hmmm, so 5.16.12 is ok for you, no issues?

I did get some feedback that the kernel developers were scrambling to fix the mess of 27 January. 5.16.10 was no good for me, perhaps it is fixed for 5.16.12.

[rockedge]

@BarryK I had no luck with kernels 5.15+ which either wouldn't compile cleanly or booted into a kernel panic. I stopped for a while to compile kernels because of the failures.

In this 5.16.12 I have overlayfs built in and support for parallel and serial ports enabled and PREEMPT SMP also enabled for low latency without having to apply the full real time patches. The RT patches have not worked well in the kernel-kit builds since version 5.4+.

So far the 5.16.12 is booting on a Fossapup64, idles well and seems efficient. Although I have experienced once so far a messy shutdown/reboot. Kicked out a lot of messages and hex dumps and seized up with me having to hard stop the machine.

I am using it now and am going to try it out on several different Puppy's. The one attempt with KLV-Airedale ended in a kernel panic. But in Puppy Linux so far so good......

For download here -> https://rockedge.org/kernels/

[Grey]

So far the 5.16.12 is booting on a Fossapup64, idles well and seems efficient. Although I have experienced once so far a messy shutdown/reboot.

Hi. Fossapup works flawlessly (I'm currently on this kernel with optimization patches for Zen 2/Ryzen cpus), but not in Jammypup. I talked to Phil about it yesterday:
viewtopic.php?p=51338#p51338

[Grey]

works flawlessly

So, flawlessly with Nvidia drivers. The "NULL pointer dereference" bug pops up with nouveau.
Just in case, I rolled back to 5.16.9

[wiak]

So far the 5.16.12 is booting on a Fossapup64, idles well and seems efficient. Although I have experienced once so far a messy shutdown/reboot. Kicked out a lot of messages and hex dumps and seized up with me having to hard stop the machine.

I am using it now and am going to try it out on several different Puppy's. The one attempt with KLV-Airedale ended in a kernel panic. But in Puppy Linux so far so good......

Off the top of my head, without yet trying it (but downloading just now), one thought that strikes me is that Fossapup64 uses aufs, whereas KLV-Airedale needs overlayfs; if overlayfs faulty, you would certainly be met with kernel panic on boot. Another OS that uses overlayfs by default is vanilladpup I believe, so you could try your kernel with that one to see if the built-in overlayfs code is faulty.

EDIT: the modules.sfs for your huge kernel contains overlayfs, so I believe therefore that overlayfs is not built into the kernel rockedge, but rather that the kernel expects that module for it. I will see if it boots on my copy of KLV-Airedale beta3 in a moment.

[wiak]

As you can see from lsmod, overlayfs is being loaded as a module with your new kernel.

It booted fine, but couldn't see my wifi interface for some reason (iwlwifi-based). I haven't checked further to see if I can ascertain why.

[dimkr]

If you need something to compare with, you can find kernels with built-in overlayfs in https://github.com/puppylinux-woof-CE/w ... #artifacts.

Vanilla Dpup uses kernel-kit-output-usrmerge-5.10.x-x86_64 and kernel-kit-output-usrmerge-5.15.x-x86_64, both have aufs=y, overlayfs=y and everything is under /usr.

[wiak]

@rockedge: I 'think' I know the reason why the wifi isn't working for me with your new huge kernel.

It is simply to do with the lib arrangement. In the fdrv you have the firmware at /lib/firmware, and in the zdrv, the modules at /lib/modules. That is the expected arrangement for Fossapup, so will work fine with Fossapup.

With KLV-Airedale the required place for modules is /usr/lib/modules and for firmware /usr/lib/firmware (/lib is simply a symlink to /usr/lib in Void filesystem structure). The result is your new fdrv tries to put the firmware at /lib but fails. (00modules is a 'trick' that auto moves /lib/modules to /usr/lib/modules via a bind mount, so that bit works). Your normal KLV-Airedale existing fdrv has /usr/lib/firmware as it internal structure, so that KLV-normal-supplied firmware has no issues - I believe therefore (but have not yet tried) that if I rearrange your new fdrv to /usr/lib/firware hierarchy all will work.

Note: I talked about possibility of using layer 02 for modules.sfs if overlayfs was built in directly to the kernel (rather than as one of the modules). Unfortunately, as I showed in previous post, overlayfs is still just a module and thus if you try using 02 for modules then overlayfs won't be found by the initrd and so you will get kernel panic - perhaps that is what you did (my wifi problem was different, as I explained just now).

[wiak]

@rockedge
I can now confirm my above post comments.
I am right now posting from KLV-Airedale using your 5.16.12 kernel. The zdrv... I simply renamed 00zdrv... but I unsquashed the fdrv..., created folder usr inside it and moved its lib into there (so now has structure /usr/lib/firmware) and finally renamed that unsquashed directory 01fdr... Obviously I disabled the previous 01fdrv....sfs (I just put a big D in front of its filename).

I then rebooted and all working fine, including my iwlwifi. So a simply problem as it turns out - nothing wrong with your kernel/modules/firmware that I can see - just needed the firmware in /usr/lib/firmware instead of /lib/firmware since Void Linux makes /lib simply a symlink to /usr/lib so /lib/firware couldn't be mounted (via the overlay over that /lib symlink - that was the simple issue). So no actual problem with your huge kernel at all - just needed minor rearrangement for Void filesystem hierarchy.

Fossapup uses the, what I think is now old, structure of /lib/firmware and /lib/modules (hence your compile working for that out-of-the-box). But I think future is moving towards Void Linux/Arch Linux type /usr/lib/firmware and /usr/lib/modules with /lib simply being a symlink to /usr/lib and /bin simply being a symlink to /usr/bin
EDIT: But as I also said above, your new huge kernel does not contain overlay driver inbuilt - only as a module; for KLV-Airedale it would be better if overlayfs was built in directly to the kernel.

[Dry Falls]

I built k5.15.13 with stock fatdog config and used the result of that for k15.19 with the woofce kernel kit. Built k5.16.9 and k5.16.12 with that derived DOTconfig with Stemsee's kit and have experienced no problems. Well, except that it's quite a bit bigger package and runs about 15-20 additional kernel processes (all the time!). I'm sticking with the 5.15x kernels. I've heard 5.17 will be smaller but will lose support for older systems.

[Vverve]

null pointer dereference problem on shutdown...EDITED..
Studying all rc.shutdown variants i find about pmedia=ataflash(pupm-13)..on old forum.
Learning basics here..better late then never..

rockedge
your 5.16.12 is flawless here ,same for peebee one..
except hard reset ,if pupmode 13,same for peebee.
onboard intel gpu here.
__________________
null pointer problem: i aint got a real clue if looking at shutdown script behavour when using pupmode13 will help,neither if its looking for a usb ¨pointer¨(sata ssd here) or if it get the drive in ¨sleep state¨ , my noob understanding of aufs and/or overlays world is still on the slow motion learning curve.
link to a clue or where the magic happpen would help!!
Humbles numerous bows to magicians...eyes cant follow hands tricks.

[rockedge]

@Vverve Thank you for the report! I re-compiled and improved the configuration on 5.16.12
This build should include the fix for the shutdown bugs.

@wiak this time overlayfs is built in the kernel. Please check if it is truly so.

• Parallel and Serial port support and low latency SMP PREEMPT is enabled.

• Virtualization support for Intel and AMD enabled.

• Some clock adjustments to better balance response.

The kernel files will be uploaded later today as 5.16.12-re

[wiak]

@Vverve Thank you for the report! I re-compiled and improved the configuration on 5.16.12
This build should include the fix for the shutdown bugs.

@wiak this time overlayfs is built in the kernel. Please check if it is truly so.

• Parallel and Serial port support and low latency SMP PREEMPT is enabled.

• Virtualization support for Intel and AMD enabled.

• Some clock adjustments to better balance response.

The kernel files will be uploaded later today as 5.16.12-re

Just downloaded KLV-Airedale beta4 and will test once I get back from my coffee... Let me know when the 5.16.12-re kernel is uploaded; it isn't there yet (unless it is the 5.16.12-prempt one, which I haven't downloaded yet. Looking forward to kernel with overlayfs built in since will be able to tweak w_init such that I can use it in say 02 layer and then pemasu compile modules issue will be resolved without any uncompression of 00modules required (though that is easy to do anyway).

[rockedge]

@wiak The kernel package is ready for download.

5.16.12-RE

• overlayfs built in

• virtualization support enabled

• parallel and serial port support = yes

• SMP and PREEMPT enabled

• CONFIG_HZ_300=y

Screenshot_2022-03-07_22-28-18.png (15.19 KiB) Viewed 1039 times

[Vverve]

The kernel package is ready for download.

rockedge ,is this ¨re¨ for klv only?
only test in imppup64 , boot ok,xauthority prob and messed firmware !?
and worst logs on shutdown(ataflash\boot)..hard resetting.
i will give it another go,re download and retry,
it was too weird.

Grey 5.16.13 shutdown ok
edited;reading around,you all seem to patch aufs...i'll wait to test your next one.
Gnan...doesnt need to compile kernels ourself...
already hardly keep the pace testing of all those you all busy coders share..
bark bark& long howling..you all rock..