[CLOSED] New kernel 6.4.7 / 8 - boot hangs on desktop with Intel Xeon & Nvidia

[peebee]

See below - looks like this problem is specific to my desktop hardware..... Intel Xeon cpu & Nvidia

This is most unusual.......

Kernel 6.4.7 was released yesterday (27 July 2023) and I built it as per the normal method I've used for many previous kernels. There were no build errors.

I then installed the new 6.4.7 into a system image that was running just fine with 6.4.6

Rebooted - and the boot hangs at the "Installing kernel modules" phase

Have tried it with punionfs=overlay with the same result so I don't think it is AUFS related....

Have tried it with loglevel=7 and loglevel=6 and although diagnostics are produced an immediate cause is not apparent.

If anybody else tries to build 6.4.7 perhaps they could report here if they have any problems??

I've looked at the kernel bug reports and cannot see any similar problem reports.....

Thanks
peebee

[rockedge]

@peebee Using F96-CE_4, I am compiling 6.4.7 with aufs 6.3 using the kernel-kit from woof-CE Jammy build option.

So far the run is going smoothly halfway finished......... DOTConfig is similar to the 6.0.12 that's currently running in this F96-CE with the PREEMPT set to low-latency. This run hopefully will complete to test if the modules will boot with the kernel smoothly.

For the 6.1.38-rt13c I used @ozsouth's 2018 kernel-kit components but not for this kernel version. I am using a just git pulled woof-CE that set up the Jammy build environment and kernel-kit.

[Grey]

I am compiling 6.4.7 with aufs 6.3

Confirm. With version 6.4 it is not going to, I manually registered 6.3. Well, also, as usual, I replaced two https links with git. After that, everything worked out.

[Grey]

I could have looked at logs and conf files. But I can't. I was too lazy to make a new Nvidia driver for this kernel and I left 6.1.22 in Fossapup. And deleted the kernel-kit folder.

[rockedge]

The build of 6.4.7 failed on 4 attempts with different scripts and configs.

This is the portion in the log where the error occurs.

Code: Select all

CC      drivers/pnp/pnpacpi/core.o
  CC [M]  sound/pci/emu10k1/emu10k1_main.o
fs/aufs/xattr.c:355:3: error: ‘posix_acl_access_xattr_handler’ undeclared here (not in a function); did you mean ‘posix_acl_xattr_header’?
  355 |  &posix_acl_access_xattr_handler,
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |   posix_acl_xattr_header
fs/aufs/xattr.c:356:3: error: ‘posix_acl_default_xattr_handler’ undeclared here (not in a function); did you mean ‘posix_acl_xattr_header’?
  356 |  &posix_acl_default_xattr_handler,
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |   posix_acl_xattr_header
make[3]: *** [scripts/Makefile.build:252: fs/aufs/xattr.o] Error 1
make[2]: *** [scripts/Makefile.build:497: fs/aufs] Error 2
make[1]: *** [scripts/Makefile.build:497: fs] Error 2
make[1]: *** Waiting for unfinished jobs.... 

and then the last messages at the end of the log ->

Code: Select all

CC [M]  drivers/hwmon/xgene-hwmon.o
  LD [M]  drivers/hwmon/nct6775.o
make: *** [Makefile:2032: .] Error 2
Error: failed to compile the kernel sources.

It appears that the AUFS patch is causing the problem in this case.

Going to try a run with OverlayFS only and no AUFS to better pinpoint the problems.

[Grey]

Let's wait 6.5. And then we'll see. 6.1.22 suits me so far

[peebee]

I am compiling 6.4.7 with aufs 6.3 using the kernel-kit from woof-CE Jammy build option.

Hi @rockedge

You need to use aufs6.x-rcN for 6.4 builds......

[Grey]

They definitely added something in July. Or removed I can't make an Nvidia driver even for kernel 6.1.42
Something there with stdarg.h
I feel that 6.5 will be even worse

[ozsouth]

Seems to me 5th July (6.1.38) & 11th July (6.4.3) were last good kernels before some big changes - which seems to have caused issues. I always worry when big changes are made that bugs can creep in. I know that ChromeOS always issues kernels about 12 behind current for safety.

[dimkr]

Rebooted - and the boot hangs at the "Installing kernel modules" phase

Where can we find the ISO that reproduces this problem?

(And I don't understand this rush to build the latest kernel, a great source of new issues on old hardware, even before aufs officially supports this kernel version)

[peebee]

Where can we find the ISO that reproduces this problem?

I've uploaded huge-6.4.7-broken-lxpup64.tar.bz2 to:
https://u.pcloud.link/publink/show?code ... sTshl7a5tV

You can try it in any NOT-usrmerge build. e.g. S15Pup64 or VoidPup64

[peebee]

6.4.7 built with AUFS=no - i.e. without AUFS also hangs at the same point in the boot sequence..........
so not an AUFS issue.........
some sort of change between 6.4.6 and 6.4.7 - change log:
https://cdn.kernel.org/pub/linux/kernel ... eLog-6.4.7

[dimkr]

Seems to work fine in a clean VM

[rockedge]

Finally made a clean build of 6.4.7 using aufsv=6.x-rcN and I have included the DOTconfig-6.4.7-x86_64-280723 as a compressed attachment.

The build run seemed to complete normally except for the fact there was no modules SFS produced by the Jammy kernel-kit.
This is what the output directory looks like:

Screenshot(20).jpg (13.74 KiB) Viewed 4384 times

This is the output of the modified kernel-kit Fossapup after building 6.1.38-rt13c successfully.

Screenshot(21).jpg (21.58 KiB) Viewed 4379 times

And I don't understand this rush to build the latest kernel, a great source of new issues on old hardware, even before aufs officially supports this kernel version

No rush and not pressing to make one. Only tried a build to see if the newer woof-CE kernel-kits work at all since I have not had many clean builds of even routine general purpose nominal performing kernels. Most have failed to compile using the kit without modifications (like ozsouth's kernel-kit components)

The directory linux_kernel-6.4.7-rock looks like it contains what goes into a zdrv SFS so could always try to squash it and give it a try.

Downloaded @peebee's build and will attempt to boot it in F96-CE_4

[rockedge]

@peebee The kernel you are experiencing the stalled boot with works with F96-CE_4 as is.

Screenshot(22).jpg (28.94 KiB) Viewed 4373 times

I'll test drive the QEMU VM running it and see how it goes then on a bare metal machine. (sounds like a good name for a electronic rock band)

[dimkr]

Most have failed to compile using the kit without modifications (like ozsouth's kernel-kit components)

If you have any build logs from these failures - please share them and I'll see what I can do

Debian testing and unstable already received the update to 6.4.x, both built successfully by https://github.com/puppylinux-woof-CE/w ... kernel.yml and seem to work fine.

[peebee]

Thank you both for testing.
The problem does seem to be specific to the hardware of my desktop - see below.
Your tests prompted me to try a boot on a laptop and indeed that booted correctly with no issues.

@dimkr is there anything I can do to find out which module is causing the problem? Assuming it is just one module of course....
Should loglevel=7 provide any clues?

My desktop spec:
Computer Profile:
Machine Dell Inc. Precision WorkStation T5400 (version: Not Specified)
Mainboard Dell Inc. 0RW203 (version: NA)
• BIOS Dell Inc. A11 | Date: 04/30/2012 | Type: Legacy
• CPU Intel(R) Xeon(R) CPU E5450 @ 3.00GHz (4 cores)
• RAM Total: 7955 MB | Used: 1056 MB (13.2%) | Actual Used: 311 MB (3.9%)
Graphics Resolution: 1366x768 pixels | Display Server: X.Org 21.1.8
• device-0 NVIDIA Corporation GT218 [NVS 300] [10de:10d8] (rev a2)
Audio ALSA
• device-0 Intel Corporation 631xESB/632xESB High Definition Audio Controller [8086:269a] (rev 09)
• device-1 NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
Network wlan1
• device-0 Ethernet: Broadcom Inc. and subsidiaries NetXtreme BCM5754 Gigabit Ethernet PCI Express [14e4:167a] (rev 02)
• device-1 USB Device: IMC Networks XHC Camera
Printers CUPS installed
• device-0 DefaultPrinter Dell_Color_Laser_1320c
• device-1 Printer CUPS-PDF
Drives Total: 1.20T
• sda ID-0: ST3320620AS (298.1G)
• sdb ID-1: WDC WD5000AZLX-75K2TA0 (465.8G)
• sdc ID-2: CT500MX500SSD1 (465.8G)
• zram0 ID-3: RAM Swap Disk (7.6G)
Partitions Mounted-FS Size Used Avail Use%
• sda2 /dev/sda2 86G 55G 27G 67%
• sdb1 /dev/sdb1 230G 151G 68G 69%

[dimkr]

Should loglevel=7 provide any clues?

Maybe. Do you have any log files to share?

By the way, are you sure you swapped zdrv with the 6.4.7 one?

[rockedge]

@dimkr Thanks for the offer! I will do this on the next round of kernel-kit builds. Almost all of the failure logs I had have been removed recently during some house-keeping to make more storage space!! Now I wish I had a few......but you helped massively with your tip looking over the failure log of the 6.1.38-rt13 builds with the kernel-kit pointing out where the errors were occurring. Now I load the log and first start a search for "error"!

Just that little tip helped better organize how to approach finding a diagnosis. So your help is always really appreciated and that it's coming from a true professional in the trenches makes it that more exciting.

[peebee]

Should loglevel=7 provide any clues?

Maybe. Do you have any log files to share?

By the way, are you sure you swapped zdrv with the 6.4.7 one?

No logs because the system freezes. Power button has to be pressed...

100% certain.

I can interrupt the boot with rdsh

[peebee]

After MANY iterations of putting extra messages into /etc/rc.d/rc.sysinit
via a ydrv.sfs
I'm finding that "hangs" are happening at various locations
with the 1st being at line 587:
udevadm settle -t 20

If that line is commented out, the boot proceeds a bit further and the next hang is within /etc/rc.d/rc.country when called from line 754 of rc.sysinit

The hang within rc.country is at line 63:
localedef -f ISO-8859-1 -i en_US --no-archive en_US >/dev/null

This "random" behaviour maybe suggests a cpu instruction issue of some sort given that my desktop has fairly uncommon Intel Xeon cpus??

Code: Select all

from rc.country:
	if [ "$ASKLANG" = "no" ];then
	#'quicksetup' is run 1st time X starts.
	#need this default locale for X to start first time...
echo -n "xx0..." >/dev/console
	if [ ! -d /usr/lib${lsuffix}/locale/en_US ] ; then
echo -n "xx1..." >/dev/console
		localedef -f ISO-8859-1 -i en_US --no-archive en_US >/dev/null
echo -n "xx2..." >/dev/console
	fi

No "xx2...." appears

[Grey]

• CPU Intel(R) Xeon(R) CPU E5450 @ 3.00GHz (4 cores)
This "random" behaviour maybe suggests a cpu instruction issue of some sort given that my desktop has fairly uncommon Intel Xeon cpus??

I don't know what's going on there. But a great processor. I have one in computer in the next room. There have never been any problems.

[Grey]

• CPU Intel(R) Xeon(R) CPU E5450 @ 3.00GHz (4 cores)
This "random" behaviour maybe suggests a cpu instruction issue of some sort given that my desktop has fairly uncommon Intel Xeon cpus??

I don't know what's going on there. But a great processor. I have one in computer in the next room. There have never been any problems.

I just called a friend. He says there may be two problems. Some instructions like sse4.1 may not be supported and the core temperature may not be determined correctly.

BUT! ONLY if your BIOS is not adapted. My processor is not only physically modified (a special sticker), but also the BIOS is prepared.

[peebee]

The version of 6.4.7 produced by Porteus shows the same "hang" problem ( https://forum.porteus.org/viewtopic.php?p=94991 )

But their version of 6.5-rc4 boots OK

[peebee]

6.4.8 has appeared - sadly hangs in the same way as 6.4.7 on my desktop Xeon + Nvidia

[ozsouth]

I note that we are now 6 weeks after the release of 6.4 & the aufs branch has not been released, despite sfjro making recent updates to other 6.x branches. This is unusually long. Perhaps there is a problem with aufs & 6.4.

[peebee]

I note that we are now 6 weeks after the release of 6.4 & the aufs branch has not been released, despite sfjro making recent updates to other 6.x branches. This is unusually long. Perhaps there is a problem with aufs & 6.4.

I believe the problem is with sfjro's development machine - he's had a ssd failure which is proving difficult to recover from.
For the moment aufsv=6.x-rcN is working OK for 6.4 - the problem may come when 6.5 is released.....

[dimkr]

For the moment aufsv=6.x-rcN is working OK

This is ironic, sfjro's data loss slowed down aufs development, so you're using an unstable aufs version against a kernel version it wasn't "released" for, risking Puppy users with data loss.

aufs builds fine, but this doesn't mean this aufs branch is stable and safe to use against this particular kernel version.

[peebee]

risking Puppy users with data loss.

Nobody is at risk without their knowledge - the availability message has the facts - the kernel is not used in any released iso - people can try it if they want - what harm is there in that??

There are NO problems with 6.4.6 with aufsv=6.x-rcN that I have seen in extensive "personal risk" use. 6.4.6 has only been downloaded 29 times and nobody has reported any problems.

The problems with 6.4.7 & 6.4.8 on my Xeon / Nvidia desktop are not due to aufs [proved].

[dimkr]

has the facts

Not all of them - it doesn't specify which aufs commit, so it's hard to tell which issues in https://github.com/sfjro/aufs-standalone/issues might affect this build.

(Feel free to disagree with me, but I think you should have mentioned that this is not a stable aufs version with sfjro's stamp of QA approval for 6.4.6. A "released" version of aufs made me lose some data, and I prefer not to hear about others with the same fate.)