What is Wrong with the Kernel-Kit?

[rockedge]

No matter what I attempt, today the kernel-kit completely fails as is directly after being created by ./merge2out

all kinds of github security failures.
I am about to just give up making them. When it's a struggle just to get it going at all, what's the point?

Last week still worked. I am really sick of this problem that at every turn it's some new more steps to take authoritative action that's difficult to implement never mind understand. Almost every thing I wanted to do ON MY OWN STUFF I can't because I'm not authorized enough. Screw it I'm done.

Code: Select all

fatal: remote error: 
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
WARNING: 'git pull --all' command failed
You have chosen to get the latest firmware from kernel.org
This may take a long time as the firmware repository is around 200MB
Cloning into 'linux-firmware'...
fatal: unable to access 'https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/': server certificate verification failed. CAfile: none CRLfile: none

[ozsouth]

@rockedge - both @Grey & I have found similar issues. At first I thought different kenels were having different requirements, but now it seems to vary with each compile. Different combinations of https:// and git:// seem to be required in build.sh. The 32bit kernels I made last week were the successes from 8 attempts. Today's 5.10.109 64bit kernel compiled first go with no git:// entries, as did my last 5.15(.30) I thought I'd make a few this week in case things got worse.

[rockedge]

No more unauthenticated Git

On the Git protocol side, unencrypted git:// offers no integrity or authentication, making it subject to tampering. We expect very few people are still using this protocol, especially given that you can’t push (it’s read-only on GitHub). We’ll be disabling support for this protocol.

it seems that some things need adjusting at both ends, A real drag this is

Git protocol troubleshooting

If you’re having trouble cloning a repository, make sure the URL starts with ssh://, https://, or git@github.com.

For existing repositories, if you’re having trouble fetching, check that the remote you are fetching from is an ssh:// or https:// URL, or an SSH pseudo-URL starting with git@github.com. You can run git remote -v in the repository to see the URLs for all remotes. If any of them start with git://, you should change the URL to a supported format.

Somewhere in this chain is the problem. The url's the kernel-kit is trying to fetch

[Grey]

similar issues.

Yeah, I see the problems that started with me about three weeks ago gradually got to the rest.

I solved them in two ways: using the correct combination of git and https, or using Personal access tokens (for Github).

Now I'll check the last option build.sh . Maybe it still works.

[Grey]

@rockedge, @ozsouth
So, I checked my last one build.sh. I ran Beediff (embedded in Fossapup, file difference manager) and compared the contents of the old and new script. It turned out that the solution from this post works.

That is, lines 435 and 475 need to be changed. Note that line 435 is not Github, but Sourceforge.

[rockedge]

@Grey Now I am seeing this as well as still getting a fatal error earlier in the run with aufs-utils

Code: Select all

Press ENTER to begin
Linux: 5.16.14
aufs=5.16
aufs_util=5.8
linux-5.16.14.tar.xz: OK
fatal: remote error: 
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
WARNING: 'git pull --all' command failed

You must now configure the kernel
Hit a number or s to skip:
1. make menuconfig [default] (ncurses based)
2. make gconfig (gtk based gui)
3. make xconfig (qt based gui)
4. make oldconfig
s. skip

Enter option: 2
make gconfig
  HOSTCC  scripts/basic/fixdep
as: error while loading shared libraries: libbfd-2.34-system.so: cannot open shared object file: No such file or directory
make[1]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 1
make: *** [Makefile:555: scripts_basic] Error 2

no matter what I try this error crashes the run: libbfd-2.34-system.so: cannot open shared object file: No such file or directory

Just last Friday I ran the exact system and built a kernel no problems. Today, without any changes, started the same system to build a fresh kernel to try some configurations and suddenly nothing works at all.

So far still messing around trying to find a combination that works. I noticed this about 2 weeks ago that there was beginning to be problems which now are turning out to be a real bummer.

[Grey]

Enter option: 2
make gconfig

This is the first time I've seen a script complain about libraries. This is because I always choose the old-school first item (ncurses)

[Grey]

I am currently compiling 5.17.2 with a patch for Zen 2. The processor temperature has risen from 35 to 69 degrees. Otherwise, no complaints

[rockedge]

@Grey I deleted everything related to woof-CE and cloned a fresh copy.

Then merge2out to a Focal system and used that kernel-kit. Made those changes from https to git on lines 434 and 475 and it seems to be working.

I am right now compiling 5.16.14 with hugepages, PREEMPT SMP, built in overlayfs and aufs5, virtualization support. I did not enable the serial or parallel port support for this one. NVME dirvers should be built in.

[Grey]

I deleted everything related to woof-CE and cloned a fresh copy.

By the way, yes. After "intensive attempts", it is best to start from scratch, and not clean up with commands
----------|
One of my neighbors has a shamanic tambourine (from Siberia). If Github and others continue to change protocols, then I have an idea. I will borrow a shaman's tambourine and beat it, pronouncing spells
----------|

[rockedge]

I was able to successfully compile the kernel. I uploaded 5.16.14-klvx and it can be downloaded from:
https://rockedge.org/kernels

Running it in a Fossapup64 and posting this from it. I will be testing it in KLV as well.

Screenshot.png (22.17 KiB) Viewed 3517 times

[LateAdopter]

With the https: URL it seems that git clone is allowed without authentication, but git pull -all requires authentication.

To get the build to work it is necessary to delete aufs5_sources_git directory, then the script recreates it and does git clone, which works. If the directory exists the script does git pull -all which fails.

[ozsouth]

Seriously, what IS happening with the kernel-kit/aufs build system?
Today, I can only compile 5.10 64bit kernels and older 32bit.
Sorry to those waiting for my next 5.15 kernel - it isn't coming.
I'm feeling steered away from kernel-kit.

[dimkr]

kernel-kit, exactly the way it is in woof-CE, is able to build many kernels and does weekly builds of 4.19.x, 5.4.x, 5.10.x and 5.15.x at https://github.com/puppylinux-woof-CE/w ... el-kit.yml.

These builds always happen in a clean Debian/Ubuntu container and the full recipe, including the installation of dependencies, is defined in https://github.com/puppylinux-woof-CE/w ... el-kit.yml.

If kernel-kit doesn't work for you, check if you don't have all dependencies, or open an issue/pull request with a fix at https://github.com/puppylinux-woof-CE/woof-CE.

[ozsouth]

@dimkr - thanks for your detailed reply. My concern was that the old method's processing has
progressively deteriorated over the last few months, after years of relative stability.
I now understand this is Puppy's way forward, but I am comfortable with the old procedure.
As I primarily use my Chromebook, I don't have the incentive to learn new ways, so maybe
it's time I left kernel/driver making to other folks using modern methods.

[dimkr]

You'll need to accept the fact things have changed, and the old ways are ... old. For example, GitHub refuses clones over git:// now (for valid reasons), aufs has more than one branch per kernel major version, and so on. Today's kernel-kit works, and if you stick to an old copy you'll need to copy most fixes that go into the latest one and it's a waste of time.

[LateAdopter]

The authentication requirements introduced by github mean that kernel-kit behaviour will be different when run remotely compared with a github action.

When I was finding a workaround for the authentication requirement, I did try the latest version of kernel-kit but it seemed to behave the same as my current version that dates from January 2020.
Once I had a workaround, I compiled 5.17.4 with aufsv=5.17.3 and it worked OK at the first attempt with my old kernel-kit.

How are you doing it and what fails?

[ozsouth]

@dimkr - yes, I can see your points. I think enough others are proficient now, so I'll concede.

@LateAdopter - I tried various combos over the last few months & they get a fatal aufs error
after menuconfig runs. Different kernel branches were needing differing combos in successive tries. A combo
that worked for one didn't work for others, then didn't work next time tried.
2 posts up, @dimkr has largely explained that. It's progress, & futile for me to keep patching each time.
I ran my 2018 version for 3 years without problems, so this is relatively recent.

[rockedge]

Recently making real time kernels with the kernel-kit is totally hit or miss. There are so many little things going wrong I can't keep up. Now it is the firmware refuses to download because of ca-security certificates that are invalid. Ever try to fix a ca-security certificates problem in Puppy? I have not ever been able to fix this reliably with Puppy Linux.

I am right now running for the 15th (yes 15) time an attempted compile of 5.15.34-rt40. The build is failing near the end. The AUFS5 v5.15.5 patches seems to be okay and the entire full real time patches are being applied correctly.

Strange is while working on one Fossapup64 the preempt full real time selection is missing even though the patches are done and then on a different Fossapup64 same exact setup the full RT selection is available.

The kernel did compile last week but the firmware was missing and incorrectly built due to git errors or download problems so despite booting up smoothly no network devices are seen at all.

Overall summation........barely works and seems useless to make a kernel I need.

I've been able to build some pretty good kernels many times using all kinds of tactics to coerce the K-Kit in doing the right thing so no lack of trying out some crazy ideas.

Update: the current compile failed after 37 minutes

I also am considering giving up making them. The reasons I'm hearing as to why the kernel-kit is in the state it's in seem hollow. Now 95% of the traffic occurring on this forum is Bots and spammers so that's no fun anymore either.

[dimkr]

The authentication requirements introduced by github mean that kernel-kit behaviour will be different when run remotely compared with a github action.

There is no requirement for authentication. What did change, is that GitHub now doesn't allow insecure git:// clones, only https:// clone, even without authentication. aufs is on GitHub, and old kernel-kit fails because it uses git://.

The kernel-kit runs on GitHub Actions are fully reproducible locally, if your host distro isn't broken.

Recently making real time kernels with the kernel-kit is totally hit or miss. There are so many little things going wrong I can't keep up. Now it is the firmware refuses to download because of ca-security certificates that are invalid. Ever try to fix a ca-security certificates problem in Puppy? I have not ever been able to fix this reliably with Puppy Linux.

You'll need to install ca-certificates or run update-ca-certificates. Again, when the host system ain't broken, kernel-kit ain't broken. The weekly builds on GitHub Actions happen in a clean container and install all kernel-kit dependencies each time to ensure kernel-kit doesn't break, and keeps working even after other distros receive updates. The only thing the CI pipeline doesn't protect against is attempts to run kernel-kit in broken distros, which can't download files over HTTPS because CA certificates are missing, can't run git because Perl is broken, and so on.

[LateAdopter]

I can understand that many attempts is very frustrating, i would have given up after three.
The kernel-kit is just an ad hoc script to automate the process. If the patches apply without error and the compile fails it's not likely to be the kernel-kit's fault.

[rockedge]

The only thing the CI pipeline doesn't protect against is attempts to run kernel-kit in broken distros, which can't download files over HTTPS because CA certificates are missing,

I am fully aware that the builds can fail if there is a problem with the distro. It would seem then quite a few are "broken" that I have run the builds on. I have been using the kernel-kit for a pretty long time so I do know that one could fail while another will succeed.

Running a brand new Fossapup64 built with woof-CE the build fails. Other attempts on a system that has built at least 5 kernels successfully up until now so I can't say that is broken since it seems to run well otherwise.

I also have used at least 4 other systems both Bionic and Fossa to see if the distro or distro/kernel combo was causing it.
Are you saying that the Fossapup64 just built by woof-CE has broken ca-certificates? I'll try the advice again with this one.

The failure point is always the same with this build. In the build.log around line 13395:

Code: Select all

  CC [M]  drivers/hwmon/tmp421.o
  CC [M]  drivers/hwmon/via-cputemp.o
  CC [M]  drivers/hwmon/via686a.o
  CC [M]  drivers/hwmon/vt1211.o
  CC [M]  drivers/hwmon/vt8231.o
  CC [M]  drivers/hwmon/w83627ehf.o
  CC [M]  drivers/hwmon/w83l785ts.o
  CC [M]  drivers/hwmon/w83l786ng.o
  CC [M]  drivers/hwmon/wm831x-hwmon.o
  CC [M]  drivers/hwmon/wm8350-hwmon.o
  CC [M]  drivers/hwmon/xgene-hwmon.o
Error: failed to compile the kernel sources.

[Grey]

Yesterday I failed to compile 5.17.5 (mainly due to aufs).
Today I have successfully compiled 5.17.3 (with patches for Zen 2). I had to make the Nvidia driver 510.60.02 (because 510.68.02 is not friends with Conty at the moment).

[LateAdopter]

There is no requirement for authentication. What did change, is that GitHub now doesn't allow insecure git:// clones, only https:// clone, even without authentication. aufs is on GitHub, and old kernel-kit fails because it uses git://.

The current kernel-kit in the woof-ce repo fails because it does git pull -all if you have run it before on a 5 series kernel because git is already present for the aufs5 branch. I don't think that "clean" removes the .git directory

Although the removal of the git: protocol was for all of github I assume that the permissions are set per repo. When the dust settles after the security changes, I expect the way to configure the repos for security will be better understood.

I assume that git pull -all requires read access to something that doesn't currently have world read permission on sfjr's repo so it does require authentication.

[rockedge]

got a bit further into the compilation before failure after freshening the system up for the job this time.

Code: Select all

  CC [M]  drivers/comedi/drivers/amplc_dio200_common.o
  CC [M]  drivers/comedi/drivers/amplc_pc236_common.o
  CC [M]  drivers/comedi/drivers/das08.o
  LD [M]  drivers/comedi/drivers/ni_routing.o
Error: failed to compile the kernel sources.

I am now using woof-CE to build a Jammy64 (I think it's called!) and will attempt with that.

[rockedge]

Jammy woof-CE build failed using a Fossapup64 as host.

Code: Select all

ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
  CC      networking/libiproute/ll_addr.o
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Trying this in the /root/.bashrc:

Code: Select all

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0

that resulted in

Code: Select all

ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
127 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
/usr/sbin/update-ca-certificates: line 207: run-parts: command not found
done.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
No busybox in the build environment!

Will now try to build a Fossapup64

UPDATE: Fossapup64 build successful and will begin testing it out!

[dimkr]

Are you saying that the Fossapup64 just built by woof-CE has broken ca-certificates? I'll try the advice again with this one.

Fossapup64 was released in 2020, and AFAIK, uses a CA certificates bundle from 2020. This means some HTTPS downloads will fail. If GitHub now doesn't allow git:// clones and everything uses HTTPS, this can explain why kernel-kit started failing for you. Also, Fossapup64 doesn't have build dependencies of recent (> 2020) kernels, like pahole, and some of these dependencies are needed only if some kernel feature is enabled, making kernel-kit work inconsistently if you don't have these dependencies.

If you're unsure why kernel-kit fails, switch to make -j1, so it stops on the first error instead of running other jobs in parallel, pushing the error upwards in the build log.

kernel-kit changes very rarely, and most changes are tiny adjustments like https://github.com/puppylinux-woof-CE/w ... 366b75d62c, so I can tell with 99% confidence, it's not kernel-kit in woof-CE or the old kernel-kit some of you are using that changed, but the world around us.

[rockedge]

@dimkr What is happening outside of the kernel-kit is the main problem for sure.

I was able to compile 2 really nice kernels 5.16.14-KLV (no aufs5) and 5.16.14-klvx (overlayfs built in plus aufs5 patched) that both exceed expectations. These are SMP PREEMPT but not full real time. These have virtualization support and NTFS read write support built in.

I did have some firmware problems with the fdrv SFS ending up empty! I made it manually for now.

I am using mostly Fossapup64 to compile the kernels.

I am having a problem with the most recent (yesterday) Fossapup woof-CE build! Efverything seems fine BUT I get no X screen because "out of frequency" errors. Usually I can drop to console and xorgwizard my way out of this by selecting VESA driver and 1024x790 resolution but this is NOT working on this model. This is on a DELL PowerEdge R210 II server machine that has as commerical grade servers do, really bad graphics (MATOX) and I am using 20+ year old emachines monitor on top of it all.

Next step is to try it out in QEMU and VirtualBox to see about the out of frequency problem.

This has slowed me down trying the new Fossapup out with the kernel-kit (testing branch)

The full real time kernel version 5.15.34-rt40 I will run with the switch to have the errors in the log better documented. It seems so close to completing successfully

I couldn't get the Jammy woof-CE build to work but I'll go back to that soon.