EXT4 error report from HDD

[greengeek]

I am a bit worried about the state of my hard drive (Hitachi sata mechanical drive) in my Toshiba S500 laptop.

During booting I see strange messages about "buddy" error on an EXT4 partition.

Here is the error message seen in dmesg:

Code: Select all

[  422.931634] EXT4-fs (sda2): error count since last fsck: 116
[  422.931638] EXT4-fs (sda2): initial error at time 1700815520: ext4_mb_generate_buddy:804
[  422.931641] EXT4-fs (sda2): last error at time 1726562013: ext4_mb_generate_buddy:744
root# 

But my HDD is actually formatted EXT2.

How can this happen?? (Using a Fossa64 9.5 derivative mostly, but sometimes Dpup32 Stretch 7.5)

Here is my blkid:

Code: Select all

root# blkid
/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
/dev/sda1: LABEL_FATBOOT="FAT32Boot" LABEL="FAT32Boot" UUID="3829-AC0B" TYPE="vfat" PARTUUID="d37c22ba-01"
/dev/sda2: LABEL="PupData" UUID="2941d455-af2b-4da3-b9cf-182678fe7eb9" TYPE="ext2" PARTUUID="d37c22ba-02"
/dev/sda3: LABEL="SWAP" UUID="5efeef16-4def-45d5-b7fa-8a5a87f9ae4b" TYPE="swap" PARTUUID="d37c22ba-03"
/dev/sdb1: UUID="4905-532E" TYPE="vfat" PARTUUID="4df4fcc8-01"
/dev/sdb2: LABEL="Ext2Data" UUID="d6b2ecb7-be69-49b4-be5f-e05bd3487b79" TYPE="ext2" PARTUUID="4df4fcc8-02"
root#

Here is my Gparted analysis:

gparted.jpg (45.63 KiB) Viewed 888 times

I'm a bit stumped.

I never have any savefile.

I have occasionally booted from a usb stick that was formatted as EXT4 - but it is not even plugged in at present.

Any thoughts how this could occur?

[ozsouth]

Ext4 driver is used for ext3 & ext2 (think it has been for about 6 years). Apparently running fsck on the drive from a USB stick should fix the corruption. Could also be failing drive. I use ext3 as it's just ext2 with a journal, so resistant to corruption & repair easier. Regarding 'buddy', I found this:
These messages are due to ext4 detecting a mismatch in the free block count between the ext4 buddy allocator bitmaps and the free block count in the group descriptor. This may be the result of corruption of the bitmaps. In the case of a mismatch, the code will update the group descriptor free block count with the calculated value from the bitmaps to ensure they match going forward. The exact cause of this issue is still under investigation in private Bugzilla, but the cause in remote storage situations appears to be hardware/firmware related in a majority of the cases.

[bigpup]

When it was setup the way you have it now.
Did you make a new partition table, partitions, and format them?

Try booting a version of Puppy from some other drive, so this problem drive is not mounted.
Do e2fsck on this drives partitions.
See what that shows.

ext2 format is the main reason the usual boot menu entries have pfix=fsck in them.
So if you are using ext2 format, this will usually find and fix errors, so Puppy will continue to boot.

Before having pfix=fsck when installed on ext2 format locations.
Puppy versions would have all kinds of boot issues, related to corruption of ext2 file system.

[greengeek]

When it was setup the way you have it now.
Did you make a new partition table, partitions, and format them?

Yes, this was a Windows laptop that I picked up from Trademe and I had no desire to keep the original software so used Gparted to write the partition table and reformat. It has been solid and reliable for a couple of years but the HDD is getting full now so I wonder if i am hitting bad sectors. I have been hastily trying to backup all the important stuff just in case...

Try booting a version of Puppy from some other drive, so this problem drive is not mounted.
Do e2fsck on this drives partitions.
See what that shows.

Ok, thanks for the tip. Will have to do this next week as I wont have access to that laptop for a few days.

ext2 format is the main reason the usual boot menu entries have pfix=fsck in them.
So if you are using ext2 format, this will usually find and fix errors, so Puppy will continue to boot.

I do have pfix=fsck in my kernel line - should that be preventing such errors? Is it possible that the fsck is actually not working? Or do you mean that the errors are being fixed and that is why I am still able to boot?

[greengeek]

Try booting a version of Puppy from some other drive, so this problem drive is not mounted.
Do e2fsck on this drives partitions.
See what that shows.

Have now run e2fsck and my drive looks pretty sick:

Code: Select all

root# e2fsck /dev/sda2
e2fsck 1.45.5 (07-Jan-2020)
PupData contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 11100879 has illegal block(s).  Clear<y>? yes
Illegal block #28684 (3634589172) in inode 11100879.  CLEARED.
Illegal block #28685 (2480852342) in inode 11100879.  CLEARED.
Illegal block #28686 (2808008132) in inode 11100879.  CLEARED.
Illegal block #28687 (2352684698) in inode 11100879.  CLEARED.
Illegal block #28688 (816575177) in inode 11100879.  CLEARED.
Illegal block #28689 (3060695347) in inode 11100879.  CLEARED.
Illegal block #28690 (3792483529) in inode 11100879.  CLEARED.
Illegal block #28691 (3821111020) in inode 11100879.  CLEARED.
Illegal block #28692 (2794641362) in inode 11100879.  CLEARED.
Illegal block #28693 (1176793989) in inode 11100879.  CLEARED.
Illegal block #28694 (2435888306) in inode 11100879.  CLEARED.
Too many illegal blocks in inode 11100879.
Clear inode<y>? yes
Restarting e2fsck from the beginning...
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'MultiFossa64_95_Trialrebuildv1.iso' in /F95ggDEV/F95MidLibre (11100694) has deleted/unused inode 11100879.  Clear<y>? yes
Pass 3: Checking directory connectivity
/lost+found not found.  Create<y>? yes
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(44403328--44403343) -(44404005--44404015) -(44404032--44404033) -(44422958--44422973) -(44436592--44436607) -(44438656--44438687) -(44438848--44439039) -(44459168--44459423) -(44499956--44499967) -(44500492--44500991) -(44567600--44568079) -(44615238--44615679) -(44617114--44617215) -(52253205--52253695) -(52258811--52258815) -(53046800--53047295) -(53099007--53099103) -(54877719--54878335) -(54889131--54889472) -(55347008--55347071) -(55348288--55348415) -(55350016--55350271) -(55351552--55351807) -(55360051--55360469) -(55365029--55365209) -(55369792--55370217) -(55372554--55372806) -(55374312--55374351) -(55374460--55374857) -(55414126--55414271) -(55417408--55417855) -(55418203--55418367) -(55418422--55418879) -(55434103--55434479) -(55441472--55441919) -(55441975--55442432) -(56310960--56311295) -(56315020--56315391) -(56323790--56324095) -(56367407--56367615) -(56444951--56446911) -(56633257--56633343) -(58615972--58618019) -(58619904--58621951) -(58622466--58624513) -(58626048--58646109)
Fix<y>? yes
Free blocks count wrong for group #1355 (0, counted=55).
Fix<y>? yes
Free blocks count wrong for group #1356 (1, counted=497).
Fix<y>? yes
Free blocks count wrong for group #1358 (14, counted=526).
Fix<y>? yes
Free blocks count wrong for group #1360 (7, counted=487).
Fix<y>? yes
Free blocks count wrong for group #1361 (0, counted=544).
Fix ('a' enables 'yes' to all) <y>? yes to all
Free blocks count wrong for group #1594 (11, counted=507).
Fix? yes

Free blocks count wrong for group #1618 (4, counted=500).
Fix? yes

Free blocks count wrong for group #1620 (0, counted=97).
Fix? yes

Free blocks count wrong for group #1674 (499, counted=1116).
Fix? yes

Free blocks count wrong for group #1675 (283, counted=625).
Fix? yes

Free blocks count wrong for group #1689 (1994, counted=4415).
Fix? yes

Free blocks count wrong for group #1691 (1996, counted=4496).
Fix? yes

Free blocks count wrong for group #1718 (1784, counted=2798).
Fix? yes

Free blocks count wrong for group #1720 (2545, counted=2754).
Fix? yes

Free blocks count wrong for group #1722 (2014, counted=3975).
Fix? yes

Free blocks count wrong for group #1728 (2209, counted=2296).
Fix? yes

Free blocks count wrong for group #1788 (0, counted=4096).
Fix? yes

Free blocks count wrong for group #1789 (1001, counted=22973).
Fix? yes

Free blocks count wrong (17615321, counted=17653716).
Fix? yes

Inode bitmap differences:  -11100879
Fix? yes

Free inodes count wrong for group #1355 (7457, counted=7458).
Fix? yes

Free inodes count wrong (15907409, counted=15907410).
Fix? yes


PupData: ***** FILE SYSTEM WAS MODIFIED *****
PupData: 124334/16031744 files (12.7% non-contiguous), 46443564/64097280 blocks
root

[dimkr]

ext2 format is the main reason the usual boot menu entries have pfix=fsck in them.

Puppy has this annoying distinction between pfix=fsck and pfix=fsckp. The former means "repair file system corruption inside Puppy save files" while the latter means "repair file system corruption in partitions mounted during the boot process". If you use a save folder or don't save at all, the former doesn't do anything. If you want to be on the safe side and enable both, use pfix=fsck,fsckp.

[bigpup]

Try using this to check the drive partitions.

Code: Select all

e2fsck -p

# e2fsck --help
e2fsck: invalid option -- '-'
Usage: e2fsck [-panyrcdfktvDFV] [-b superblock] [-B blocksize]
[-l|-L bad_blocks_file] [-C fd] [-j external_journal]
[-E extended-options] [-z undo_file] device

Emergency help:
-p Automatic repair (no questions)
-n Make no changes to the filesystem
-y Assume "yes" to all questions
-c Check for bad blocks and add them to the badblock list
-f Force checking even if filesystem is marked clean
-v Be verbose
-b superblock Use alternative superblock
-B blocksize Force blocksize when looking for superblock
-j external_journal Set location of the external journal
-l bad_blocks_file Add to badblocks list
-L bad_blocks_file Set badblocks list
-z undo_file Create an undo file

---------------------------------------------------------------------------------------------------------------------------------
@dimkr
Where are you getting the info that e2fsckp is a good command?

Is this a specific pfix= command in all the pfix= options for boot entries?

Everything I am reading says it needs to be e2fsck -p if you are going to use it to check a drive.
Well, doing this is not the same as boot menu entries using pfix= commands.

.
.

[dimkr]

@dimkr
Where are you getting the info that e2fsckp is a good command?

There's no such thing. I'm only recommending pfix=fsckp instead of pfix=fsck, which won't help OP unless the corruption is within a save file and not a partition.

[bigpup]

@dimkr

So pfix=fsckp is a boot menu entry command in the different pfix= optional commands?

This is all the pfix= commands that I know about, but this is from the Puppy Linux original boot options list.

pfix=ram Run totally in RAM ignore saved sessions
pfix=nox commandline only, do not start x
pfix=copy copy .sfs files to RAM (slower boot, faster running)
pfix=nocopy do not copy .sfs to RAM (faster boot, slower running
pfix=fsck do filesystem check on lupusave (and host partition)
pfix=clean file cleanup (simulate version upgrade)
pfix=purge more radical file cleanup (to fix broken systems)
pfix=<n> Number of save files to ignore

Do you know if this has been updated with more optional entries?
If yes.
Where to get this new list of pfix= options?

[dimkr]

So pfix=fsckp is a boot menu entry command in the different pfix= optional commands?

If I understand the question, yes.

Where to get this new list of pfix= options?

Whoever added pfix=fsckp added it to https://github.com/puppylinux-woof-CE/w ... README.txt but not to https://github.com/puppylinux-woof-CE/w ... so.sh#L230: this is probably one of the reasons why I see people recommending pfix=fsck when one actually needs pfix=fsckp. I bet many just don't know the latter exists, and we probably also have installers that only pfix=fsck instead of pfix=fsck,fsckp. To be honest, I'm not sure why one would want pfix=fsck or pfix=fsckp instead of pfix=fsck,fsckp: if you trust e2fsck, let it repair everything, otherwise don't let it touch any of your file systems.

[Marv]

I@dimkr I think this is an appropriate place to ask this. In the init script (using Noblepup64 as an example, fsckp is limited to EXT file partitions, ie line 128 is #vfat) fsck_app='fsck.fat -y' ;;. Is there a compelling reason for this? In most of my personal pups, I don't use a save file or folder so the boot partition isn't mounted and I can easily check it on boot. For my users, I usually add a small savefile customizing the look'n'feel for them. That way I can use a standard y (or a) drive for all. Thus in those systems the boot partition is mounted early on and the init would seem to be a logical place to do a fs check.

Thanks in advance,

[dimkr]

I@dimkr I think this is an appropriate place to ask this. In the init script (using Noblepup64 as an example, fsckp is limited to EXT file partitions, ie line 128 is #vfat) fsck_app='fsck.fat -y' ;;. Is there a compelling reason for this?

I have absolutely no idea. The people who wrote this script made many unexplained changes throughout the years, like commenting out this line, and they don't maintain it anymore.

[Marv]

Thanks, I'm running with it (and the exfat line) uncommented in my most rebooted pup and may carry that change in my ydrive/adrives. I've logged some diagnostics from the change and so far I see no problems with it in my installs of NoblePup64, LxPupSc64, or fossapup64-small. My data drives, mostly EXT4 now -finally-, get checked before mounting in rc.local.

[bigpup]

Fsck can not work on any possible formats.

It is for ext 2, 3, or 4 formats (file systems).

Here is some info about fsck
https://docs.redhat.com/en/documentatio ... k-ext2-3-4

With some info about a few other type formats and programs to check them.

For ext4 formats there is now fsck.ext4 which is suppose to be more checking of this format.

Do not confuse this with what the Puppy boot commands pfix=fsck do.
pfix= commands are specific code for the Puppy boot process to perform.

[dimkr]

Do not confuse this with what the Puppy boot commands pfix=fsck do.

This could become this. I don't think the (short) list of supported file systems is documented anywhere.

Sadly, it looks like nobody cares about this init script anymore and nobody maintains it. It has various arbitrary limitations beyond this arbitrary decision not to repair file system errors on corruption-prone file systems like FAT32 and exFAT: for example, KFILESYSTEMS and FSCKLIST could be removed. In addition, https://github.com/puppylinux-woof-CE/initrd_progs is very very very outdated, so I wonder how reliable Puppy pfix=fsckp really is.

[Marv]

Fsck can not work on any possible formats.

It is for ext 2, 3, or 4 formats (file systems).

Do not confuse this with what the Puppy boot commands pfix=fsck do.
pfix= commands are specific code for the Puppy boot process to perform.

The line in question called fsck.fat which indeed does check vfat partitions. fsck.exfat does the same for exfat partitions.
Not confused, see the earlier posts in this thread for pfix=fsck vs pfix=fsckp.

[greengeek]

I'm only recommending pfix=fsckp instead of pfix=fsck, which won't help OP unless the corruption is within a save file and not a partition.

Thank you for this critical information - all these years I have been using fsck when I had no save file. Doh!!
Very pleased that fsckp sorted out my damaged disk. Not sure what exactly was damaged (partition table? Directory list? Buddy list? File allocate table? Who knows what?) - but at least something seems better (in the sense of no more error messages..)

Mind you - seeing the error messages (also visible during the boot phase) gave me an opportunity to copy off my most important files.
I wonder if I would have seen the errors if the fsckp had been properly set up and running normally over the last few months (during which time my hdd was probably starting to degrade).
Does anyone know if fsck or fsckp do any highlighting of errors that are found? Any logs we should be checking regularly??
Or do these utilities just quietly "fix" the problems?

I really do need to get my auto backups / cloning automated.

[dimkr]

Does anyone know if fsck or fsckp do any highlighting of errors that are found? Any logs we should be checking regularly??

They should be logged to /initrd/tmp/bootinit.log, but I'm not sure if this requires loglevel=7 or the default verbosity (equivalent to loglevel=3) is sufficient.

Generally speaking, corruption is less likely if you have pfix=fsckp and reboot every once in a while: most errors are not fatal and problems begin when the file system accumulates many corruptions. In addition, regular fsck is much faster because it has fewer (or zero) errors to fix each time. For the last couple of months, I've been using ext4 without journalling (= more prone to corruption compared to "normal" ext4) on a really old USB 2 flash drive (from <=2015, maybe 2010) and pfix=fsckp that fixes errors at least once a week; I still haven't lost any files.

Or do these utilities just quietly "fix" the problems?

Puppy runs them in "assume 'yes'" mode - they automatically proceed to fix all errors they find without you having to confirm first.

[Marv]

In the same vein, I'd wager many of us have a largish partition on the primary SSD (or second SSD) that has data, portables, shared profiles, etc. and is automatically mounted at boot. I still have a few spinners around but none in daily use. On SSDs I generally don't run journalling in EXT4 and like to error check that disk or partition at mount.

In all the pups I know, that disk or partition is mounted by a line in /etc/rc.d/rc.local, usually placed there by Pmount like:
mkdir -p /mnt/sda2; mount -t ext4 /dev/sda2 /mnt/sda2 #pmount.
Making that line:
e2fsck -p /dev/sda2; mkdir -p /mnt/sda2; mount -t ext4 /dev/sda2 /mnt/sda2 #pmount
adds error checking (-p or -y, your choice).

Simple and maybe obvious but I think useful.