Bookworm stopped working sends a cryptic message: "device full"

[williams2]

My OS, since I installed Earlyoom, is rock solid.
To be fair, it was rock solid before.

I don't think earlyoom will help if the tmpfs fills up.
As long as there is enough free ram,
it doesn't care if the tmpfs is out of space.

earlyoom can keep a logfile showing what it kills and when and why.

If you can delete enough files to get enough space, you should be able to shutdown gracefully.

I still wonder if clicking the save button flushes the tmpfs or not
or just copies to the save file/folder.
Just run this (sizes of dirs in tmpfs)

du -h /initrd/mnt/tmpfs/ | sort -hk1

or this: (sizes of files and dirs in tmpfs)

du -ah /initrd/mnt/tmpfs/ | sort -hk1

Run before and after clicking Save

If the Save button deletes most of the files in tmpfs,
that gives you more working space.
But I suspect it just copies to the save file
and does not clean the tmpfs at all.

[dimkr]

I still wonder if clicking the save button flushes the tmpfs or not
or just copies to the save file/folder.

Copies.

The aufs implementation of snapmergepuppy moves files (will truncate and rewrite a file if you save after flipping one bit), so if you're writing a file while it's saving, sometimes it 'saves' the previous contents or something in between. Binary files like browser cache (most browsers write sqlite databases) are very easy to corrupt in a Puppy with aufs under PUPMODE 13, unless you only save when the browser isn't running. In addition, sometimes the browser may delete a file while snapmergepuppy is busy saving other files, leaving both files in the save when only one should exist. The saving procedure is full of race conditions.

The overlay implementation in woof-CE (I wrote it) copies because 1. overlay forbids direct changes to lower layers, this messes up cache and can easily lead to data loss and 2. you can retry saving when the file is still in RAM. With overlay, ramdisk space consumption increases until you delete new or changed files (because they have a copy in the ramdisk) or reboot.

[Governor]

I still wonder if clicking the save button flushes the tmpfs or not
or just copies to the save file/folder.

Copies.

The aufs implementation of snapmergepuppy moves files (will truncate and rewrite a file if you save after flipping one bit), so if you're writing a file while it's saving, sometimes it 'saves' the previous contents or something in between. Binary files like browser cache (most browsers write sqlite databases) are very easy to corrupt in a Puppy with aufs under PUPMODE 13, unless you only save when the browser isn't running. In addition, sometimes the browser may delete a file while snapmergepuppy is busy saving other files, leaving both files in the save when only one should exist. The saving procedure is full of race conditions.

The overlay implementation in woof-CE (I wrote it) copies because 1. overlay forbids direct changes to lower layers, this messes up cache and can easily lead to data loss and 2. you can retry saving when the file is still in RAM. With overlay, ramdisk space consumption increases until you delete new or changed files (because they have a copy in the ramdisk) or reboot.

This is critical info. I think this is an issue that ought to be addressed in one way or another. IMO, there ought to be safeguards in place so that a user does not experience what I did, ie. I could basically do nothing and had to use the power button to turn off the computer. This type of experience will drive people away from Puppy never to return. Does anyone care to guess how many people this has happened to? I don't. My simple suggestion is a warning that appears on the screen (in a box which can be closed again) advising the user to close browser(s) all applications before saving. This warning could appear whenever the user clicks on the save button and there is a browser are applications running. Or, even just the warning about possible data corruption without checking if there are apps running.
Thanks!

[Governor]

In my case I have a dedicated system drive partition with a dedicated sessions folder. During boot similar to @Governor I get it's list of save folders where I select one.

While using the system, over time, the cache expanded to fill the 30GB partition. AND ultimately the save expanded to fill the partition.

My PMEDIA=atahd at boot.

I assume responsibility as I have NOT moved the browser cache out of save session pathing. (several weeks of running on this test PC without rebooting)

After cache cleaning, the folder in use reduced back down to approx 600MB. 1st time lon this distro on this PC.

This is the example experience I have had.

I use save folders. Does this still apply?
If so, can someone please explain how to permanently move the browser cache out of the system folder.
What is the difference between absolute and relative links?
Thanks!

I noticed that my 'Lucas Chess' desktop icon has this path: /root/.local/share/applications/LucasChessR.desktop
It is a definite problem for me when applications automatically install themselves to a pre-determined location. This is poor/lazy programming, IMO.

Now that I think about it, this current problem with Bookworm may be related to this 'unauthorized' installation of Lucas in my root folder.
I see that /root/LucasChessR is where the chess program is, but I did not put it there! I moved the entire LucasChessR folder and I copied the .desktop file and changed the contents to reflect the new location.

I changed the cache location in Firefox about:config to /mnt/nvme0n1p4/Firefox_cache.
I hope this helps.

[Governor]

The writable save layer is definitely the ram tmpfs.

The error message is "out of space"
not "read only FS"

The tmpfs is using 354M, which is quite reasonable.

Firefox Portable is unzipped in ram,
whether it's running or not.

Firefox is taking about 283M, most of what is in tmpfs.

I don't know where Bookworm puts the sfs files.
They are not in tmpfs, unless you booted nocopy.

This is similar to viewtopic.php?t=11819
who also uses the same mode (pupmode 13 ?)

I'm not sure pupmode 13 is always stable.

The data shows it running normally.
Until no space left.

If a file is deleted while it is in use,
the file name will be deleted,
but the file will still be taking up space,
it will just be invisible.
Such an orphaned inode is usually fixed by rebooting and doing a fsck.

I am booting nocopy because I was trying not to run out of RAM. Which is the worse risk, running out of RAM out running out of disk space?
If I boot with copy and use a swap disk, would the swap prevent running out of RAM?
I will change the boot procedure back to copy and see if there is any noticeable improvement, although I know problems usually come suddenly.

[Governor]

The writable save layer is definitely the ram tmpfs.

The error message is "out of space"
not "read only FS"

The tmpfs is using 354M, which is quite reasonable.

Firefox Portable is unzipped in ram,
whether it's running or not.

How can that happen? FF is on a different partition from the OS, and I don't see how it can do anything unless I start it up.

Firefox is taking about 283M, most of what is in tmpfs.

I don't know where Bookworm puts the sfs files.
They are not in tmpfs, unless you booted nocopy.

This is similar to viewtopic.php?t=11819
who also uses the same mode (pupmode 13 ?)

I'm not sure pupmode 13 is always stable.

The data shows it running normally.
Until no space left.

If a file is deleted while it is in use,
the file name will be deleted,
but the file will still be taking up space,
it will just be invisible.
Such an orphaned inode is usually fixed by rebooting and doing a fsck.

I have been informed that fsck is dangerous.
viewtopic.php?p=125262#p125262

[fredx181]

The writable save layer is definitely the ram tmpfs.

The error message is "out of space"
not "read only FS"

The tmpfs is using 354M, which is quite reasonable.

Firefox Portable is unzipped in ram,
whether it's running or not.

How can that happen? FF is on a different partition from the OS, and I don't see how it can do anything unless I start it up.

From your output of du -h /initrd/mnt/tmpfs/ | sort -hk1 here: viewtopic.php?p=127437#p127437
It shows FirefoxESR-portable64 (included in BWP) in tmpfs/pup_rw/opt, it's NOT from a different partition, it's from the OS itself:

Code: Select all

/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/firefox64/updated/updates
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/firefox64/updated/updates/downloading
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/firefox64/updates/downloading
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/profile/cache2/doomed
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/profile/datareporting/glean/tmp

[Governor]

The writable save layer is definitely the ram tmpfs.

The error message is "out of space"
not "read only FS"

The tmpfs is using 354M, which is quite reasonable.

Firefox Portable is unzipped in ram,
whether it's running or not.

How can that happen? FF is on a different partition from the OS, and I don't see how it can do anything unless I start it up.

From your output of du -h /initrd/mnt/tmpfs/ | sort -hk1 here: viewtopic.php?p=127437#p127437
It shows FirefoxESR-portable64 (included in BWP) in tmpfs/pup_rw/opt, it's NOT from a different partition, it's from the OS itself:

Code: Select all

/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/firefox64/updated/updates
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/firefox64/updated/updates/downloading
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/firefox64/updates/downloading
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/profile/cache2/doomed
/initrd/mnt/tmpfs/pup_rw/opt/FirefoxESR-portable64/profile/datareporting/glean/tmp

I do not understand it. How do I stop this? Rox file manager is not showing it. Ok, I found the .mozilla folder and deleted it. How can I determine if there are other folders that should be deleted in tmpfs?

EDIT: After reboot, .mozilla is still gone. So that is one thing fixed.
Thanks.

[Governor]

In my case I have a dedicated system drive partition with a dedicated sessions folder. During boot similar to @Governor I get it's list of save folders where I select one.

While using the system, over time, the cache expanded to fill the 30GB partition. AND ultimately the save expanded to fill the partition.

My PMEDIA=atahd at boot.

I assume responsibility as I have NOT moved the browser cache out of save session pathing. (several weeks of running on this test PC without rebooting)

After cache cleaning, the folder in use reduced back down to approx 600MB. 1st time lon this distro on this PC.

This is the example experience I have had.

I have switched Bookworm to pmedia=atahd which gives me PUPMODE=12.
I am hoping there are fewer problems than with pmedia=ataflash. Is there a resource available which explains the difference?
Thanks.

[Clarity]

Theoretically "...flash" intends to reduce writing of the save session changes until the PC is requested to shutdown/reboot. '...hd' can write changes more frequently without waiting until the end.

The rationale is that flash are memory devices which have a limited write-cycle lifespan, while system drives (spinning disks) have much-much longer read-write lifespans.

[dimkr]

The best resource I know is https://bkhome.org/archive/puppylinux/d ... works.html, many things are no longer true (like the use of unionfs rather than aufs or overlay) but the general idea is still the same.

I've been working on optimizing PUPMODE 13 lately (see viewtopic.php?p=122662&hilit=sqlite&sid ... 3b#p122662) and make it more reliable, especially when running out of space or on sudden shutdown. I think that many assumptions made when PUPMODE 13 was introduced are no longer correct, and maybe they were incorrect even then. For example, if the browser history is a binary file that mostly grows (extra data is appended to the end of the file, while the beginning doesn't change or doesn't change much) and you write the whole file to the flash drive every time you save (instead of just appending extra data), maybe PUPMODE 13 actually increases writing to the flash drive instead of reducing it. In addition, the way it copies files today is very dangerous - cp truncates the file, then writes the contents, so save2flash doesn't save the file atomically: if power is lost (battery dies, for example) after the file is opened but before the contents are flushed to disk, you're left with an empty file and not with the previous contents of the file.

The optimized implementation I wrote only writes modified blocks to the flash drive (skips unchanged blocks) and grows or shrinks files before it starts copying changed blocks to ensure there's enough space, among other improvements. Together, these improvements can reduce writing by a factor of over 20 (according to iostat) under typical use and sometimes reduce the time it takes to save from minutes to 5 seconds or less, if most changed files haven't changed much and most changes are extra data appended to the end of the file.

(My work happens in https://github.com/vanilla-dpup/woof-CE but I backported these improvements to my 10.0.x dpup releases as well)