Can a Save Folder Be Cleaned Up of Deletions?

[JusGellin]

I understand that as a puppy system (in this case PuppybookwormPup_10.0.6) gets older and the save folder is used, that the save folder will continue to grow in size.
Even if deletions occur, the folder won't get smaller but will be white marked only. It will keep those deletions.

I made a test of measuring my savefolder size to begin for the last save:
1.4G

Then I made a directory named TEST at the root
/TEST
I put a couple of large iso files in it and then did a save to folder.
Now the savefolder size is:
3.0G

Next I deleted /TEST and again did a save to folder which still had the following size as expected:
3.0G

So is there any way to clean up the savefolder when deletions occur like this?

I'm usually careful to watch that I don't do this to make the save folder increase in size.
But I want to see if there is any way if this happens and how to clean that up.
Thanks

[bigpup]

Are you running in pupmode 13, where you have to click on save icon on desktop, to update what is in the actual save folder?

I just tried it running in pupmode 12, where the save is directly written to, as stuff is done.

Using a save folder.
I saw a change in size from adding a big file and then deleting it.

to check this, navigate in Rox where the save folder is located.

right click on it.
Select properties.
Get the size reading here.

[bigpup]

the whitelist thing, is only on builtin programs and files, that are originally in a Puppy versions different sfs files.
The sfs files are read only.
If you delete or remove something in these sfs's, it is only given a .wh at the end of it's name, not really deleted.
To the operating file system they are no longer seen.
This is what Remove Builtin packages program does.

If you did a remaster of the Puppy as you have it setup.
The remaster program would build the new SFS files with none of the items whitelisted in them.
So, in the remaster the items would not be in it's SFS files.

[bigpup]

All file systems when you completely delete something (like Rox does). The data is still there, but the information for it is deleted from the file allocation table.
The locations the data is in is now marked as usable space.
But the space still has the deleted stuff in it.
The space can now be overwritten with new stuff.

Because of this way of deleting data.
A disc recover program can have a good chance of recovering data from a drive, as long as no writing to the drive has been done.

the data stored on a drive is just a bunch of ones and zeros.

Deleting would take forever to complete if it had to completely change all those ones and zeros.

Usually to complete totally delete the data and get to a fully clean drive.
You write all ones or all zeros to every location on the drive.

[JusGellin]

@bigpup
Thanks

My o' My! I thought I understood this better.
I need Pup101 training.

I think I see why the save folder didn't decrease in size like I thought it should when I deleted the /TEST folder with 2 isos in it.

This time this is what I did:
1. Make folder /TEST containing 2 iso files
2. Did a save to the savefolder
3. Rebooted
4. /TEST came back up as expected
5. Deleted this folder and its contents
6 Did a save to savefolder
7. Rebooted
8. Expected to not have /TEST - but it was still there.

I thought that since I made this folder, that when I deleted it and saved to savefolder, it would be gone, but it wasn't.

Am I misunderstanding how this works and this is normal behavior?
If folders and files are created and saved to savefolder they can't be deleted?

Please help me understand this better.
Thanks

[fredx181]

..
I think I see why the save folder didn't decrease in size like I thought it should when I deleted the /TEST folder with 2 isos in it.

This time this is what I did:
1. Make folder /TEST containing 2 iso files
2. Did a save to the savefolder
3. Rebooted
4. /TEST came back up as expected
5. Deleted this folder and its contents
6 Did a save to savefolder
7. Rebooted
8. Expected to not have /TEST - but it was still there.

I thought that since I made this folder, that when I deleted it and saved to savefolder, it would be gone, but it wasn't.

Am I misunderstanding how this works and this is normal behavior?
..

No, not normal, it should be gone after you deleted it and rebooted, I think it's a bug (I tested same as you and indeed /TEST is still there despite having it deleted before rebooting).
Btw, If you'd do the same in a sub-directory of / , e.g. /root/TEST or /etc/TEST it works as it should be (gone after reboot when deleted) from what I tested.

[JusGellin]

@fredx181
Great! Thanks for verfying this.
I was beginning to doubt that I'm not learning anything.
But now I can go on to understand more.

Thanks again

[JusGellin]

@fredx181
I tried my testing in a subfolder (/root/TEST) and the savefolder worked like I thought it should.
Adding the /root/TEST with some isos and saving to savefolder caused the savefolder to increase in size.
Removing /rootTEST and saving to savefolder caused the savefolder to go back to the original size.

That's how I thought it was supposed to work.

This topic title was from my observation for what was happening.

Thanks

[JusGellin]

the whitelist thing, is only on builtin programs and files, that are originally in a Puppy versions different sfs files.
The sfs files are read only.
If you delete or remove something in these sfs's, it is only given a .wh at the end of it's name, not really deleted.
To the operating file system they are no longer seen.
This is what Remove Builtin packages program does.

If you did a remaster of the Puppy as you have it setup.
The remaster program would build the new SFS files with none of the items whitelisted in them.
So, in the remaster the items would not be in it's SFS files.

Thanks for this refresher for me about the whitelist.
Because I was seeing the savefolder increase in size and stay there when I added and removed from the root directory (/),
I was trying for an explanation for what was happening and thought the whitelist was causing this.

But @fredx181 trying my same test indicates that this shouldn't happen so I don't need that wrong reason anymore.

Thanks

[Chelsea80]

@bigpup

Very interesting topic, however it has left me a bit confused so excuse my lack of knowledge.

My highlight in bold

the whitelist thing, is only on builtin programs and files, that are originally in a Puppy versions different sfs files.
The sfs files are read only.
If you delete or remove something in these sfs's, it is only given a .wh at the end of it's name, not really deleted.
To the operating file system they are no longer seen.
This is what Remove Builtin packages program does.

If these SFS files are read only then how can something be deleted or removed ?

[fredx181]

@JusGellin and everyone.
So the rule is (bug or not ): Don't create anything directly at / because you never get rid of it ! (well without using dirty tricks)
(not sure though if it's only a problem with e.g. pupmode=13 (save on demand or at shutdown) could work ok with real-time saving, didn't test that)

[wiak]

Sounds like a serious bug to me. Presumably rsync is being used to update the save folder. If you delete anything during a running session in save on demand mode, it should be deleted on the target when saving back the result surely? The issue could be that the session RAM copy may be simply getting a whiteout and that is simply being rsync'd back when surely that whiteout should in this case trigger an actual deletion in snapmergepuppy routine?

I confess that I am on android phone and havent looked into it.

[bigpup]

@bigpup

Very interesting topic, however it has left me a bit confused so excuse my lack of knowledge.

My highlight in bold

the whitelist thing, is only on builtin programs and files, that are originally in a Puppy versions different sfs files.
The sfs files are read only.
If you delete or remove something in these sfs's, it is only given a .wh at the end of it's name, not really deleted.
To the operating file system they are no longer seen.
This is what Remove Builtin packages program does.

If these SFS files are read only then how can something be deleted or removed ?

This is stuff built in that comes already in the Puppy version.

To delete something that is in one of the Puppy versions SFS files.

First you whitelist it with remove builtin packages program. the change is actually stored in the save.

Then do a remaster of the SFS file and what is in it.

The remaster process will see the stuff that is in the different SFS files named with .wh.
Not include these in the remastered version of the SFS files.
So it builds a remastered version of what is now built in.

the main reason to have these SFS files, that make up the complete Puppy version, read only.
So changes of any kind are only in the save file/folder.
The original Puppy OS is still completely unchanged, if you do not use a save file/folder.

[dimkr]

surely that whiteout should in this case trigger an actual deletion in snapmergepuppy routine?

https://github.com/puppylinux-woof-CE/w ... erlay#L104

[ozsouth]

So the rule is (bug or not ): Don't create anything directly at / because you never get rid of it ! (well without using dirty tricks)
(not sure though if it's only a problem with e.g. pupmode=13 (save on demand or at shutdown) could work ok with real-time saving, didn't test that)

I have found that anything in the main / folder in pupmode 5 (boot without savefile/folder) does not save if you create a savefile/folder on first reboot; but once a savefile/folder is made, on reboot (pupmode 12), additions/deletions from / are saved thereafter.

[wiak]

surely that whiteout should in this case trigger an actual deletion in snapmergepuppy routine?

https://github.com/puppylinux-woof-CE/w ... erlay#L104

This thread being about Puppy Linux mainline distros and not the likes of Kennel Linux, I won't say much about the latter, which uses completely different save2flash/snapmergepuppy code written by fredx181.

What I will say, for reference, is that the problem does not seem to exist in KL. There is a different relatively minor issue. When item is deleted from /, and a save2flash made, a whiteout is left behind and appears in the on-media savefolder. With overlayfs, the whiteout being just a char device that takes up virtually no space, I can't say it is much of a problem, and perhaps it is correct in case since if a user wanted a folder like /myjunk deleted that would include if underlying sfs addons contained such a directory.

I also tried creating a folder /myjunk/filea on Puppy Bookworm, saved to flash, then deleted it and saved to flash again, and I can confirm the deletion did not work (but works fine in KL distros albeit leaving a zero sized whiteout in the save folder). I then took brief look at the Puppy github woof-CE code, but unfortunately I don't know Puppy organisation though I've since worked out what /initrd/pup_rw/ and /init/pup_ro1/ are used for having never known before. However, since I didn't write the snapmergepuppy code for KL/FR distros I don't off the top of my head know the ins and outs of it and not being familiar with that Puppy implementation can't quickly work out what is missing. I expect fredx181 could figure it out much more quickly - probably some sed or find statement missing something, but just a guess.

Without having time to analyse the code, I did nevertheless wonder if line 104 or maybe line 127 for removing directories (I really don't know): rm -rf "$BASE/$N" is where the deletion should be getting done since I also noted that $BASE (albeit a symlink to actual location) is: BASE="/initrd/pup_ro1", which I understand is the media save location (?) and pup_rw the session save in RAM (?). Anyway, even if that rm -rf line is the one that does the removal prior to saving back to media, it clearly doesn't work for the items stored in / in Puppy. Hopefully someone will fix that since others will likely end up at times with much larger save folders than they expected following deletions.

[wiak]

I've now booted into BookwormPup itself and was surprised to find the snapmergepuppy code there is a bit different to that in previously linked github variant.

https://github.com/puppylinux-woof-CE/w ... erlay#L104

Perhaps the github one works fine (I have no time to test it)? If so it would not be good to not have made sure the advertised most current Puppy did not have the corrected code inside it... I assume the github code has the same fault therefore... or otherwise would be ridiculous situation.

EDIT: couldn't resist quickly copying in the github code to Bookwormpup snapmergepuppy, prior to heading off to sleep, and seems to have same delete not working on items stored in / issue indeed.

[Clarity]

Does this problem cascade into a new 'remaster'?

[dimkr]

I've now booted into BookwormPup itself and was surprised to find the snapmergepuppy code there is a bit different to that in previously linked github variant.

The most likely explanation is that you compared snapmeregpuppy a apmergepuppy.overlay by mistake, they're very different.

[wiak]

I've now booted into BookwormPup itself and was surprised to find the snapmergepuppy code there is a bit different to that in previously linked github variant.

The most likely explanation is that you compared snapmeregpuppy a apmergepuppy.overlay by mistake, they're very different.

Maybe, was tired late at night, but I thought Bookwormpup64 also used overlay(?)

[dimkr]

Maybe, was tired late at night, but I thought Bookwormpup64 also used overlay(?)

It does, but every recent Puppy has both snapmergepuppy and snapmergepuppy.overlay, even if the default or the only supported option is overlay.