Observations and questions on Persistent Data on USB

[Snail]

Running Fossapup on a USBFlash stick, I boot with Pupmode=13. I have set the automatic save interval to 0 i.e. Never. I also set the prompt to save or not on exit. I haven't been using this setup long but I often don't save on exit, especially if I have just been browsing. All this is to minimise writes to the USB of course.

As I boot, I see a message that my save folder is loaded into RAM. Why then is the persistence partition mounted after boot and set up so it can't be unmounted? It seems the opposite of what you should want for security and not needed, since the whole save folder is copied into RAM anyway?? It seems to me that it could be setup so the partition is only mounted when required for saving?

Unless the partitions for the Puppy files and persistence are mounted noatime every read is accompanied by a write, to update the access date stamp on the files being read. Is this flag being set? How do I check? (Also, there are other flags that might be relevant to this aspect, unfortunately I've forgotten what they are.).

I was a bit horrified by the size of the save folder, about half a gigabyte. Shutting down was really thrashing the USB. I only have about 7 Meg in the downloads folder. The rest will be the Fossapup updates from Quickpet, about 3 or 4 Palemoon updates in very short order and the Palemoon cache stuff. Simply deleting all the Palemoon history more than halved the size of the save folder. However, there are pages on the web giving detailed and involved instructions on how to delete the Palemoon cache. None of them match the latest Palemoon version. Does simply deleeting everything in history amount to the same thing, or do I need to do more to limit the crud?

[LateAdopter]

I set Vivaldi's disk cache to a directory under /tmp, so it disappears when I power off. Vivaldi doesn't complain about having to create it each time. I only did that because it wouldn't obey a size limit.

Type mount in a terminal to find out the current mount parameters.

[bigpup]

How the save works.

The save contains any changes that you want to keep.
Settings, configurations, added software, etc......

It is loaded when Puppy Linux boots, so that all these changes are available to use, and you do not need to do them all over again.

The save is actually a Linux file system inside a file or folder.
The save is loaded (layered into the operating file system) when Puppy Linux boots.
It does not load into RAM.
But all the parts of the file system in it, become layered into the operating file system.
To the operating file system, all looks and operates as one complete file system.
When something in the save is accessed and used.
It is read and put into ram.

The save is read/writable at all times.
Write control depends on pupmode the Puppy Linux OS is operating in.
Pupmode is determined by detecting a save to use, device Puppy is installed on, command options that can be in the boot menu entries.

These are the ones used the most.

pupmode 5 -> uses a save ramdisk (a set aside section of RAM) as the save. Auto writes to it at any time something needs to be in the save.
This is the very first time you boot Puppy Linux and have not made a save to use.
On first shutdown if you choose to make a save, the stuff in the save ramdisk is written to the save file or folder.

pupmode 12 -> When a save is found to use. Writes are done as it is needed. Do something that needs to be put in the save and it gets written to the save, at the same time.
This is used when installed on a hard drive or a drive that how often writes are done, is no big issue.

pupmode 13 ->writes are auto made at a set time, when you click on the desktop save icon, or at shutdown.
Usually used on devices that number of writes you want to limit. (USB flash drives, etc....)
This Pupmode also uses a save ramdisk, that is auto written to, any time something needs to be in the save.
Stuff in the save ramndisk only gets written to the actual save, when auto save, save icon, or shutdown process does it.
settings in Puppy Event Manager -> Save Session
Auto save can be set to any time period to do an auto save or set to 0 and do no auto save.
Ask to save at shutdown can be set, so you get asked to update the save. (only saves if you select save)
If not selected to ask, save will be auto updated at shutdown.

[bigpup]

Stuff stored by browsers is the biggest user of save space.
How to delete this stuff from the save is determined by settings to control this in the browsers settings.
Most offer a cleaning option in the preference settings.
If this cleaning has not been done before you choose to update the save.
The stuff will be in the save ramdisk and put in the save when you update the save.
Any preference settings, you have made for the browser, will need to be stored in the save.

Some other programs can produce data, that needs saving or not, depending on how you want the program to operate, the next time you run it.
that stuff needs to be written to and stored in the save.
Example:
I have Geany editor setup with specific default settings for how I want it to work.
Those settings are stored in the save.
When I start Geany, it is already configured with my settings.

In pupmode 13 operation.
Just because you change or delete something.
The save is not changed until you update it. (auto save, save icon, or shutdown save update).

[Flash]

Firefox lets you tell it where to put its cache. If you tell it to put the cache in /tmp, the cache won't be saved.

[mikeslr]

If Fossapup64 had palemoon 'builtin' --I don't recall-- and you updated and Saved, the 'new version' was written to the SaveFile/Folder. So you will have two versions occupying Storage Space; but the files in your SaveFile/Folder have priority -- will be used. As others have said, the Changes you make are not preserved unless you execute a Save. One of the reasons I use portables which I locate outside the internal Puppy File Structure.

Booting from a USB-Key, you can unpack a Palemoon portable on that Key. Then edit /usr/share/applications/Palemoon.desktop's Exec= to read something like:
Exec=/mnt/home/Palemoon64/LAUNCH
[Note the path, punctuation and capitalization must be correct; and some older versions of Palemoon were started with a script named 'plmn' rather than 'LAUNCH']*.

Restart-X [or use Menu>Update/fixmenus] so that Puppy Menu now points to the external rather than internal version.
Execute a Save to preserve it.

The external version will update without a Save being needed, the update overwriting the files in the external folder. Portables are designed to be 'self-contained': configurations, addons and web-cache are immediately written to its folder, not RAM. Remember to configure it or install addons to prevent cache from consuming the entire world.

One other thing to keep in mind. Packages (tar.gz, pets, debs etc.) are compressed so as to require less time and bandwidth to download. [Also for security]. Their installation or extraction decompresses them. Neither a SaveFile nor a SaveFolder are compressed; nor is an extracted portable's folder. Installation or extraction will result in 2 to 3 time more Storage Space being used than the size of the downloaded file.
[Applications packaged as SFSes --e.g. LibreOffice.sfs-- are and remain compressed].

-=-=--
* MikeWalsh's portables now contain a script to create a menu entry without regard to where you've located the portable. I don't recall, however, if that will overwrite the 'old' desktop files or if you will have two menu entries.

[williwaw]

Why then is the persistence partition mounted after boot and set up so it can't be unmounted? It seems the opposite of what you should want for security and not needed, since the whole save folder is copied into RAM anyway?? It seems to me that it could be setup so the partition is only mounted when required for saving?

Easy OS has some different save and mount options you may be interested in. try a search for "lockdown" in the search bar on this page https://easyos.org/

[mikeslr]

Snail wrote " Why then is the persistence partition mounted after boot and set up so it can't be unmounted? It seems the opposite of what you should want for security and not needed, since the whole save folder is copied into RAM anyway?? It seems to me that it could be setup so the partition is only mounted when required for saving?"

As others have pointed out, the contents of a SaveFile/Folder System are not copied into RAM. What happens at bootup is that (in layman's terms) an index is created in RAM as to where files are located. The files in a SaveFile/Folder are only copied into RAM as and when you need them. Of course, if you've remove the USB-Key, the index now points to a structure which is no longer present.

You can, however, accomplish what you want but not if your use a SaveFile/Folder. amethyst (formerly known as nic07) published a suite of tools, nicOS-Utility-Suite, viewtopic.php?p=12983#p12983 which includes a Save2SFS module. That module will copy the contents of your SaveFile/Folder, Applications in RAM not yet Saved, and optionally the contents of either or both a ydrv.sfs and an adrv.sfs and package those as either a new adrv.sfs or new ydrv.sfs. [Fossapup64 was published with an adrv.sfs. It is recommended that your rename that ydrv_fossapup64_9.5.sfs and create a new adrv_fossapup64_9.5.sfs. The contents of an adrv.sfs have priority over a ydrv.sfs, only lower than the contents of a SaveFile/Folder and the applications you've installed but not as yet Saved. [You select between adrv & ydrv; the module does the rest of the naming].
After the new adrv.sfs is created, move it adjacent to fossapup64_9.5.sfs and edit your boot menu to read pfix=ram.
[You can delete or move-out-of-the-way your old SaveFile/Folder. It's no longer needed; but you may want to be sure everything is working OK before deleting it. On each shutdown/reboot, you'll be asked if you want to create one. You can add another stanza to your boot menu, one pfix=ram, the other not so that you can choose to boot with or without making use of a SaveFile/Folder].

On bootup, fossapup WILL copy (and cache in RAM) the contents of the new adrv.sfs. After bootup, you can unplug the USB-Key; re-inserting it only when you need to write to it.
Note, however, adrv.sfs and ydrv.sfs are READ-ONLY files. There are other ways to 'update/change' them. But the easiest is to once again run nicOS-Utility-Suite's Save2SFS module.

[williams2]

mounted noatime ... How do I check?

Type mount in a terminal (console)

Most filesystems in Puppy are mounted with the noatime option, for exactly that reason.

[williams2]

detailed and involved instructions on how to delete the Palemoon cache.

from my firefox-start script:

Code: Select all

rm -rf /tmp/firefox
rm -rf /root/.mozilla
rm -rf /root/.cache/mozilla/

which deletes all traces of Firefox, every time I start Firefox.

/root/.cache/mozilla/ probably can be deleted anytime.
Pale Moon will have a different name in the cache folder.

/root/.mozilla contains my Firefox personal configurations.
After /root/.mozilla is deleted, the configurations that I preciously saved are unzipped from a .tar.gz file.

/tmp/firefox are the firefox files (executables, library files, etc)
After /tmp/firefox is deleted, the firefox files that I previously saved are unzipped from a .tar.gz file.
(The latest Firefox version 103.0.1)

So everytime I start firefox, it is completely deleted, and any ,malware.
and firefox is reinstalled, pristine.

[Snail]

Thanks for all the interesting and helpful comments.

From what I can gather so far, Mike's suggestion to make a new adrive sfs to hold the set-up customisation changes and installed upgrades seems to be a good idea. It would mean that all that stuff, which is currently most of the 200MB in my save folder would be loaded to RAM at boot up and would not be written back to the USB on normal shutdown. I could then operate in pupmode = 5 or 13 and decide on whether or not to save any fiddling around I'd done in the session. In addition, the contents of the new adrive being compressed will take up less space than the same data in the original save folder.

Mike also pointed out that the original adrive, now renamed as the ydrive will still contain the original software before updating. It would be nice if that could be culled , by editing the sfs or some cleverer way. Is that what a remaster does?

My save folder ballooned out to 500MB at one point. Simply using the Palemoon History/Clear Recent History/Everything and accepting the defaults reduced the save folder to "only" 200MB. I have just tried deleting the cache directory ~/.cache/moonchildproductions/palemoon/kjitwq4y.default did not save anything more than just using the standard history clearing procedure. I guess it could be set-up in a script though.

Are worries about Flash cookies (LSOs) a thing of the past. They used to appear in a .macromedia directory but Fossapup doesn't seem to have one.

[bigpup]

Mike also pointed out that the original adrive, now renamed as the ydrive will still contain the original software before updating. It would be nice if that could be culled , by editing the sfs or some cleverer way. Is that what a remaster does?

Basically a remaster is getting the Puppy version setup the way you want it.
Delete stuff, add stuff, settings configes, etc.......
Then making a new ISO of your remastered version.

You have to be very careful deleting stuff.
Some programs are used by other programs and removing them will break stuff.
Remove builtin packages program, can remove anything you want, but it can also remove stuff that will break the operating system.
Stick to removing named programs that are in the menu.

If you are ruthless in removing stuff.
You may lower the size of Fossapup64 by around 100 to 150 MB.
Programs in Puppy Linux are not very big, except for browsers.

[mikeslr]

Ditto what bigpup wrote about being cautious.
When you use Menu>Setup>Remove builtin only remove an application if you know what it does; and even then read the warning you'll receive before committing to removal about what may be broken and what files may no longer be needed.

Remove builtins doesn't actually remove anything. Remember, with the exception of your SaveFile/Folder you're working with READ-ONLY files. What remove built-ins does is write a white-out (.wh) file to your SaveFile/Folder. When you run a Remaster application and it 'sees' a white out file, it doesn't copy that file or follow links to the application 'whited-out'. Whited-out applications/files are not copied to the 'work-folder' which Remaster will compress creating your new puppy_version_#.sfs, e.g. puppy_fossapup64_9.5.sfs.

Remastering only changes the puppy_version_#.sfs. It doesn't change the contents of adrv.sfs, ydrv.sfs. It doesn't replace the vmlinuz, fdrv.sfs or zdrv.sfs. But if the remaster application you use can also create a new ISO, you usually have the opportunity to use your current versions of those files rather than the versions the Dev included in the ISO.

I wouldn't use the builtin Remaster application: (a) You have to know which --if any-- hidden files in /root and /etc to include and manually copy them to the proper location in the 'work-folder' and (b) it interrupts the work-flow while you do that.

There are two Remaster modules in the nicOS-Utility-Suite, viewtopic.php?p=12983#p12983. This post explains how they differ. viewtopic.php?p=58590#p58590

My preference is to exclude web-browsers from what amethyst refers to as the base file, i.e. puppy_version_#.sfs. In most instances I will boot using a SaveFile/Folder and web-browse using a portable not included in any 'system' files. That is I've unpacked it to, for example /mnt/home/Iron64-portable. These are easy to update or replace.

But on occasion I may want to boot from a USB-Key and unplug it. For that purpose I include a web-browser in the adrv.sfs*. Sometimes that's the only application I'll have in the adrv.sfs. The aforementioned Save2SFS module can then be used to quickly rebuilt it after an update of the web-browser*.
Recommend that you review the entire, short, nicOS-Utility-Suite for pointers on how to make the best use of its included tools.

-=-=-=-=
* Unpack the portable in /opt (optionally creating menu entries). Run Save2SFS to create an adrv.sfs which will have that setup. When you update that web-browser it writes the changes to /opt in RAM. Running Save2SFS again replaces your old /opt in the adrv.sfs with your new one.

[Snail]

Now I have a REALLY basic problem. I have no idea how to install pets downloaded from anywhere except the repositories already set up in the Puppy Package Manager (ppm). I HAVE tried searching through this forum but had no luck in finding any help. Surely this info should be in a sticky somewhere at the top of the forum?

In my past Puppy life, I never had to learn this, because the pups I used had everything I could want already set up in the ppm and, later, in Quickpet. Fossapup seems OK in Quickpet but the ppm doesn't seem to be set up at all. Only the Ubuntu repositories are selected initially. The Fossa repository is empty and the other modern Pup repositories readily accessible in the setup panel don't contain much and mostly just themes at that. The warning about setting up additional repositories takes me to a Barry page that indicates a big hassle should I attempt it. What repositories would I want anyway?

I thought of converting the Nico pets to sfs files. Pet2sfs isn't in Fossapup. Can I use Packit? If so, can it take a .pet file as input or will I have to unpack it somehow? How would I unpack the .pet if that is required? Will an sfs made from a .pet set up the menus correctly?

I would have thought that Nico's stuff was good enough to be available to every modern Puppy with minimal hassles. How could that be achieved? WoofCE? One of the repositories, e.g. noarch?

[mikeslr]

Pets will install from wherever you download them. Just Left-Click them. The same is true for 'debs' and IIRC slackware's tzx and several other packaging formats.

When you first open Puppy Package Manager it may be configured to only show Fossaupup64's own repo. [Not in Fossapup64 just now and have the memory of a sieve, but] IIRC, its panel on the left can be re-configured to show Ubuntu's repos.

I suggest that you don't get involved with the hassle of trying to add other repos. If Ubuntu's repos via PPM and/or indirectly via pkgs.org's search box don't have what you need, you're better off installing pkg-cli or mistfire's updated version. See this post, viewtopic.php?p=62371#p62371 for links and info. pkg-cli does dependency checking.

To convert a pet into an sfs you Right-Click the pet and select an extraction application such as pExtract or UExtract. The latter is my favorite as it can decompress packages in almost all formats. If not already builtin (it would be on the Utility Submenu) you can download it from here, viewtopic.php?p=3263#p3263. Just left-click to install it.
Extraction will result in a folder ending with the word 'extracted'. Within that will be a folder with just the name of the pet. Fossapup64 has both 'Create a pet' and 'Create an SFS' or 'dir2sfs' on its Right-Click menu. Just Right-Click the folder and select the appropriate choice.
In the absence of such Right-Click options AFAIK all recent Puppys have the dir2sfs and dir2pet modules built in. Just open a terminal in the parent of the folder --parent, you can see the folder as a folder-- you want to package and type the command such as:
dir2sfs NAME_OF_FOLDER or dir2pet NAME_OF_FOLDER, substituting the actual name. Spelling and capitalizations matter.
The SFS will display the same menus as the pet would have. Converting pets to SFSes is much easier than SFSes to pet. To do the latter you have to Left-Click the SFS, Select 'View Contents' and copy the contents from the window which opens into a new folder: your choice of what you name it. Right-Click that, then dir2pet and answer some questions. The one which takes some advanced planning is on what part of the Menu you want it to appear.
One other application you might find helpful in your future endeavors is PaDS, viewtopic.php?p=6355#p6355. You can place any number of pets, debs, SFSes, and several other types of packages in a folder, then Right-Click the folder and select 'Combine to SFS'. See the above thread, however, for limitations, such as your folder can't have the same name as one of the packages being included. Further discussion on the 'old forum', https://oldforum.puppylinux.com/viewtop ... b3#p658359.

[bigpup]

Installing additional software
https://forum.puppylinux.com/viewtopic.php?t=1819

[Snail]

I downloaded NicOS utilities Pet that you pointed to Mikesir and made an sfs, according to your instructions. I then loaded the sfs and clicked FixMenus. Alas, there are no icons or menu items to be seen. I was able to find the utilities by digging down into the directory ~ /nicOS-Utility-Suite-2022/usr/sbin. I can run the utility programs by left-clicking on them in the directory window. How do I fix the menus?

If I boot to ram, using the savefile, and use Nico's Classic Remaster, I believe, from reading the help files, that I will end up with a new master sfs that incorporates the adrive, as well as all the other modifications that have been made? If I boot from this remastered sfs, what will happen if there is no adrive in the boot menu? What if there is? I hope that it would be fine without the adrive, if it was absent, and ignore it if it was there. Can someone clarify please?