Does PupMode 12 have any advantage over PupMode 13?

[bigpup]

I repeat again.
You are all making specific statements, about how this all works, and at no time are you talking about a specific Puppy version.
This has all been tweaked over the years and it does not exactly work, the exact same way, in all Puppy versions.

Do not try to think, one of the Puppy like OS's on this forum, work exactly like Puppy does.

Example:
SFS package loading and unloading
(these are added program packages, not the Puppy OS SFS files)
Depending on the Puppy version.
The SFS packages, may have to be located in /mnt/home, to be seen to load or unload.
Located any other place and they will not be seen to load.

Even if the Puppy version can see SFS packages in other locations.
/mnt/home is still the best location, because it is not inside the save.
The main idea of loading and unloading SFS packages.
So a program can be used and not have to be installed, which would place it in the save.

[wiak]

A little demonstration:

creating a 15GiB file each byte being a zero (15 billion zero bytes)
dd if=/dev/zero of=zero.bin bs=1M count=15360

making a squash fs containing a 15G file:
mksquashfs 000tmp/ zero.sfs

Code: Select all

# ll 000tmp/bin/zero.bin zero.sfs 
-rwxr-xr-x 1  15G May 15 23:32 000tmp/bin/zero.bin
-rwxr-xr-x 1 8.0K May 15 23:38 zero.sfs

free before and after mounting the zero.sfs file:

Code: Select all

# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        162M        2.4G        824M        1.7M        904M        2.3G
Swap:            0B          0B          0B
# 
# mount-FULL -o loop /mnt/home/zero.sfs /mnt/tmp/
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        163M        2.4G        824M        5.7M        908M        2.2G
Swap:            0B          0B          0B
# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        163M        2.4G        824M        1.7M        904M        2.3G
Swap:            0B          0B          0B
# 

free before and after unmounting zero.sfs:

Code: Select all

# du -ah /mnt/tmp/
15G	/mnt/tmp/bin/zero.bin
15G	/mnt/tmp/bin
15G	/mnt/tmp/
# 
# umount-FULL /mnt/tmp/
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        172M        2.4G        824M         10M        912M        2.2G
Swap:            0B          0B          0B
# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        172M        2.4G        824M        1.8M        904M        2.2G
Swap:            0B          0B          0B
# 

rename zero.sfs to ydrv so it will automatically be copied to ram and mounted as an aufs layer:

Code: Select all

# cd /mnt/home/
# mv -i zero.sfs ydrv_bionicpup64_8.0.sfs
# wmreboot

Very interesting overall thread this one. Understanding how layers work and the different methods of achieving copy2ram and save persistence effects via mounting and/or copying into tmpfs followed by cp or rsync back on exit is fundamental to how live systems work, and the explanations of @william2 are spot on generally.

I'm maybe looking at the wrong thing, but something looks a bit odd above though. When you do umount-FULL /mnt/tmp the RAM used according to free seems to go up... to 172M. If any diffference I'd expect RAM used to go down a little bit. Previous non-mounted value of used looks like it was 163M.

I'll do some tests of my own - seems unlikely that mounting an sfs would take extra 10M as amethyst suggests (tho I've never measured it); I believe tinycorelinux, for example though, uses sfs files (tcz extension) and mounts tons of these prior to linking them in to the overall filesystem (tinycorelinux doesn't use a unionfs). Would be expensive if 10M per sfs mount.

Copy On Write (COW) and using layers and whiteouts and so on is not unique to Puppy of course, but still fascinating to understand how the many wee live systems available work. Credit for most of these (actually ancient) 'tricks' actually goes to Sun Microsystems as far as I can determine it (Knoppix, Puppy, and various others of course were also 'early-ish' pioneers in small Linux live system use - but 2004/2005 so comparatively pretty recent really!...): "Sun Microsystems introduced the first implementation of a stacked, layered file system with copy-on-write, whiteouts (hiding files in lower layers from higher layers), etc. as the Translucent File Service in SunOS 3, circa 1986. (wikipedia)". BSD has used Union mounts since at least 1995 (also from Wikipedia UnionFS article), so I suppose we shouldn't get too excited about the way it works now, but the technique(s) remain quite fascinating. I've enjoyed this kind of layered system manipulation since contributing to User Mode Linux project back in 2001, which though different, also employs similar concepts (COW files and so on).

[amethyst]

I repeat again.
You are all making specific statements, about how this all works, and at no time are you talking about a specific Puppy version.
This has all been tweaked over the years and it does not exactly work, the exact same way, in all Puppy versions.

Do not try to think, one of the Puppy like OS's on this forum, work exactly like Puppy does.

Example:
SFS package loading and unloading
(these are added program packages, not the Puppy OS SFS files)
Depending on the Puppy version.
The SFS packages, may have to be located in /mnt/home, to be seen to load or unload.
Located any other place and they will not be seen to load.

Even if the Puppy version can see SFS packages in other locations.
/mnt/home is still the best location, because it is not inside the save.
The main idea of loading and unloading SFS packages.
So a program can be used and not have to be installed, which would place it in the save.

You can load an extra sfs manually from any location on-the-fly during a session (with the sfs_load utility). Which versions of official Puppy do not allow that? You can also install a pet "temporarily", just make sure your session is not saved (or uninstall it before your session is saved). In some cases it may be useful to install a pet temporarily rather than loading an extra sfs.

[JASpup]

I'm maybe looking at the wrong thing, but something looks a bit odd above though. When you do umount-FULL /mnt/tmp the RAM used according to free seems to go up... to 172M. If any diffference I'd expect RAM used to go down a little bit. Previous non-mounted value of used looks like it was 163M.

I noticed that too. Should one worry about accountability on the negative action side?

I.e., 'Oh, unmounting increases ram usage, but only a little bit.'

[bigpup]

You can load an extra sfs manually from any location on-the-fly during a session (with the sfs_load utility). Which versions of official Puppy do not allow that?

It depends on version being used and when the changes got into the sfs-load.
Someplace in this topic, is when changes were made, and new version was released.
https://oldforum.puppylinux.com/viewtop ... 58&t=64354
I would say around the time of Tahrpup release and Slacko 6.3.2, so they may or may not have the updates to sfs-load.
Any older Puppies probably not.

You can also install a pet "temporarily",

Not if you are running in pupmode 12 or pupmode 2.
that one does apply to all Puppy versions.

[amethyst]

You can load an extra sfs manually from any location on-the-fly during a session (with the sfs_load utility). Which versions of official Puppy do not allow that?

It depends on version being used and when the changes got into the sfs-load.
Someplace in this topic, is when changes were made, and new version was released.
https://oldforum.puppylinux.com/viewtop ... 58&t=64354
I would say around the time of Tahrpup release and Slacko 6.3.2, so they may or may not have the updates to sfs-load.
Any older Puppies probably not.

You can also install a pet "temporarily",

Not if you are running in pupmode 12 or pupmode 2.
that one does apply to all Puppy versions.

Strange, I've been loading extra sfs's on the fly since Pup 412 days. I'm currently running Racy5.5 with sfs_load installed (a .pet I installed). Works perfectly. Prior to this a script like sfs-activate could be used (used that with Pup412). What did change is the sfs version. Since Pup 431 version 4 is being used.
Not if you are running in pupmode 12
Why not? Install it and later on uninstall it in the same session before you reboot/shutdown.

[bigpup]

Strange, I've been loading extra sfs's on the fly since Pup 412 days.

Not talking about just being able to load or unload SFS packages.
Talking about some Puppy versions will not see them unless they are located in /mnt/home/
Also, there was a max limit of 6, that could be loaded at one time.
Again, what version of Puppy and what version of sfs-load is in it, will affect the load unload options.

About pet temporary installing.
The discussion was about temporary installing and it not getting into the save.
Pupmode 12 does not use a save ramdisk.
Installing a pet goes directly into the save, along with any other stuff, the pet provides, to change system files.
Depending on the version of PPM used to uninstall the pet.
It may not uninstall everything.
PPM has been tweaked and improved.
The latest version of PPM, is not in every Puppy version, that people seem to want to use.

[amethyst]

Strange, I've been loading extra sfs's on the fly since Pup 412 days.

Not talking about just being able to load or unload SFS packages.
Talking about some Puppy versions will not see them unless they are located in /mnt/home/
Also, there was a max limit of 6, that could be loaded at one time.
Again, what version of Puppy and what version of sfs-load is in it, will affect the load unload options.

About pet temporary installing.
The discussion was about temporary installing and it not getting into the save.
Pupmode 12 does not use a save ramdisk.
Installing a pet goes directly into the save, along with any other stuff, the pet provides, to change system files.
Depending on the version of PPM used to uninstall the pet.
It may not uninstall everything.
PPM has been tweaked and improved.
The latest version of PPM, is not in every Puppy version, that people seem to want to use.

The point is you said it can't be done in Pupmode 12. THAT's WRONG. IT CAN BE DONE. BTW - It's of course possible to disable any savings even in pupmode 12 with a bit of know how. The loading of more than 6 extra sfs files has been experimental for ages. I can easily do it even with Racy. Anyways, use pupmode 13 instead for temporary installing .pet files easier but not necassarily a must.

[bigpup]

I am not talking about anything but normal operation.
I guess we do not agree on what a temporary pet install is.

That sfs load limit of 6, is going back in Puppy history. Probably should have not talked about it.
But I wonder how well that gets tested in each new Puppy version.
It is not something I test for.

With all the code changing getting done in Woof-CE.
Who knows for sure, what did work, no longer does.
Only way to really talk about this, is how it works in a specific Puppy version.

[amethyst]

Well temporary install means temporary, you install it and later on uninstall it IF IT WAS SAVED TO YOU SAVEFILE AT ANY STAGE. The loading of extra sfs's works the same on all puppy versions (at least from Racy onwards) in my experience. BTW - what's, normal in Puppy? The whole thing is driven by a bunch of scripts. Changing/creating scripts, etc. to suit your needs does not make Puppy "less normal". In fact, it should be encouraged as it is novation.

[mikeslr]

'Oh, unmounting increases ram usage, but only a little bit.'

CleanRAM recommended, viewtopic.php?p=25645#p25645.

I thought this might be of interest even if not exactly on point: I've been comparing the differing effects booting with a SaveFile vs. an ydrv.sfs, and using the boot arguments "copy, nocopy and normal i.e -no argument provided". I should mention that the SaveFile occupies about 1792 Mbs on my USB-Stick and that the OS is Bionicpup64. By the way, employing the Save2SFS module of nicOS-Utilities-Suite, viewtopic.php?p=12983#p12983 the ydrv.sfs only occupies 436 Mbs: a compression effect of 4.11 times.

RAM & Cache Usage.png (50.56 KiB) Viewed 901 times

My focus had been on 'Actual Used RAM'. But take a look at the figures relating to Cache, especially compare 'Cache' usage under the 'nocopy' argument with the alternatives. The figures relating to the use of an adrv with the 'nocopy' argument are really peculiar.

I suspect that somewhere therein lies the answer to the question I've frequently posed: "With Puppy entirely functional from RAM --e.g., booted from a removed USB-Key-- where is the information obtained from the storage medium located. I think rufwoof mentioned in some post the suggestion that it was compressed in RAM. I think the terminology pupsys-info uses is misleading, that 'compressed in RAM' is what's referred to as 'cache'. If so, on its own --that is without the employment of zram, Puppys --Bionicpup64, at least-- applies a compression ratio of between 4 and 5. And it may make a difference in what becomes 'Actual Available RAM' if the source of the information on storage was previously compressed, e.g. in an adrv.sfs rather than uncompressed, e.g. in a SaveFile.

[williams2]

@mikeslr

CleanRAM recommended

I wrote a simple script named cache-clear which I used in the little demonstration of sfs mounting.

Code: Select all

#!/bin/ash
sync
sync
echo 3 > /proc/sys/vm/drop_caches
echo 'cache cleared'

which does exactly the same thing as CleanRAM, using the same method.

There are 2 syncs because sync blocks (waits) until all the data blocks have been written, except the last block. When the last block of data starts to write, sync unblocks (stops waiting) and the next command in the script executes. The 2 syncs makes sure that all of the data blocks have been copied before the next command in the script executes.

I don't think it is useful to clear cache memory when Puppy is running normally. The kernel automatically manages the cache and buffer memory spaces constantly in the background, and works quite well. The kernel can instantly make use of memory that is in the cache, if needed.

[williams2]

The difference between 1.7M of buffer space and 1.8M of buffer space is not really significant.
1.7M and 1.8M are rounded numbers. A difference of even 1 byte could be enough to change a rounded 1.7 to a 1.8

And the multitasking operating system is constantly doing things in the background that can affect test results.

The results of the test seem to show that mounting and unmounting an sfs file, containing a 15GiB file, does not use a significant amount of memory.

Someone asked about the meta-data, whether it all gets copied into ram when the file system is mounted. Meta data is the filename and the path and permissions and the size of the file and what blocks of the file system are being uses to contain the file's data, etc, etc. The actual data in each file is not copied to ram until it is actually accessed. I think the same is true of the meta data.

Addressing the question of Mode 12 vs. Mode 13, Mode 12 writes directly (through a buffer) to the save file/folder, and Mode 13 write to ram, which may or may not be copied to the save file/folder.

Mode 12 uses less ram, which might or might not be a significant amount, depending on the circumstances.
Data is almost immediately written to the save file/folder, which can be a good thing if the operating system shuts down unexpectedly, eg. a power failure.

Mode 13 uses more ram, which might or might not be a significant amount, The data can be written to the save file/folder at the convenience of the user, who might choose not to save changes at all. This can be useful if installing a piece of software did not work well. It could be useful if it is suspected that some sort of malware got into the system, maybe through an internet browser. It might help a usb flash drive to last longer (fewer writes.) it might be faster to read and write to a ram layer, but buffers and caching helps the system to run faster, with fewer accesses to the hardware being necessary, in either mode.

[williams2]

@mikeslr

especially compare 'Cache' usage under the 'nocopy' argument

Space used in a tmpfs (a file system in ram) is included in the amount of space used in the cache, as reported by the kernel.
Space used by files in a tmpfs can not be cleared from the cache.
Files in a tmpfs can be deleted, of course.

Puppy system sfs files that are copied to the ram tmpfs file system, use space in ram which shows up as space used in the cache.

For example, 911M + 800M = 1711M = 1.7G

Code: Select all

# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        472M        2.1G        826M        1.8M        911M        1.9G
Swap:            0B          0B          0B
# dd if=/dev/zero of=/tmp/dd.bin bs=1M count=800
800+0 records in
800+0 records out
838860800 bytes (839 MB, 800 MiB) copied, 2.03455 s, 412 MB/s
# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        472M        1.3G        1.6G        1.7M        1.7G        1.2G
Swap:            0B          0B          0B
# rm /tmp/dd.bin 
# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        520M        2.0G        873M        1.8M        959M        1.9G
Swap:            0B          0B          0B
# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        520M        2.0G        873M        1.4M        959M        1.9G
Swap:            0B          0B          0B
#
# cache-clear 
cache cleared
# free -wh
              total        used        free      shared     buffers       cache   available
Mem:           3.5G        517M        2.0G        877M        1.8M        963M        1.9G
Swap:            0B          0B          0B
#

where cache-clear is sync;sync;echo 3 > /proc/sys/vm/drop_caches

[JASpup]

@mikeslr you're on a interesting thought track.

SFS compression is a delight.

I consider the topic point settled but significant enough to warrant these tangents. PUPMODE 12 is like forced browser updates, too paternalist without an opt-in, and the option is editing your boot config -- too inside baseball.

For your chart, I sneeze and I have a 29MB browser config in ram. The smallest in memory is 11MB.

Related to your chart I (for my own system) would focus on that usage. What does it cost to put config and cache on a disk? flash vs. disk?

Firefox standalone app isn't a speed nuisance, even on USB 2.0 (32 systems), but after all this focus I'm still not certain what to do with config data. It eats memory and threatens our privacy.

Either way adrv has benefits beyond memory usage when roughly equal to pupsave. I'm happy when the system is running and no physical partitions are mounted.

[gyrog]

pupmode=12:

The old unionfs, current aufs, and overlayfs, all have the same requirement, the top layer, the RW layer, must be a directory on a Linux partition.
The simplest way to do that is a directory on an existing Linux partition, in Puppy terms, a save-folder in pupmode=12.

For absolute minimum use of RAM, the thing you can control is the amount of tmpfs in RAM, and for absolute minimum tmpfs in RAM use pupmode=12 with pfix=nocopy.
Then the only tmpfs in RAM would be "/tmp".

pupmode=13:

Depends on the 'snapmergepuppy' script.
And 'snapmergepuppy' depends on aufs, what 'snapmergepuppy' does is not supported by overlayfs.
Overalyfs is included in the kernel, aufs is not.
Should support for producing new aufs patches to new kernels disappear, then so does pupmode=13.

So what's the good features of pupmode=13, that are worth implementing in another way?

"Being able to select 'SAVE/NOSAVE' on shutdown".

Are there others?

Note: As part of my project of running Puppy with an 'init' script using overlayfs to build the stack,
I implemented a 'SAVE/NOSAVE' feature for pupmode=12.

It's time we started reducing Puppy's dependence on the particular features of aufs, as much as we can.

[JASpup]

For absolute minimum use of RAM, the thing you can control is the amount of tmpfs in RAM, and for absolute minimum tmpfs in RAM use pupmode=12 with pfix=nocopy.
Then the only tmpfs in RAM would be "/tmp".

useful clarity

stands to reason then pupmode=13 would only add /root tmpfs to RAM, not that it is carefully managed when there

the other idea then is to see this visually without having to conceive of it abstractly

if RAM is just a warehouse that does not hinder performance at 80% vs 50% utilization, we would only need to know when we're getting close, but otherwise accurate account of it should be normal

i'm regularly apprehensive about taxing 2gb

8gb in Puppy with four browsers open I do not feel like I'm getting close, as it seems more than needed

[JASpup]

So what's the good features of pupmode=13, that are worth implementing in another way?

"Being able to select 'SAVE/NOSAVE' on shutdown".

Are there others?

Note: As part of my project of running Puppy with an 'init' script using overlayfs to build the stack,
I implemented a 'SAVE/NOSAVE' feature for pupmode=12.

It's time we started reducing Puppy's dependence on the particular features of aufs, as much as we can.

Since my understanding is pupmode=12 is continuously saving to the pupsave, I have to understand how implementing the shutdown save option is a feature, unless you mean simply getting 12 to behave like 13.

If you want the ideal pupmode=13 customization, JWM needs the X-series (XFCE) Puppy Event Manager save options:

x-pem.png (10.8 KiB) Viewed 813 times

An experienced user is rarely going to regret forgetting to save a custom save upon exit, and the -0=Never option allows us to skip the save question where the [+0] option is more likely to produce an undesired save.

[gyrog]

@JASpup
In pupmode=12, the directory on disk is directly mounted as the top of the aufs stack.
So, when a file is written to the stack, it gets written directly to disk, except for the normal automatic buffering done by kernel+filesystem, there is no "saving" process at all.
Just like writing a file outside the stack, e.g. to '/mnt/home'.

Since pupmode=13 itself cannot survive a transition from aufs to overlayfs, I would like Puppy to be more prepared for such a transition.
The point of my example of pupmode=12 with a "save/nosave" query on shutdown is that some features of pupmode=13 may be able to be implemented in another way, whereas some features of pupmode=13 cannot.
For instance, there is no point in spending effort on a timed "save" feature, it simply could not work under overlayfs.
But the feature of "save/nosave" is worth exploring for a different implementation, even if it leads to a completly new pupmode,
since I already know of 1 way of doing this under overlayfs.

By all means use pupmode=13 if that best suits your needs, but be aware that it may or may not have a long term future.
The problem is that we have no idea if aufs will die forcing Puppy to use overlayfs, or if it does die, when it will die.

[JASpup]

@JASpup
In pupmode=12, the directory on disk is directly mounted as the top of the aufs stack.
So, when a file is written to the stack, it gets written directly to disk, except for the normal automatic buffering done by kernel+filesystem, there is no "saving" process at all.
Just like writing a file outside the stack, e.g. to '/mnt/home'.

Not sure the difference between saving and writing unless you mean automation (for the latter).

Since pupmode=13 itself cannot survive a transition from aufs to overlayfs, I would like Puppy to be more prepared for such a transition.

My struggle is to comprehend if you're refering to an extant compatibility challenge or customization shuffle.

Iow, something is going to need adaptation for progress, or we are trying to do something new for the sake of it.

The point of my example of pupmode=12 with a "save/nosave" query on shutdown is that some features of pupmode=13 may be able to be implemented in another way, whereas some features of pupmode=13 cannot.
For instance, there is no point in spending effort on a timed "save" feature, it simply could not work under overlayfs.
But the feature of "save/nosave" is worth exploring for a different implementation, even if it leads to a completly new pupmode,
since I already know of 1 way of doing this under overlayfs.

I've never been inclined to use a timed save feature, need empathy for why anyone would, and it seems healthy to condition users to manual saves, and not particularly demanding, as it's a fight just to get the save icon off the desktop in pupmode=13.

By all means use pupmode=13 if that best suits your needs, but be aware that it may or may not have a long term future.
The problem is that we have no idea if aufs will die forcing Puppy to use overlayfs, or if it does die, when it will die.

Since Puppy is leaning away from full installs, making it a background feature, what other pupmode should we be expecting as normal? 12 because it's the only other possible? 5 is nice (in it now), but not for new users who expect to save anything.

[gyrog]

@JASpup,

writing:
When a program thinks it's writing to disk, it is, e.g. pupmode=12.
There is no Puppy "save" code involved.

saving:
For all those pupmodes where the RW layer at the top is a diretory in a tmpfs in RAM, e.g. pupmode=13,
when a program thinks it's writing to disk, it's not, it's just writing to a tmpfs in RAM.
To provide persistence, Puppy "save" code is required at shutdown, to actually write the files in RAM to disk.

pupmode=13:
I get the impression that the main attraction is the "save/no_save" on shutdown.

There is a real threat that pupmode=13 may have to be discarded, because it cannot transition to overlayfs.
We can simply ignore that possability, and just carry on as we are now with aufs.
Or we can take out some insurance against that possability, and try to develop Puppy to be less aufs dependent,
but retaining the current useful features, by implementing them another way.

Hmmm..., maybe my comments are a little of topic, since they are more about future Puppies than using the current Puppy.
Although for me, even in current Puppy, I will avoid becomming attached to pupmode=13.

[JASpup]

pupmode=13:
I get the impression that the main attraction is the "save/no_save" on shutdown.

It's the opposite: the main attraction is controlling your saves.

The beginner wants to learn how a pupsave works so they can save changes to a live boot.

The intermediate wants to control what's in a pupsave and how it changes, including as written the choice of not saving at all, or even being reminded of it at shutdown.

The advanced wants back into pupmode=5 with a remaster or alphabet drive, and maybe some automated .sfs loading.

I've only seen it briefly, but apparently we can't control saving in pupmode=12, making 13 the preferred choice.

If overlayfs transition is going to thwart current use, the insurance is facilitating what I've stated.