Tricking Puppy into thinking it has more RAM

[greengeek]

Does your Puppy have limited RAM?

It can be difficult using Puppy to process video files or rip/shrink/copy DVDs on a PC that has only 1 or 2GB of RAM.

It may be possible if you have a large savefile but I do not use savefiles anymore as I experienced too many issues with corruption and machine hangs. I prefer to do my video processing by booting Puppy "live" into RAM and manipulating my files in /root. (avoids many permissions problems and just seems to work well for me)

But obviously this is problematic if I do not have much RAM.

So how to get around this?

In my experience video processing chokes - not always because it runs out of RAM - but also because the kernel runs out of cache (I can't really justify this last comment properly but I have seen lots of error messages and hangs that lead me to believe this is the case).

So what I do now is to trick Puppy into thinking it has way more RAM than it actually does. Not just storage - but actual RAM.

This trick is based upon the way that a "live" puppy (no savefile) handles it's swap partition and it's personal storage. During the boot process Puppy has a look at any swap partitions that are available and allocates a slice of that space as "bogus RAM" that it can use as overflow (quite separate to the remainder of the swap partition - which is still used as "swap").

This trick can be achieved with a usb stick but I currently use a small SSD as they are designed to be more robust than usb sticks in the long term.

I use a SATA3 to USB adapter cable to plug the SSD into whichever is my fastest usb port (most laptops have a mix of slow and fast ports) and format the SSD as follows (based on a 120GB SSD):

Partition 1: 20GB FAT32
Partition 2: 60GB EXT 2, 3 or 4
Partition 3: 30GB LINUX SWAP

Then I reboot with this SSD connected and Puppy grabs a sizeable chunk of the SWAP to use as temporary "personal storage space" (or "working RAM").
(It wont correctly grab the extra "personal space" if you manually attach the swap using the "swapon" command - it has to occur automatically during the boot process on a Puppy that has no savefile)

You can then see this reflected in the overall personal storage space total by hovering the mouse over the "Partview" icon in the tray. If it shows a working area of 15GB upwards then I know I am good to go with video processing.

RAMtrick_.png (10.92 KiB) Viewed 1324 times

.

This technique allows me to use a generic puppy, on a generic laptop that has no HDD and no savefile and make a start on my video processing. I store the resulting files on Partition 2. (Can copy them later to the FAT32 Partition 1 if I need to transfer them to a Windows system).

It doesn't really matter how slowly the system runs - at least this way Puppy has enough brain room to get it's job done without choking itself to death. (same technique works to use improve browser performance on low spec machines).

I can also go a step further by setting the boot flag on Partition 1, then copying Puppy files to directories on Partition 2 and running Grub4dos to make the whole thing bootable.
(But if you do this be careful to remove any other usb stick that happens to be fitted - as most puppies except the very latest ones struggle to handle more than one usb stick plugged in during boot).

This gives me a portable, bootable SSD containing one or more Puppies just for the purposes of the heavy lifting needed when handling video.

Great for testing machines that have been binned because of supposedly "insufficient specs".

Caveats:

1) There are probably much better ways of doing this. (I just dont know them)

2) SSDs sometimes use more power than you expect - especially the larger SSDs which have more nand chips. Surprisingly they can often use more power than 2.5" hard drives.

If you find that an SSD overloads your usb port's available current you may have to use this technique with a usb stick as they use way less power, or even use an old low power 2.5" spinner style HDD if you can find one that has enough GB. Many of the older laptop HDDs are limited to 500mA current draw so they can work fine on a lower power usb port.

3) The huge swap space has to be available during the boot process - you can't plug it in later.

[JASpup]

useful how-to

[fredx181]

Then I reboot with this SSD connected and Puppy grabs a sizeable chunk of the SWAP to use as temporary "personal storage space" (or "working RAM").
(It wont correctly grab the extra "personal space" if you manually attach the swap using the "swapon" command - it has to occur automatically during the boot process on a Puppy that has no savefile)

Any idea how this is "automatically" done ?

[greengeek]

Any idea how this is "automatically" done ?

No unfortunately i don't know the details but I believe this functionality is part of the initrd.gz.
I recall reading about it a few years ago (possibly in one of BK's blogs) and got the impression that it is part of Puppy's inherent layering system that adds available system RAM to the allocated personal space (stolen from swap) to make a large pseudo RAM.

I am probably expressing this all wrong but that was the impression I got and this method certainly seems to work well for me anyway.

My understanding is that swap space is used in a specific way with memory pages being stacked, cataloged, swapped and purged and is susceptible to clogging (for want of a better word) whereas the "personal space" is operated in a non-paged way using a layered system which works better for Puppy than the swap paging does. (forgive me for that non-technical spitballing).

Will post back if i can find a reference that explains how the available RAM is combined with the stolen portion of the swap area.

EDIT : I found this link which states about halfway down:

A significant difference occurs the second time that Puppy is booted from USB. If there is enough RAM (remembering that the ramdisk can also use a swap partition if it exists, so the available space is effectively the size of RAM plus size of the swap partition), then Puppy will mount pup100 onto /mnt/pupxxx directory in order to copy all the files from /mnt/pupxxx to /root.

Don't know why it refers to a second usb boot - I think this is outdated information but the principle is the same.

[bigpup]

The correct info is here:
How Puppy Works
http://bkhome.org/archive/puppylinux/de ... works.html
The pupmode, Puppy is running in, is important on exactly how it runs.
Lower down in this info, is how each pupmode works.

This info only applies to official Puppies, made using Woof CE, and their derivatives.
Puppy like OS's on the forum, only the developer, that made it, knows how they work.

[fredx181]

The correct info is here:
How Puppy Works
http://bkhome.org/archive/puppylinux/de ... works.html
The pupmode, Puppy is running in, is important on exactly how it runs.
Lower down in this info, is how each pupmode works.

This info only applies to official Puppies, made using Woof CE, and their derivatives.
Puppy like OS's on the forum, only the developer, that made it, knows how they work.

Thanks, I see it's near the bottom at "Step 2".("#we can make the ramdisk real big now...")
A bit hard to understand for me right now, complicated stuff, but very interesting....

EDIT: @greengeek I'd say that the title is a bit misleading, actually it's not really more RAM, but more space available for files when running without save (or with pupmode=13 ??) by using swapspace (however you did explain that very well)
Perhaps better would be "Tricking Puppy into thinking it has a bigger RAMDISK" (but then again, it's not a real RAMDISK, as it's built from swap, oh well...)

[greengeek]

I'd say that the title is a bit misleading, actually it's not really more RAM, but more space available for files when running without save (or with pupmode=13 ??) by using swapspace (however you did explain that very well)
Perhaps better would be "Tricking Puppy into thinking it has a bigger RAMDISK" (but then again, it's not a real RAMDISK, as it's built from swap, oh well...)

Yes, the title could be better. But I'm not sure what it should be.

It has been changed by one of the moderators - my original title was "Tricking the RAM" - because this trick does seem like a way to get Puppy behaving as if it has more actual memory.

But if the title referred to "Ramdisk" it might put off users that could benefit from this simple setup.

And of course I don't understand the technicalities under the hood - I don't know exactly how the software is interfacing with the hardware.
Is this a Ramdisk - or is it a DiskRAM ??

I dunno.

Sometimes we are working in the nether regions...

[williams2]

From the tmpfs manpage: "Mount options ... size= ... Specify an upper limit on the size of the filesystem. ... The default ... is size=50%."

You can set the maximum size of a tmpfs to any size you like. If size is set to less than the amount of physical ram, it helps to prevent Out Of Memory conditions.

For example:

Code: Select all

# mkdir /mnt/mytmpfs
# mount-FULL -t tmpfs -o size=10M tmpfs /mnt/mytmpfs
# df -h /mnt/mytmpfs/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            10M     0   10M   0% /mnt/mytmpfs
# cp /mnt/home/fossapup64-9.5.iso /mnt/mytmpfs/
cp: error writing '/mnt/mytmpfs/fossapup64-9.5.iso': No space left on device
# 
# df -h /mnt/mytmpfs/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            10M   10M     0 100% /mnt/mytmpfs
# mount-FULL -t tmpfs -o remount,size=64G tmpfs /mnt/mytmpfs
# df -h /mnt/mytmpfs/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            64G   10M   64G   1% /mnt/mytmpfs
# cp /mnt/home/fossapup64-9.5.iso /mnt/mytmpfs/
# df -h /mnt/mytmpfs/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            64G  409M   64G   1% /mnt/mytmpfs
# ll /mnt/mytmpfs/
total 409M
-rwxr-xr-x 1 409M Jun 21 14:12 fossapup64-9.5.iso
#
# umount-FULL /mnt/mytmpfs/
#

This creates a tmpfs with a maximum size of 10MiB
Copying a 400MiB file to the tmpfs does not work, it displays a "No space left on device" error.
Remounting the tmpfs with a maximum size of 64GiB (I have much less ram than 64G) works. The 400MiB file copies with no complaints.

A tmpfs can be remounted with a different maximum size any time you like.

The code that increases the tmpfs size is in /etc/rc.d/rc.sysinit

PS the Puppy OS uses tmpfs, it does not use ramfs or zram.

[greengeek]

PS the Puppy OS uses tmpfs, it does not use ramfs or zram.

Except maybe some of the newest ones? I think i read somewhere that the newest Slacko incorporates a zram. Dont know where or why.

[williams2]

ramfs is an older ram file system. It does not make use of swap space. Space used in a ramfs does not show up if you type free I think some of the newer Fatdogs can be configured to use tmpfs or to use ramfs.

tmpfs is a newer file system than ramfs. Tmpfs does make use of swap space. Space used in the tmpfs does show up as space used in the cache space, when you type free.

zram is a compressed file system in ram. AFAIK, I think that EasyOS uses zram by default. Most of the mainsteam Puppy varients, like XenialPup and BionicPup and Fossapup use tmpfs, not zram. I also seem to remember some of the latest Puppy varients experimenting with zram, but none that I have actually tried.

I try not to make statements that might be taken as absolute truths, not to be questioned or challenged. I can easily be wrong. Sometimes it is helpful to simplify a response, to make your point clearer. I often do that, rather than cluttering up a response with unessential detail.

[greengeek]

Sometimes it is helpful to simplify a response, to make your point clearer. .... rather than cluttering up a response with unessential detail.

This is unfortunately a technique I struggle with. It's easy to get deeply lost in detail before seeing daylight ahead.

[dancytron]

Zram can also be used as swap space so that when you start to swap it isn't the instant massive slow down. You need to set it to swap before your regular swap file/partition.

This is what I used to set it up in Debian Dog.

Discussion on Old forum Debian dog thread with setup scripts etc.

http://murga-linux.com/puppy/viewtopic. ... 747#980747

Reference:

https://wiki.debian.org/ZRam

Code: Select all

root@ChromeDog64:~# swapon  
NAME       TYPE      SIZE USED PRIO
/dev/sdb3  partition 1.4G   0B   -1
/dev/sda3  partition 4.9G   0B   -2
/dev/zram0 partition 1.4G   4K   10
/dev/zram1 partition 1.4G   0B   10