Why ext2 on zram rather than using a tmpfs with zram as swap?

[artemis]

I was just reading https://bkhome.org/news/202205/a-rethin ... cture.html and a question i had is, why bother formatting the zram device as ext2 and using it directly rather than using a tmpfs with zram as swap?

I have tested this a lot on my 2GiB ram computer and tmpfs files can in fact get moved to swap devices - including zram. This defers the compression until it is actually necessary and allows the zram swap device to get used for both tmpfs files and program memory alike. Is there a benefit to using ext2 on the zram device instead of doing this that I'm not seeing?

Thanks for the interesting project updates!

EDIT: i wonder also if this would help with "the elephant" of containers, since a tmpfs could be created for each container very cheaply, and space is only used up as necessary. you could even limit how much data each container can create by changing the tmpfs sizes, if you wanted. and they would all share the same zram swap device as necessary over the runtime of the system.

this is technically also possible with ext2 on zram, except with zram you have to decide how many block devices to create when you load the module if i recall correctly, so you would have to pre-allocate them so to speak and managing them would get messy

[BarryK]

I did consider tmpfs with zram swap, but flipped a coin and went for ext2 on zram.

The ext2 on zram has more flexibility... I had some ideas in the back of my mind, working on one of them now.

I have done a build without containers, but have now created the infrastructure to have containers, with the containers sharing the same zram.

A bit premature to say anything about this, as I am not sure if there is going to be a gotcha. Will do a build tonight and see if the method works. If it does, containers are back, and I will post the details.

I was thinking, as woofQ now has a global variable to disable containers, that is building a more Puppy-like distribution. If there was also another global variable to set the distro name, instead of "Easy" and "EasyOS", then someone could use woofQ to build their own Puppy-like distro. Just a thought.

Well, more Puppy-like than with containers anyway. Still quite different under-the-hood.

[BarryK]

Regarding zram size, I have set it about 120 - 150% of available RAM. mke2fs is run on that, but the actual RAM used is quite small. RAM usage will grow as files are written to the zram.

Regarding the module, I have it builtin to the kernel, not a module. The default is just one block device; however, more can be created at runtime. The init script in the initrd accesses a certain file in /sys to add two more.

[user1111]

Another alternative to tmpfs backed by zram for swap when you're OK with fixed sized space allocations is to use a compressed filesystem within tmpfs

# Create a 1GB btrfs filesystem within tmpfs (/dev/shm)
cd /dev/shm
dd if=/dev/zero of=btrfs.fs bs=1M count=1024
mkfs.btrfs /dev/shm/btrfs.fs

# Mount that filesystem specifying to use lzo compression
mkdir /mnt/btrfs
mount -o compress=lzo /dev/shm/btrfs /mnt/btrfs

# Test it
cd /mnt/btrfs
dd if=/dev/zero of=bigfile bs=1M count=2048

... and file created ok, that is stored in a lzo compressed filesystem based in tmpfs, where df -h indicates 73MB used, 841MB available of the 1GB total /mnt/btrfs size after creation of that 2GB of zeros file (/mnt/btrfs/bigfile) (/dev/shm with 1.7GB size on my system shows 1GB used, 677MB available).

I did initially try mounting it using -o compress=lz4 instead, but my Fatdog 812 system didn't permit that.

Going the other way, most compression/slowest and

https://git.kernel.org/pub/scm/linux/ke ... 0d7ee5b2e7

indicates very slow compression for mount -o compress=zstd:15 ..., but that might still be OK for EasyOS given a desire to maximise 'changes' storage space/minimise amount of writes to USB. Mostly it is the writing (compression) that is slow, decompression/read is still relatively quick. But perhaps the progressive writes (changes) may not be a factor, maybe no slower than if writing to a mechanical storage device. Whilst reading (to write to USB when a save occurred) is still relatively quick.

dd'ing another 2GB of zeros into compress=zstd:15 did take noticeably longer, but still quicker than if to disk, and resulted in df -h showing 61MB used (compared to 73MB for when using compress=lzo above).

For another test I unsquashfs'd my fd64.sfs (main fatdog sfs) to /mnt/btrfs mounted with zstd:15 compression, was sluggish, at times with the spinner/bar mksquashfs extraction line halting for a number of seconds. Of the 2448MB uncompressed size of that fd64.sfs it got to 64% extraction completed before ending with a out of space error showing 32432/50555 (64%) progress status, so stored around 1570MB of total 2448MB of fd64.sfs content/data on that measure.

Repeating the unsquashfs except into a -o compress=lzo and the bar/spinner kept running continually, but wasn't that much/noticeably quicker and it got to 31974/50555 (63%) before being out-of-space. In other words zstd:15 highest compression level wasn't that much better than just straight lzo compression. Conclusion for me from this type as I test/do-it is that the higher levels of compression available might make relatively little difference in the EasyOS usage case.

Should also say that the tests were run without any swap being available, on a oldish 2 core 4GB ram AMD laptop system.

[BarryK]

@rufwoof ,
That's very interesting. You are good at "lateral thinking".

Regarding the ext2 in a zram, yesterday I realised the obvious (sometimes, what is obvious can be elusive):

https://bkhome.org/news/202205/containe ... -zram.html

[user1111]

https://bkhome.org/news/202205/containe ... -zram.html

Containers do have lots of attractions. For example, running Easy Dunfell I could run Easy Bookworm in a container, if I want access to the bigger package repository. Or, I want to run an app with the isolation and security offered by a container.

Containers do turn what otherwise might be seen as being for fun/toy into a more serious system.

Running as root otherwise is vulnerable to the likes of a zero day browser exploit enabling /etc/passwd (shadow), /etc/wpa_supplicant, .ssh folder keys ...etc. potentially to be remote copied that along with your geolocation could be passed/sold on.

One of my boot choices uses pfix=nox and from that cli I have the option to startx and boot into a regular real root gui desktop (main fatdog system), or running ./x ... starts my container that uses the main systems sfs as its base, but has no drive icons, root=restricted userid ...etc. As part of that in its initialisation I change the root password, remove wpa_supplicant.conf, remove .ssh keys ...etc. I also iptables (firewall) block access to my router admin IP at the host level, which the containerised root can't drop that rule. Combined with KASLR and running in its own X (Xephyr) with capabilities dropped, unshared, chroot ...etc. such that root is just a regular userid, along with running in ram with no saving/clean booting by default ... and collectively that's a contender as a serious system.

OpenBSD additionally generates a new kernel at each boot, creates a new one using tar and a kernel.tgz ... in readiness for the next boot using that new version. Wouldn't be difficult to replicate similar. OBSD also uses W^X, separates out partitions (slices) and sets the permissions to restrict some files/libs having the ability to write when they're just supposed to be read only. Again other than a bit laborious - not difficult to replicate. Pledge and unveil are process/program restrictions - somewhat like removing capabilities on a per program basis and unveil is similar but restricts paths - such as chrome not being able to access your .ssh folder or suchlike. i.e. with relatively 'trivial' changes/extensions, could be a linux distro right up there alongside the best, if not better (fast reinitialisation back into a clean/pristine boot and generally far faster and with a much broader range of devices being supported).

Fatdog - that I use to run a similar container system as EasyOS, by default already runs network processes as a restricted userid. Just all part of raising the security bar ideally with the least overhead. In the case of OpenBSD that bar raising does come with considerable overhead (slowdown) - where in the latest 7.1 that's in part being addressed by pushing the processor into near turbo type settings - so quicker, but much hotter and still considerably behind the likes of EasyOS in speed. With all cores running full blast all of the time (default for 7.1 OpenBSD) on my laptop and the base does become uncomfortably hot - inducing setting the cpu's back down to half speed and being back into 'sluggish' mode.

[user1111]

Regarding the ext2 in a zram, yesterday I realised the obvious (sometimes, what is obvious can be elusive): https://bkhome.org/news/202205/containe ... -zram.html

I did create a 1.4GB /dev/shm btrfs filesystem, on my 1.7GB free laptop system and managed to fully unsquashfs fd64.sfs (2.5GB unsquashed) into that, and then reformed another fd64.tst.sfs from that (onto hdd) using lz4 compression. Still with 350MB of btrfs 'space' left available.

A little swap can be useful, resulting in a possible recovery position - a slow near frozen system but where you might be able to kill the offending process, where otherwise without swap the system might simply just totally freeze/halt.

I did notice that after having used tmpfs in the main system, that when I opened my (Fatdog based) container that /dev/shm was available, but when I ate into that - the system crashed. Conceptually could be a security flaw, contained system (browser/whatever) inducing a buffer overflow in order to gain control over the instruction pointer (controlled crash in a manner that has the instruction pointer potentially pointing to code that the buffer overflow process loaded).

In my case I'd be inclined to allocate perhaps a 200MB tmpfs swap, a big chunk of rest of tmpfs to btrfs, and be careful with tmpfs usage within the container system (marking whatever had been used in the main system as also being unavailable in the contained system).

[dimkr]

You don't really have to choose between zram and something else: search for "writeback" in https://www.kernel.org/doc/Documentatio ... v/zram.txt. zram can be configured to flush the block device contents to another block device in the background, making zram act as a compressed in-memory cache for frequently accessed files.

[user1111]

Just tested using btrfs zstd compression for a 2GB tmpfs allocation, unsquashing my 2.58GB uncompressed fatdog sfs content data (that's more usually around 850MB when lz4 Xhc compressed with mksquash, 650MB if high xz compressed) into that and it left 314MB of indicated free space (df -h) remaining.

Allocating a 2GB of lz4 zram in tmpfs and using the same btrfs zstd compressed filesystem within that and following extraction of the sfs into it df -h indicated 629MB of free space still remaining.

Sometimes you can get better compression by compressing already compressed data, the above is indicative that lz4 compressed zram isn't a tight compressor, which is common knowledge anyway (its fast but not tight). For such a large difference however that is also indicative that zstd in btrfs isn't that tight either and that applying both can yield modestly better total compression.

Perhaps also a function of btrfs zstd and zram lz4 compression being applied to different elements, blocks/filesystem and files ??

For the tests I shutdown/rebooted prior to immediately running each test, so should be a relatively consistent/equal starting point/baseline. Here's the code for the combined zram and btrfs case ...

Code: Select all

# btrfs filesystem using zstd compression stored within 
# lz4 compressed zram

modprobe zram num_devices=1
# check ... cat /sys/block/zram0/comp_algorithm for available methods
echo lz4 >/sys/block/zram0/comp_algorithm
echo 2GB >/sys/block/zram0/disksize
mkfs.btrfs /dev/zram0

# Mount that filesystem specifying to use zstd compression
mkdir /mnt/btrfs
mount -o compress=zstd /dev/zram0 /mnt/btrfs

# Test it
cd /mnt/sda2/FATDOG812-FINAL
unsquashfs -f -d /mnt/btrfs fd64.sfs
sync
df -h

So if you're prepared to dedicate more cpu cycles for the benefit of squeezing that bit more into available ram space, then a combination of lz4 compressed zram and zstd compressed btrfs is one option. Noteworthy however is that when I ran dd if=/dev/urandom of=bigfile bs=1M count=1000 .... to fill up the remaining available space, as I had no swap activated it did throw a wobbly following the dd exiting with out-of-space, requiring a reboot. But that was my own fault for allocating a 2GB zram into a system with just 1.7GB of tmpfs being available.

Speed wise and quicker than mechanical HDD to extract into, but I suspect not much different to if I'd used a SSD. As such I would envisage that its of little practical use, more just a academic type observation. That said and decompression wise (reading) for both lz4 and zstd is relatively quicker than for compressing (writing), so for compress/write once, decompress/read many type applications there could be possible usage cases.