BTRFS and "compression"

[Clarity]

BTRFS is a "part" of the FD811 complement! Those developers should be rewarded for advancing this for us.

BTRFS in Fatdog.jpg (30.58 KiB) Viewed 782 times

Over my decades, I have constantly used compressed NTFS for my data storage needs. It has been very useful with little CPU demand as data flows thru the Windows/Linux I/O subsystems.

Creation of compressed NTFS has always been done via Windows to USB/SDcards/HDD/SSD partitions. And, I have, in the past, used them in LVMs. It has always been stable and data integrity has always been a paramount feature in its use.

Over the past days, @Rufwoof is showing us his findings in his tests with compression and it is, now, sparking interest in filesystems that afford on-the-fly compression via the I/O subsystem+filesystems for the benefit they provide in storage as well as benefits to speeding data use via less bytes to/from physical media.

As BTRFS offers changes, I wonder if there is a "mirrored" (not data, rather functionality) way to setup a compressed BTRFS such that it is done once at partition's formatting and have that "respected" throughout the life of the partition without user needing to ever revisit "compression". The mirror I am referencing is one that matches what has been tradition in NTFS. Once the partition is formatted with the compression set on, it is NEVER revisited and all OSes KNOW how to use it. Thus upon mounting, there is NO special effort to "alert" the OS of compression as this is all handled in the filesystem's cooperation with the OS. This means, as is the case for a compressed NTFS parttition, you mount/umount without regard of whether there is compression or not. Thus, I am asking if this kind of cooperation is inherent in BTRFS partition compression use in Linux?

I have tested BTRFS in the past on other OS, but not ever tried compression. BTRFS has matured such that more and more Linux are migrating to its use because of its current stability. So now may be a good time to dip toes into the water of its use with its compression feature(s).

Anyone have knowledge or experience with compression in BTRFS?

I know of this link

[user1111]

One of the answers in this post/thread

https://unix.stackexchange.com/question ... ermanently

suggests

btrfs property set /mnt/your-subvolume compression lzo

and that validation can be made by grabbing your Btrfs subvolume UUID with btrfs filesystem show /mnt/your-subvolume, you can then validate that the LZO compression feature is currently enabled with:

cat /sys/fs/btrfs/<UUID>/features/compress_lzo

that should return 1 if it s enabled

... So guessing that if sda3 is formatted to btrfs and you

Code: Select all

btrfs property set /mnt/sda3 compression lzo

it should preserve that setting.

[01101001b]

BTRFS is a "part" of the FD811 complement! Those developers should be rewarded for advancing this for us.

Yep. Now that I've learnt to compile my own kernel (Woo-hoo! At last! ) enabling btrfs (and xfs) support in-kernel is a must for me

Anyone have knowledge or experience with compression in BTRFS?

I use BTRFS regularly in my system (along with XFS) because its management of disk space is fantastic. However, I don't use any of its fancy features (snapshots, compression and so on).

A few years ago I tried to do what you're trying to do too: Using BTRFS the same way you use NTFS. Well, you can't (or, better said, I couldn't).

For starters, if you use compression, Btrfs MUST be mounted declaring that fact explicitly. There it's not like Ntfs because btrfs compression does not work (or it's managed) exactly the same way as Ntfs (the same can be said about defragmentation) (EDIT: @rufwoof seems to have hit the nail in the head).

Long story short, after a few attempts in my day, I desisted, cause I had some corruption with my data (not due to btrfs) and compression made it quite difficult to retrieve any data (in fact, by that time general opinion was not to enable it "happily and easily" unless you really knew what you were doing).

Last time I lost data (again) while using btrfs (and I repeat, not because of btrfs) without compression, I could restore almost the entire load of files (by the way, btrfs restore feature blew my mind! I had never met a file system before that allow me to extract data from my disks without previous mounting of the unit... although maybe I might be telling quite a nonsense and that IS the usual way with any FS. If so, anyone feel free to correct me).

Thing is, while checking the results later, I found some file were "incomplete".

Before anyone jump to conclusions, those files were restored "incomplete", neither because of the "failure" of the disk nor by some fault of the file system (btrfs), but because the way btrfs store data, that is, CoW (Copy-on-Write).

CoW is a GREAT feature, but in my case I prefer turning it off. Again, I must explicitly state that option while mounting the volume (mount -o nodatacow /dev/volume /mnt/volume); otherwise, CoW will be used.

I have tested BTRFS in the past on other OS, but not ever tried compression. BTRFS has matured such that more and more Linux are migrating to its use because of its current stability. So now may be a good time to dip toes into the water of its use with its compression feature(s).

Be careful, though, where you use it. Btrfs is known to have a hard time under intense disk activity (like running virtual machines under qemu with specific disk performance settings enabled). That's the reason I use btrfs for my data and xfs for my binaries (I always found ext2/3/4 too bloated and "slow" for my taste, regardless what some benchmarks say). I don't use ntfs by the tiny detail I hate to see its driver hanging in user-space

[jamesbond]

btrfs was originally developed by Oracle as a competitor for ZFS, which belonged to Sun. When Oracle bought Sun, they got ZFS too, so the motivation really dropped out and Oracle redirected its employees to do something else instead and btrfs languished in limbo for years. Apparently RedHat picked the pieces afterward and continued on their own (hence the progress), but now that RedHat is owned by IBM (which also owns XFS), we'll see what happens.

What stopped me from using btrfs in the past was its "fsck" tool could not even fix common corruption problems; today, that seems to be no longer the case, at least for the common ones, so it's probably safer. Note that there are certain corruption types in ext4 that e2fsck cannot fix as well.

---

That being said, btrfs, and ZFS, were originally meant to manage large number of large capacity disks connected through ultra high-speed controllers, in unimaginable number of combinations possible, with exotic features which only made sense in a large enterprise environment. Hence, they forego some of "home desktop" features (e.g. data preservation in time of sudden power loss - as they assumed that the systems would run with uninterruptible power supplies), and they require large resources to operate.

Of course, the meaning of "large" has changed with time - what was once "large" is now "small"; but still the point if you want to run either one, be prepared to dedicate some of the resources (namely disk space and RAM and some CPU power) to it; or suffer the consequences.

There is another filesystem that I like too, at least in terms of the features; and I think they're comparable to both ZFS and btrfs, if not better. It's called the HAMMER filesystem from DragonFlyBSD. Unfortunately, nobody has ported that to Linux yet. It would be interesting if they have done so.

[jamesbond]

I don't use ntfs by the tiny detail I hate to see its driver hanging in user-space

I don't use ntfs because if it gets corrupted then I need to find Windows to fix it. The "fsck" for ntfs (ntfsfix) only fixes a vanishingly small subset of problems, let's just say that if you run ntfs without Windows handy, be prepared to lose that data if that partition ever gets corrupted.

[Clarity]

... if it gets corrupted then I need to find Windows to fix it...

TRUE! Further, I have found another disturbing issue, that occurred recently. Using Linux to format a NTFS "with compression feature (-C)" will, from an observers viewpoint, work without a hitch. But, when trying to manipulate via Windows "format&create" it exposes a problem that can ONLY be rectified by going back to Linux, removing-saving the data, deleting the partition, then returning to Window to recreate it ... with compression.

I am hopeful to get BTRFS to provide the level of compression enjoyed over the years with NTFS compression and stability.

I am wondering what is the "best" or most convenient way to have Linux (Fatdog) to mount any BTRFS without the need to know the knowledge necessary for recognition of the compression that it is generated with. I never needed this for NTFS, because Linux has been "trained" to do this recognition when mounted without user knowledge or need.

So, now, as BTRFS continue its rise, maybe something can be done at the system level such that the mount command can do a rw mount and the I/O subsystem recognizes all things necessary in the same manner as it does for NTFS.

BTW: FD811 will boot using SuperGRUB2 with its ISO on a "non-compressed" BTRFS. The BTRFS was created with a folder BOOTISOS which contains the FD811 ISO. It is found on that partition, presented to user for selection, and it boots. Currently tested on a 64bit BIOS PC. I expect that the same will occur on a UEFI with secure-boot disable.

[user1111]

Cross link to setting up Fatdog with ZFS mirroring and compression where the mirror is on a different box/HDD viewtopic.php?p=5315#p5315

In my case, laptop connects via wifi, a old desktop is running fatdog as a data storage area, and where I'm mirroring laptop zfs contained data changes over to the desktop system using a samba based share.

As I understand it, once you set zfs compression type it only applies to new/changed files thereafter.

zfs set compression=gzip <poolname>

and that setting persists once set. Useful however is that you can modify that relatively easily, so for instance if you were about to copy in a large number or sized already compressed data/file, then you could turn the compression off beforehand, and then turn it back on again afterwards.

[Clarity]

Thanks everyone for useful experiences.

In NTFS for Compressed partition(s), it is set once at creation and the I/O subsystem and filesystem "know" how to handle data to/fro without user/admin knowledge. Thus, externally, it is handle like any other filesystem.

But I hear the "cautions" you share and will have to try to figure if there is Linux techniques for compressed BTRFS partitions; either manually or via other techniques to make I/O subsystem, for BTRFS, aware.

Any info I find, I will post back here to help others.

Thanks again, everyone