Formatting solid-state drives - latest thoughts?

[dogle]

Although I'm a (very) grizzled old dog now, I'm posting this in the Beginners' patch 'cos I suspect that some guidance from our friskier experts may be valuable to many of our newcomers - as well as for me.

Back in the day, there were many worries about the write-life of S-S storage - quite a concern if you wanted to backup VIS (Very Important Stuff) ... I much regret the ongoing demise of optical disks, which were pretty bulletproof with care, although optical drive failures could be a pain.

One hopes that S-S products have improved somewhat since then - at least the branded stuff - and now write-life is perhaps less worrisome, but formatting a new drive may still benefit from some forethought? As in:

- Choice of FAT32 / ntfs / ext2 / ext3 / ext4 / etc. etc. for a particular partition? Is ext4 now becoming the norm despite perhaps heavier journalling writes?

- Choice of formatting if you feel your RAM might be a bit short for today's bloated traffic, and want to set up a swap partition / swap folder / swap file(s).
- How to choose between these options?? (I presume that swap functions will entail heavy write traffic?).
- Having chosen, how to size ????

This is really a huge subject and I shall as ever be so grateful for informed responses ... with reasons!

[Marv]

Another grizzled old dog here. I have used Samsung EVO or the older Intel X-25 SSDS exclusively for a number of years now and still use FAT32 for my smallish boot partition (historic-grub4dos reasons) and EXT2 for my larger data/portables partitions (NEVER have had a failure there, slightly more space efficient than the journaled FS, and again, an old grizzled dog). Were I to start from scratch I would probably use EXT2 for the boot partition also as every now and then I forget and run up against the lack of symlink support in the FAT32 FS.

As an aside, zram is a swap system that uses selective compression in RAM rather than disk space. It is available and enabled by default in all the newer pups/kernels I run now (another jammypup64 E1, F96-CE_4, LxPupSc64 23.01 +2, and VoidPup64 22.02). It came over from the Dpups IIRC, seems well debugged at this point, and might be worth exploring. zramctl is the control program for it and if it is running you should be able to check the status etc in terminal:

Code: Select all

# zramctl
NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle       5.5G   4K   74B   12K       4 [SWAP]

Just my thoughts,

Edit: Forgot to mention, no windows here for a number of years, about 9 linux machines in the stable, hence no NTFS etc. Not a religious matter, just no need.

[mikewalsh]

@dogle :-

SSD "endurance" is a moot point now. It's improved to the point where Backblaze - one of the industry's largest cloud storage providers - are publishing their annual reports to state that SSDs are in fact now AS reliable as HDDs. And the data is still coming in; it's early days yet for their small-ish fleet of SSD server boot drives (only 2,500 ATM). So I don't think you need to worry about that aspect any more.

As for Ext2 vs Ext3 vs Ext4; the main reason for the preponderance of Ext4 to be found everywhere is not necessarily because it's "better"......it's mostly because it's the newest iteration of the Ext file format, and to most people, newest is ALWAYS best & most secure. Need I say more?

(I've used Ext3 for a decade with Puppy, and never had a scrap of trouble with it. YMMV...)

Swap? Rule of thumb says twice as much swap as RAM, but with these more modern systems now coming with 16, 32 or even 64 GB of RAM, the current thinking is that if you have plenty of RAM for your needs, then swap is not really necessary. It evolved back in the days when RAM & storage space were pretty small by today's standards. I started off with 128 MB of RAM and a 20 GB HDD. I now run 32 GB of RAM, and almost 5 TB of storage....better than 50% more RAM than the size of my original HDD, and around 250 times the amount of storage.

DON'T, whatever you do, put swap on an SSD. It does involve heavy read-write operations WHEN it starts to get used, and despite SSD reliability these days, flash memory "cells" DO still have a limited number of write cycles. Far better to put it on a secondary HDD, if possible.

Hope my witterings give ya SOME "food for thought"!

Mike.

[gychang]

I seemed to recall some folks recommending unformatted partition of 10% is a good idea in SSD. Apparently helps with longevity of the SSD...

[mikewalsh]

I seemed to recall some folks recommending unformatted partition of 10% is a good idea in SSD. Apparently helps with longevity of the SSD...

Mm-hm. I believe it came about originally due to older, earlier drives having controllers that were NOT that sophisticated in the early days, so the "10%" rule came about. Even with modern drives being very, very good at everything, it's still not a bad idea due to the way a solid-state drive works.

The old mechanical HDDs simply overwrote data in-situ, wherever on the drive the erase/re-write operation happened to be requested. SSDs, on the other hand, have their storage organised into "blocks" & "pages".....and because of the way NAND flash operates, even if the controller is only being asked to delete the contents of just one, single cell, the contents of a whole "page" have to be erased, then re-written. So the majority of the contents of the page that are NOT being deleted have to be copied off somewhere temporarily, the page erased, then the temporary data has to be re-written back to that page (minus the bit it was asked to originally delete, of course).

Bear in mind that there is also "wear-levelling" going on in the background all the time, which tries to ensure via specialised algorithms that ALL parts of the drive are being written-to equally. And then there's TRIM - "rubbish-collection" in plain English - which keeps the file-system structure clean as time passes. For all of this, the controller has to have somewhere it can move blocks/pages to, while it erases them, tidies them up, marks them as 'clean' & ready for re-use before finally rotating them back into service again..!

(All of which is happening literally thousands of times a second, of course...)

--------------------------------------------------

So; yes, definitely. Even with ultra-modern, ultra-high-speed PCI-e 5.0 Nvme SSDs, it still makes sense to help the drive's controller by making its job as easy as you can. It makes the drive run better, because it remains clean, tidy, and functioning as it was designed to. And it'll last longer before needing replacement, too! Which IS what you want, when all's said & done...

Mike.

[dogle]

Wow, thanks everyone for your truly illuminating comments.

Oh, how ‘moth and rust doth corrupt’ - Mike, can you believe I have got so far behind modern times still to have my own ‘rule of thumb’ for swap space to be half, rather than your double, available RAM .... I’m somewhat surprised to have Topview telling me that I’m using significant swapspace on my traditional HDD ... but I agree completely that swapspace on SSD is NOT a very good idea.

( For the record, ATM I’m still long running radky’s DPup Stretch - luvvit! - as my mainline Puppy, and on a box with still but 2GB RAM right now, and notional swapspace now about 256 MB, largely used .... Oh yes, I’m well overdue, and planning for, a thorough “upgrade” - cursed expression! - because things are starting to break now, particularly in respect of my now-superseded Palemoon browser).

So, I'm planning major revamps in which SSDs are likely to feature now, but I’d like to avoid thread drift here by raising some different further queries threads, especially in relation to Mike’s glorious work on ‘portables’, particularly browsers.

Mike, would you like me to continue to post this sort of stuff in the beginners’ patch, or do you feel it might be more appropriate elsewhere?

[bigpup]

I seemed to recall some folks recommending unformatted partition of 10% is a good idea in SSD

Modern SSD drives have this covered by actually having some of the memory cells set aside as extra not used.
You will not see these as available to use.

The SSD internal controller will use these extra cells, to replace any normal cells that go bad, as the SSD gets used.

So, extra unused memory cells are already built into the SSD.

All the unallocated space, that will be shown to you, can be used for partitions and formatted.

Now it is a good idea to never fully use up all available space in a partition.
Always have a little free space.
This allows wear leveling, some space to move stuff around, in the partition.

But with the modern wear leveling processes in SSD's.
I wonder if this is really needed.

My thinking is this.
If all space in a partition is being used.
Wear leveling is going to take longer to move stuff around, to level out memory cell usage.

Wear Leveling
NAND flash memory is susceptible to wear due to repeated program and erase cycles that are commonly done in data storage applications and systems using Flash Translation Layer (FTL). Constantly programming and erasing to the same memory location eventually wears that portion of memory out and makes it invalid. As a result, the NAND flash would have limited lifetime. To prevent scenarios such as these from occurring, special algorithms are deployed within the SSD called wear leveling. As the term suggests, wear leveling provides a method for distributing program and erase cycles uniformly throughout all of the memory blocks within the SSD. This prevents continuous program and erase cycles to the same memory block, resulting in greater extended life to the overall NAND flash memory.

There are two types of wear leveling, dynamic and static. The dynamic wear algorithm guarantees that data program and erase cycles will be evenly distributed throughout all the blocks within the NAND flash. The algorithm is dynamic because it is executed every time the data in the write buffer of the drive is flushed and written to flash memory. Dynamic wear leveling alone cannot insure that all blocks are being wear-leveled at the same rate. There is also the special case when data is written and stored in flash for long periods of time or indefinitely. While other blocks are actively being swapped, erased and pooled, these blocks remain inactive in the wear-leveling process. To insure that all blocks are being wear-leveled at the same rate, a secondary wear-leveling algorithm called static wear leveling is deployed. Static wear leveling addresses the blocks that are inactive and have data stored in them.

[mikewalsh]

@bigpup :-

Modern SSD drives have this covered by actually having some of the memory cells set aside as extra not used.
You will not see these as available to use.

The SSD internal controller will use these extra cells, to replace any normal cells that go bad, as the SSD gets used.

So, extra unused memory cells are already built into the SSD.

Hm. News to me, I admit.....but very much in line with the general "convenience-above-all-else-and-let's-make-things-as-easy-as-possible-for-the-user" attitude that is beginning to prevail within the R&D segment. Some of whom are actually starting to think a few steps ahead..!

----------------------

@dogle :- Nah, you're fine. You're by no means the first of the "older crew" that often use the "Beginner's" section for a lot of enquiries. This, and "Users" seem to get used as a catch-all for so many topics anyway; if we feel it's necessary, it's simple enough to move stuff around to a different location...

(It occurs to me that part of the reason your swap space seems to get used a lot MIGHT be due to having "swappiness" dialled-up too high. Many of the Debian/'buntu-based Pups have it set to "60" by default; it really should be down around "10" or so. At "60", it starts to use swap quite regularly.....whereas, at "10", it'll use your RAM much more of the time, and only uses swap when necessary.

RAM is there to be used. None of us buy it to sit there and admire it, do we?)

Mike.

[dimkr]

A modern Puppy will auto enable zram-based swap (in RAM, not in your SSD) if no swap partition is found, reduce swapiness to 5 (swap is unused until needed) and run fstrim periodically.

I have a laptop with a 7-8 years old cheapo SSD and Puppy runs great, even survives the I/O errors it has every once in a while. I use pmedia=usbflash so I can save twice if I had errors in the first attempt. Other distros just freeze and lose data.

[dogle]

Thanks again, fellas, for these really valuable insights!