Questions about 32-bit Windows vs 32-bit Puppies, EXT4 and filesystem limits....

[Grogster]

Hello everyone.

Having just about decided on what I plan to run on my new SAMBA servers(see my other thread - Remote Desktop stuff), I wanted to ask a simple question, but keep it away from that thread for reasons of isolation and simplicity.

In Windoze, a 32-bit system is limited to 4GB file-size with FAT32.
This does not seem to be the case in Linux - so long as you can get the basic system up and running, and you are using a supported Linux filesystem, those barriers do not seem to apply - even for 32-bit distros of Linux.

Do I have that right?

I have a 32-bit Fossa-based Puppy(see other thread mentioned above), that is 100% suiting my needs both in SAMBA, and in VNC remote-desktop access - but it is still in the 32-bit system.

Is it still OK to run 32-bit based Puppies these days, or should I ALWAYS be looking to have 64-bit running?
I used to use the standard Bionic64, but it did not have VNC support for remote desktop, so I have been playing with the FriendlyBionic one in both 32-bit and 64-bit. The 32-bit one does everything I need, is fast, and has no issues. The 64-bit one still has issues, but I expect they will be fixed soon enough.(again, see other thread.)

My point being, is it still OK to use 32-bit?
My main concern is LARGE hard-drives shared under SAMBA with a 32-bit Puppy. Bigger then 4TB. Say, 6TB, 8TB or 10TB using EXT4, but running a 32-bit Puppy.

I always format to EXT4, but I think that perhaps I am holding onto old ideas from Windoze, where you could not have a HDD with a files bigger then 4GB under 32-bit Windoze and FAT32. I think there was also a volume size limit under FAT32, but I forget what that was now.

I don't think that applies under Linux, yes?

That is my main question.

I have had a couple of glasses of wine tonight, so I might simply not be remembering the details correctly with respect to Linux-based filesystems.

[dimkr]

File systems that support files over 4 GB or more than 2^32 files use 64-bit counters, even if the kernel runs on a 32-bit architecture. The 4 GB limit for file sizes in FAT32 is a limitation of the FAT32 file system and not the driver or the CPU (because the size of a file is saved in a 32-bit integer, and cannot exceed 2^32), and you'll have the same limit even if you run a 64-bit OS and try to create a large file.

Same with protocols like SMB - some protocols use 32 bit numbers, so they're limited to 4 GB of data by design. Other protocol use 64 bit numbers, so they don't have that limitation.

Most proprietary applications nowadays are 64-bit only, and the few distros that still have a 32-bit flavor are either old or don't receive much developer attention. If you can use a 64-bit OS, do it: RAM consumption will be a little bit higher (not much), performance should be better (unless you're really low on RAM), security should be much better (because x86_64 has extra security features) and there's a bigger variety of 64-bit software. 32-bit OSs are used by a tiny minority of users, they're no longer a priority for OS vendors, and naturally, the kernel is gradually becoming buggier on old hardware or dropping support for it.

If you want to try two current and nearly-identical Puppy builds (64-bit and 32-bit) just to see how they perform, I recommend you to try https://github.com/vanilla-dpup/release ... 6_64-9.2.0 and https://github.com/vanilla-dpup/release ... -x86-9.2.0.

[dogcat]

FAT32 size limit is 2 Terabytes, some discussion at this thread from user Jafadmin
viewtopic.php?t=1049

One thought I had when reading your post, the partition sizes can be whatever you want, if you have an 8TB drive it can maybe be smallerized into multiple partitions.

I would think if your Fossa32 Samba server is networking ok with your windows system then no need to reinvent the wheel, unless of course there is some 64-bit only Linux software that you require.

[mikewalsh]

@Grogster :-

My view is this; if you specifically need certain up-to-date software, or intend to spend a lot of time on-line, then yes; you HAVE to go 64-bit. Most browsers are now 64-bit only, and almost all software is only being written for 64-bit systems.

If, however - as in your case - you're running file servers on a purely local network, with no access to the web involved.....and a 32-bit system will do everything you need, quickly & efficiently.....then I would stick with it. It's a good way of re-purposing older hardware, and keeping it out of the landfill.

Just bear in mind that 64-bit "tech" HAS been with us for not far short of two decades. It's definitely a 'mature' technology, and is thoroughly well-understood by now.

Dimkr is one of our top developers, and everyone here has enormous respect for him.....but it's only natural that devs are always going to recommend the very newest of everything. They are, after all, working with this stuff all the time, and are in a better position to judge the pros & cons than most. However, it's perfectly possible to mix systems of different architectures if they serve their purpose, and those systems are effectively isolated from each other.

For anything web-facing, you ought to go 64-bit. For anything else, if it does what you want, and you're happy with it.....

Mike.

[dimkr]

FAT32 size limit is 2 Terabytes, some discussion at this thread from user Jafadmin
viewtopic.php?t=1049

Don't confuse maximum partition size with maximum file size

[dogcat]

FAT32 size limit is 2 Terabytes, some discussion at this thread from user Jafadmin
viewtopic.php?t=1049

Don't confuse maximum partition size with maximum file size

Nope.

I always format to EXT4, but I think that perhaps I am holding onto old ideas from Windoze, where you could not have a HDD with a files bigger then 4GB under 32-bit Windoze and FAT32. I think there was also a volume size limit under FAT32, but I forget what that was now.

[mikewalsh]

@Grogster :-

There is ONE other item worth mentioning. Bill (rcrsn51) some years ago published a very simple, Python2-based HTTP web server. This lets you view, download & upload between two boxes on a local network via the browser.

Simplest way I've ever come across of sharing stuff locally......because it's architecture-independent.

You can find it here:-

viewtopic.php?p=57550#p57550

Just remove the 'fake' [.gz]. Then click to install. You'll find a Menu entry under Menu->Network. The post contains a link to the original thread on the old Forum.

Mike.

[ally]

for what its worth I run openmediavault on a raspberry pi to serve 12 HDDs

runs fine

[Grogster]

Thanks very much everyone, for your replies.

So, lets take an example: FriendlyBionic 32-bit Puppy - everything is working great.
If I install a new HDD that is, say, 8TB in size, but I am using EXT4 filesystem, will I need to break it up into 2TB slices, or will Puppy with it's EXT4 system, be able to bypass any 32-bit Windoze type limit, and format the entire drive to one 8TB volume?

This is the bit that is confusing me the most - trying to work out if the 2TB FAT32 volume limit, is only a Windoze thing for FAT32, or is it also still a limit for a 32-bit Linux?

I get the impression from dimkr, that if it's 32-bit - even Linux - it will be still limited to 2TB volumes, as that is the maximum size that can be addressed with 32-bit. Do I have that correct or wrong? (i have a feeling I have this wrong)

If I am going to be limited to 2TB volumes even with a 32-bit Linux, then I will have to move to 64-bit, as these days, as I upgrade my server HDD's, I am buying bigger and bigger volume ones, usually 6TB or bigger, and tend to format that drive all as ONE volume. I was using standard Bionic64 for those though, so I don't want to bugger things up by moving down to 32-bit etc.

Hopefully, you can see what I am getting at.

As far as filesize is concerned, 99% of all my files are way less then 4GB each, but there might be the occasional OS image file that exceeds 4GB - will that work in EXT4 under 32-bit Linux, or am I chasing my own tail here?

[mikewalsh]

@Grogster :-

Hm. That was a bit misleading, wasn't it? Whoever mentioned it - was it dimkr? - really threw the cat among the pigeons!

The FAT32 file-system was invented by Micrososft.

The 4GB limit applies to individual file-sizes. The 2TB limit applies to storage partition sizes. These are baked into the FAT32 standards/specs, and have been there ever since its beginning, I suspect.

This probably only got mentioned because the subject happens to be a sticky at the top of the Beginners section.....and because you're 'mixing' Windows & Linux here. In all honesty, FAT32 tends to be used mainly for flash storage.....USB flash drives, and the like.....and mostly because of its low overhead. It's the universal standard all flash storage is formatted with at the factory.

Believe it or not, the FAT standard - File Allocation Table - has been around, in one form or another, since the mid-70s.....when it was originally developed for formatting floppy disks used in DOS-based systems. And as disk capacities increased, the original became FAT12.....followed by FAT16, then finally FAT32.

https://en.wikipedia.org/wiki/File_Allocation_Table

The number refers to 'cluster' sizes, which had to get larger to cope with ever-increasing pools of data. (Don't ask what 'clusters' are, because the average user has no conceivable need to know what they are or even how they work).

And FAT file-systems - like all Windows-based types - still need regular "defragging". Linux file-systems, on the other hand, don't.......not until they get over 95% full. Under normal usage, it's not necessary; due, apparently, to the design of the file-system & the much more efficient way it handles data.

---------------------------------------------------

Top & bottom of the matter boils down to this:-

• Modern Windows insists on NTFS, and isn't happy with anything else.

• Linux requires its own file-systems - Ext2, Ext3 or Ext4 - to function as it was intended to. (Ext2 was an older standard; it didn't use "journalling", which helps with file-system recovery in the event of meltdown. Ext3 & Ext4 do.)

Why anybody in their right mind would want to format a disk-drive of any size with FAT32 is beyond me. (Unless they had a very special use-case for it, of course...)

Forget about FAT32 & 2TB limits. They are not necessary here. They're not even relevant. They're an unnecessary distraction.....and by rights should not have even been mentioned.

If you intend to install new drives to any box that's going to run Linux, then go with Ext3 or Ext 4. With those, you can't go wrong.

Hopefully, the preceding "waffle" will clarify some of your concerns.

Mike.

[bigpup]

If I install a new HDD that is, say, 8TB in size, but I am using EXT4 filesystem

This may not be applicable if you are setting up a server.

Something to think about.

If you make one partition on the 8TB drive, using all the free space on it.
If anything happens to the file system (the ext4 format).
You have lost use of the complete drive.

You will have a hard time running any kind of repair program to fix the corrupted file system.
Why.
It will take hours for the program to run a check and repair on that big a file system, covering 8TB of space.

If you partition the drive into several partitions formatted ext4.
The partitions are smaller in size and easier to check and repair.

If something goes wrong with one partition.
You only loose use of it.
The other partitions still work.

Think of it this way.
Is this the best way to put a book together covering everything or have it broken down into a bunch of smaller books.
.

short version.jpg (19.54 KiB) Viewed 951 times

[dimkr]

I get the impression from dimkr, that if it's 32-bit - even Linux - it will be still limited to 2TB volumes, as that is the maximum size that can be addressed with 32-bit. Do I have that correct or wrong? (i have a feeling I have this wrong)

FAT32, the file system, uses 32-bit counters for file sizes, and that is why files are limited to 4 GB. This file system, by design, is also limited to 2 TB volumes. No matter if you use a 32-bit or 64-bit OS, as long as you use FAT32 partitions, you'll have the same limits.

But if you use some other file system, which doesn't have these limits and uses 64-bit numbers to represent things like file size (say, ext4), you'll be able to use files larger than 4 GB even from a 32-bit OS, because even 32-bit OSs can read 64-bit and 128-bit integers from disk.

[wizard]

@Grogster

Don't know if you saw that Friendly-Fossa64 rc4 has fixed the remaster samba bug that disabled Samba Simple Management. Should work fine now for both your VNC and file sharing server. Find it here: viewtopic.php?t=4681

Thanks
wizard

[williams2]

A block device (can be hardware or virtual (in ram))
consists of blocks of data.
Formatting a block device writes data to some of the blocks of a block device, so that the block device can be used to hold/contain a file system, which means that "files" and directories can be accessed by name, and copied, pasted, moved, read, written, edited, etc.

A physical hardware device probably has a size that each data block has. For example:

Code: Select all

fdisk -l
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt

This tells you that this hardware block device /dev/sda has blocks of data that are 4096 bytes (4k).
which is typical for a hard drive. A long time ago, (in computer years), hard drives were much smaller, and the data could be read and written 512 bytes at a time.

fdisk is saying that the hard drive has physical data blocks of 4096 bytes,
but the computer is pretending that each data block is 512 bytes.
That means that you need to have 8 times more addresses to address each 512 byte block of data than you would need to address each 4096 bytes data block.

32 bit addresses can hold rpn 2 32 ^ or 4294967296 (4GiB)
64 bit addresses can hold rpn 2 64 ^ or 18446744073709551616 (a lot bigger)

You can format a partition using 32 bit addresses like this:
mkfs.ext4 -n -b 4096 -O ^64bit /dev/sdb1

You can format a partition using 64 bit addresses like this:
mkfs.ext4 -n -b 4096 -O 64bit /dev/sdb1

You can configure mkfs.ext4 to use either 64bit or ^64 bit (not 64 bit) in the file /etc/mke2fs.conf, like this:

Code: Select all

ext4 = {
features = ^64bit,has_journal,extent,huge_file,flex_bg,metadata_csum,dir_nlink,extra_isize

or like this:

Code: Select all

ext4 = {
features = 64bit,has_journal,extent,huge_file,flex_bg,metadata_csum,dir_nlink,extra_isize

A byte is 8 bits, for example, 00000000 and 11111111. A byte (8 bits) can be 256 different numbers, (or whatever states the byte is holding, it does not need to be numbers)

The point is that Puppy, or any Linux OS, can make an ext4 file system using 32 bit addresses or with 64 bit addresses..

Also, by default, ext4 uses a journal, but the journal only keeps the metadata of files and directories, that is, the names of the files and dirs, and ownership and permissions, but not the actual data (not the data that was in the file). With a journal, a fsck (file system check) will take maybe 0.1 seconds. Without a journal, a fsck might take hours.

[Grogster]

dimkr said:

But if you use some other file system, which doesn't have these limits and uses 64-bit numbers to represent things like file size (say, ext4), you'll be able to use files larger than 4 GB even from a 32-bit OS, because even 32-bit OSs can read 64-bit and 128-bit integers from disk.

Perfect. Thanks. That has clarified it for me. I never use FAT32 HDD volumes in Linux, unless I happen to be copying from that, to an EXT4 Linux drive.
ALL my Linux HDD volumes, I always use EXT4 and have for the past few years.
Just to clarify that for members thinking I am perhaps trying to format 8TB drives in FAT32 under Linux or something.

It gets complicated very fast, doesn't it!

@ wizard - thanks for the updated link, I will check that thread now.

[Grogster]

bigpup said:

If you make one partition on the 8TB drive, using all the free space on it.
If anything happens to the file system (the ext4 format).
You have lost use of the complete drive.

You will have a hard time running any kind of repair program to fix the corrupted file system.
Why.
It will take hours for the program to run a check and repair on that big a file system, covering 8TB of space.

I thought that fsck was pretty quick on EXT3 and EXT4, even for big drives?
I certainly agree, that if you have to scan the whole drive, those huge drives will indeed take hours and hours.
I used to run into that issue with NASlite2 which used ext2, and occasionally it would want to do a fsck - I usually replaced the 's' with a 'u' when that happened, as even then on 2TB and 3TB volumes, it too fscking ages....

I thought ext3 and ext4 didn't have that issue?

In any event, I run two servers. One is the main unit, the other is outside in the shed and is a backup of the first one, so that if a drive dies and I lose what's on it, I can fit a new drive, then copy the contents back from the backup server. Currently, I do all my updates to the backup server manually, but that is becoming a pain, so my next thread will be on how I setup something like Rsync etc, to schedule backups from the main server, to the one in the garage.

There are also various flavours of RAID to consider, and some NAS boxes use ZFS, but I find that good as ZFS seems to be, it is extremely CPU intensive from what I have read.

But I digress....