What Causes Thumbdrive Corruption?

[JusGellin]

I just noticed something when using a thumb drive that has an activity led to copy some large files using BookwormPup64_10.0.6.
If I copy something big like a backup copy of the puppy save file, it starts blinking.
This continues even after the rox file manager shows the copy has been completed.
The blinking continued for 4 minutes more.
The laptop I'm using has USB 2 .0 and 3.0. But I'm not sure what the two connections on the laptop are using.

It looks like the rox file manager is misleading when it shows the copy operation is complete.
This could account for sometimes when using a USB thumbdrive, that it becomes corrupted
and I have to reformat it to use it again.

A lot of the newer thumbdrives don't have the indicator, so this can unknowingly happen.

Now that I know this is happening, I saw that running the terminal command sync is supposed to indicate when it is safe to remove
the thumb drive.

But I noticed that when I did a large copy and ran sync that even that can be misleading because it sometimes stops running
while the flashing continues. I can restart the sync and eventually after several minutes the flashing stops and sync no longer
runs.

I tried doing the same thing with xfe file manager, but it give permission denied when copying the save files.

Is this a normal thing for the rox file manager to act that way? Wow! I guess this accounts for sometimes I think I am having problems with my thumbdrives.
I even see this happening when I run a Bookworm iso from a usb ventoy. I've had the ventoy usb stick get corrupted a couple of times as well.

Please let me know if I'm seeing this right and how I should deal with this properly.

Thanks

[mikeslr]

Just confirming what you've observed.

Copying is always a two step process. You just don't notice it as frequently when copying to a fast hard-drive. The copy is first written to RAM, then from RAM written to the target media. Rox just shows the first step. And if you remove a USB-Key being copied to before the copy has been completely written you can end up with a corrupted thumb-drive.

Rsync should prevent premature removal of the thumb-drive as that application will continue to run until completed. I believe the same is true if you try to unmount the thumb-drive or even restart-x.

[JusGellin]

Thanks @mikeslr
I kind of knew that before, but really experiencing it helps me understand it better.
You explanation about it going to ram is great.
That information about using rsync looks to be a good way for copying. I've learned quite a bit about how useful rsync is.

My learning - be more careful with thumbdrives!

[dimkr]

Some file systems are more prone to corruption compared to others: those without journaling, like FAT32. Puppy supports automatic repair of file system corruption if you boot with pfix=fsckp, but only for ext4, and very recent versions also repair F2FS.

[JusGellin]

Puppy supports automatic repair of file system corruption if you boot with pfix=fsckp, but only for ext4, and very recent versions also repair F2FS.

So that's what that setting is.
Thanks

[williams2]

sync used to block (wait) until all of the blocks of data were all copied.
But it would stop blocking (stop waiting) the instant that the last block of data was ready to start copying.

That is, sync unblocks before the last block of data starts to copy.
I do not know if this is a bug or a feature.
For many years, this was the way sync worked.
I don't know if the bug or feature still behaves that way.

In a shell script, I always use 2 syncs.
The first sync waits for all but the last block of data to be processed.
The second sync waits for the last block of data to be processed.

Block data devices like hard drives often have ram buffers built into the hardware.
So, even after the operating system has finished copying blocks of data,
internally,the device might still be writing data from the devices internal buffer.
The operating system would not know, it treats the device as a black box.

In a script, after a sync;sync it might be safer to sleep 5 or 10 seconds before displaying a "safe to unplug usb drive" message.

[wizard]

@JusGellin

wizard's rule #12 "All computer hardware is not created equaL"

This is very true for thumb drives, my testing has shown many have write speeds that are a fraction of even USB v1. Also why I only buy Sandisk thumbdrives and speed test every one when received.

When copying a large amnount of data to a thumbdrive I also right click its icon on the desktop and choose "Unmount". The drive will not unmount until the copy operation is complete.

wizard

[williwaw]

The laptop I'm using has USB 2 .0 and 3.0. But I'm not sure what the two connections on the laptop are using.

might have to get out the magnifying glass to identify......
https://endurtech.com/reference-guide-t ... ort-icons/

[JusGellin]

@williwaw
I found the manual - Both USB's are 3.0
But the symbol indicates USB 2.0

@JusGellin

wizard's rule #12 "All computer hardware is not created equaL"

This is very true for thumb drives, my testing has shown many have write speeds that are a fraction of even USB v1. Also why I only buy Sandisk thumbdrives and speed test every one when received.

When copyding a large amnount of data to a thumbdrive I also right click its icon on the desktop and choose "Unmount". The drive will not unmount until the copy operation is complete.

wizard

The unmount looks like a good trick. How do you do a speed test on them?

In a script, after a sync;sync it might be safer to sleep 5 or 10 seconds before displaying a "safe to unplug usb drive" message.

How can a script be written to do this? I wasn't sure how to make it indicate it was done.
This is as far as I got:

Code: Select all

#!/bin/sh
#
# wait for sync
sync
sync
sleep 5

Thanks to all of you

[williams2]

Code: Select all

# printing a message to the terminal
echo "safe to unplug drive"

# popping up a message in a window
gxmessage "safe to unplug drive"

If you don't have gxmessage installed, you can use
yad or gtkdialog, etc.

A script to erase the unused bytes in a savefile with zeros:
dd shows the speed of the writes to the drive
(dd in some Pups does not support display of progress status)

Code: Select all

#!/bin/sh
echo 'zeroing save file ...'
dd status=progress if=/dev/zero of=/root/dd.bin
sync
sync
rm /root/dd.bin
sync

[JusGellin]

@williams2
Thanks a lot.

[wizard]

@JusGellin

How do you do a speed test on them?

I usually use Crystaldiskmark (portable) which runs under MS Windows, but you can use our friend @mikewalsh speed test from here: viewtopic.php?t=2836

wizard

[JusGellin]

you can use our friend @mikewalsh speed test from here: viewtopic.php?t=2836

wizard

Really nice and perfect to use

Code: Select all

# popping up a message in a window
gxmessage "safe to unplug drive"

This works well. I found that I needed to add a couple more sync commands because they would stop running on a large file.
Then it was ok.

Thanks

[wizard]

@JusGellin

If you use XFE file manager, try this:

XFE file manager in two pane mode.

-put the source directory in one pane and the destination USB in the other (sdb1 in this example).
-when you start a copy operation the File Copy progress window will open

copy.jpg (13.98 KiB) Viewed 649 times

-the progress window will close (but the copy may not be done)
-in the mnt directory, right click sdb1
-choose Unmount
-the Unmount window will open and remain open if the copy is not complete

umount.jpg (43.98 KiB) Viewed 649 times

-when the copy is done the Unmount file system window will close and the Success window will open

done.jpg (48.81 KiB) Viewed 649 times

It's now safe to remove the USB.

wizard

[wizard]

@JusGellin

Here's a little script created with YAD and based on the suggestions by @williams2 ..

Start your copy, then run copydone.sh. You'll get a "In progress" message and then a "Copy complete" message.

This should work with any file manager and any media type.

wizard

[JusGellin]

@JusGellin

Here's a little script created with YAD and based on the suggestions by @williams2 ..

Start your copy, then run copydone.sh. You'll get a "In progress" message and then a "Copy complete" message.

This should work with any file manager and any media type.

wizard

Thanks
This works really well. I like the pop-ups that let you know what is happening and when it is complete.
Plus it keeps the thumb drive mounted.

When I tried the dismount trick, the only thing that let you know it completed was the file manager for the thumb drive disappears and
the icon for it comes back up with it dismounted.

@JusGellin

If you use XFE file manager, try this:

I tried this and it works, but the pop-up messages work better for me.

[mikewalsh]

@JusGellin :-

I, too, use the 'Unmount' trick with the drive icon. It always works, because the drive icon will not 'change state' UNTIL sync is complete...

I learnt this one a LONG time ago!

(*shrug...*)

Mike.

[williams2]

sync would normally work like this:

Code: Select all

# first, copy file1 to file2
cp file1 file2

# then, finish writing to hardware
sync
sync

gxmessage "file has finished copying"

That is, copy the file, then sync
Sync is not intended to work in parallel with a copy command.
It might not work as expected.
Sync in parallel with copy could make your hard drive work harder and might cause thrashing.

[JusGellin]

ummmm!
Things just aren't all that simple are they?

Does setting the dismount to see when it is available have any effect as well?

Thanks for the information to think about.

[williams2]

A drive should not unmount until all the data has been written.
Or there could be file system corruption

Usually, you want to unmount a drive, then unplug it.

You could click a sync script just before unmounting a drive.

[dimkr]

Just unmount the drive before you unplug it. The kernel flushes unwritten changes when you unmount.

[JusGellin]

When I started just using Linux, I was impressed that it seemed on my PopOS system that I could just remove the thumbdrive without unmounting it.
It seemed to work well, but I wasn't using very large files. But sometimes I noticed "something" seemed to corrupt the thumbdrive.
The file manager I was using didn't seem to have an unmount so I thought that Linux just takes care of this.

Boy, was I wrong when I went to using puppy and tried transferring large files -- more thumbdrive failures!
Now from this topic going through this, I really see the importance for unmounting the thumbdrive first as indicated in this discussion - thanks all.

I went back to look at the PopOS file manager, and wouldn't you know it, I found the unmount!! I sure mislead myself thinking this wasn't necessary.
Now I'm a believer!!

[wizard]

@williams2

Sync in parallel with copy could make your hard drive work harder and might cause thrashing.

Installed "dstat" from the repo which can display drive read/write speeds and then ran copy operations between the internal HDD and a USB HDD. As you can see running sync in parallel with the copy operation slows it down by about 15% (right column is write speed). Might be the disk thrashing you suggested.

no-sync.jpg (5.91 KiB) Viewed 473 times

with-sync.jpg (5.97 KiB) Viewed 473 times

As a result, I wrote another script that uses dstat to indicate when a copy process is done. Take a look at this post for details: viewtopic.php?t=12022

Thanks
wizard

[williams2]

harder and might cause thrashing

Maybe a bit over stated.

The OS is constantly flushing buffers in the background, in parallel, whether the sync command is used. or not.
It probably knows what it's doing.