Overcoming USB copy delay

[greengeek]

One of Puppy's original benefits was the ability to protect USB drives by accumulating write data in memory temporarily rather than writing repeatedly direct to the usb drive - which would potentially shorten the life of the drive.

However - the downside of this has been that the write process has to play "catchup" at inconvenient times - such as at shutdown or when trying to unmount drives etc, which means that the user had to wait for the data to be transferred out of memory - even if it took minutes to transfer a couple of gigs across a slow bus onto a slow usb stick.

I experienced data corruption at times which i attributed to my having pulled usb sticks out of the PC before such data transfers had finished. The problem was that the system would say that the transfer had finished when in fact it was still underway.

If a usb stick had an LED which flashed to indicate the data transfer i could tell that i should wait - but with no LED it was pot luck.

These days I still experience the problem where the data is still transferring long after it appears that the copy has ended - but now i know to use the "sync" command in a terminal to force the data to transfer out of memory - and i wait for the terminal prompt to re-appear before pulling the stick.

What i am wondering is if there is some way to force any given write to begin immediately and tell me truthfully when it has finished?

Let me offer an example: lets say i have a 3GB video file in /root and want to copy it to a usb stick - i can start the copy (rox window to rox window) and hope that the transfer will start immediately but really i have no idea when the system will start to flush the data from memory (which is where /root lives...).

This is especially problematic if i am generating other NEW video files at the same time and filling up /root and approaching the limit of my personal save space (which is a total of system RAM + swap partition).

So my question is: is there any way to overcome the inherent write delay and tell the system that i want a specific copy to start NOW and complete as soon as possible (and that i dont care about preserving usb stick life)?

I dont want to have the uncertainty of thinking that my RAM is running out simply because the system is not planning on emptying RAM writes until it decides that some random timer will run out in x minutes and force the usb writes then.

Hope that makes sense.

I am guessing that there is some PUPMODE that will correct this anomaly - although i really want this to be under my manual control rather than treating my usb sticks as HDDs and writing at all times. I seem to be currently running PUPMODE 5 (system boots from usb and has no savefile).

[taersh]

I think the smartest solution would be to modify Rox filer (and/or any other file manager) to do the sync immediately after the copy is done AND before the copy dialog/form disappears. So, the guys who already had worked on refining Rox filer would be the people to ask for this.

[garnet]

So my question is: is there any way to overcome the inherent write delay and tell the system that i want a specific copy to start NOW and complete as soon as possible (and that i dont care about preserving usb stick life)?

Use the "sync" mount option.

Code: Select all

mount -o sync /dev/sdb2 /mnt/sdb2

Assuming your usb stick is /dev/sdb2.

[bigpup]

I seem to be currently running PUPMODE 5 (system boots from usb and has no savefile

I am a little confused.
Is this the way you always run?

Pupmode 5 only uses a ramdisk for save.

After you actually make a save file/folder and boot using the save.
Normally, a USB install will be running in pupmode 13.

How much RAM is in the computer?

I wonder if your issues, are trying to do too much, with too little RAM.

I hate to say it, but Puppy is not that good at warning about running out of RAM.

Also, if you have a swap file or partition.
Running in pupmode 5 is going to cause use of the swap, if the ramdisk is running out of RAM.
There is also the setting that controls when swap is used. At what percent level of RAM usage.

Also, you could have a large usage of RAM cache going on.
The cache would have to clear, to give more free RAM, for the ramdisk to use.

[mikewalsh]

I seem to be currently running PUPMODE 5 (system boots from usb and has no savefile

<snip>....How much RAM is in the computer?

I wonder if your issues, are trying to do too much, with too little RAM.

I hate to say it, but Puppy is not that good at warning about running out of RAM....</snip>

Tell me about it..! And this is one of the primary reasons I've upped the original 4 GB in this new HP desktop all the way to 16 GB. I had enough years of having to watch the amount of RAM in use with the old Compaq's 3 GB..... Now, I can do whatever I want, secure in the knowledge that I'm never getting anywhere near the system's limits.

I may yet go to 32 GB. Haven't decided yet...

(*shrug*)


Mike.

[Jafadmin]

As far as I know, there are no OS's that can guarantee data integrity if you remove removable media without dismounting first or using the "Safely remove device" option.

As a matter of course, I always issue the 'synch' command and allow it to complete before removing storage devices.

[garnet]

As far as I know, there are no OS's that can guarantee data integrity if you remove removable media without dismounting first or using the "Safely remove device" option.

DOS ヽ(♡‿♡)ノ

[greengeek]

Use the "sync" mount option.

Code: Select all

mount -o sync /dev/sdb2 /mnt/sdb2

Assuming your usb stick is /dev/sdb2.

Thanks - just tried this idea and got unexpected results.

If i mount async it takes about 10 seconds to copy a 100MB file to the stick (and the copy dialog disappears halfway through the actual copy)

If i mount using sync it takes about 3 or 4 minutes to do the copy - but the copy dialog stays open until the writes have actually finished and the LED stops flashing.

Not sure what to make of this - more testing required.

[greengeek]

I think the smartest solution would be to modify Rox filer (and/or any other file manager) to do the sync immediately after the copy is done AND before the copy dialog/form disappears.

Yes that would be a significant benefit.

[greengeek]

I am a little confused.
Is this the way you always run?

Pupmode 5 only uses a ramdisk for save.

After you actually make a save file/folder and boot using the save.
Normally, a USB install will be running in pupmode 13.

Yes this is how my system has been running for the last 5 years. (It is probably a distraction if i explain how my pup is set up but basically i used jrb's technique of swapping the zdrv with the main pup.sfs so that my personalisations are in the zdrv and as a result i don't need a savefile)

How much RAM is in the computer?

I have 2GB of RAM - but a 6GB swap partition. What i find is that the system uses the swap partition as if it was RAM - so that i can copy a 3GB video file into /root and manipulate it quite happily with ffmpeg or similar (which can potentially double the RAM requirements and runs close to stalling the pup once the swap is all used up...). Seems as if i could happily add 20GB of swap rather than trying to double the internal RAM - so i never bothered to increase the internal.

I wonder if your issues, are trying to do too much, with too little RAM.

Yes the problem with running out of free RAM is caused by having too little memory - i know that i can get past that problem by adding more sodimms, or by adding more swap partition space, or by limiting the size of video files that i am manipulating. But the problem with puppy not writing data OUT of /root as soon as i start a copy is a different problem. Sometimes i just want to ditch some data out of root and i want it gone now rather than waiting for it to decide it is ready to run a sync and flush the requested "copy" out of memory.
(I can manually force it with the "sync" command but i would like to make it happen automatically as soon as i command Rox to do the copy)

I hate to say it, but Puppy is not that good at warning about running out of RAM.

I agree. I read in various places that "Linux is good at managing swap functions" - yet i don't see that on my system. Never any warnings, just dead in the water once the bottom rung of the "partview" icon starts flashing red.
But in all fairness - i should have noticed the partview icon stepping down to orange - and got my other "root file dump" copies out of the way sooner.

Also, if you have a swap file or partition.
Running in pupmode 5 is going to cause use of the swap, if the ramdisk is running out of RAM.
There is also the setting that controls when swap is used. At what percent level of RAM usage.

Swap usage has always been an oddball thing to me. It almost never looks as if swap is in use - but then if i copy a massive file to /root (or other non-HDD directory) it seems happy to use the swap to hold that file as if it was just extended RAM. Seems all or nothing. Like swap is "shut for business" until the exact moment when puppy is about to die - then the swap door opens and all is good. (Till swap hits 99% full and puppy goes fully tits up).

Also, you could have a large usage of RAM cache going on.
The cache would have to clear, to give more free RAM, for the ramdisk to use.

Got me there - cache is a mystery to me.

[garnet]

If i mount using sync it takes about 3 or 4 minutes to do the copy - but the copy dialog stays open until the writes have actually finished and the LED stops flashing.

Isn't that what you want?

[greengeek]

If i mount using sync it takes about 3 or 4 minutes to do the copy - but the copy dialog stays open until the writes have actually finished and the LED stops flashing.

Isn't that what you want?

Well i do want the copy dialog to stay open until the copy is finished - but i don't want the copy to take ten times as long in syncmount as it did in asyncmount.
(The actual data transfer took ten times as long using sync mount)

[garnet]

Well i do want the copy dialog to stay open until the copy is finished - but i don't want the copy to take ten times as long in syncmount as it did in asyncmount.
(The actual data transfer took ten times as long using sync mount)

What is the filesystem on the usb stick? I hope it is not ext3 or ext4?
How much is the total time to: copy in asyncmount + run "sync" in terminal and waits until the "sync" is completed? I think this total should be the same as copy in syncmount, no?

[greengeek]

I've upped the original 4 GB in this new HP desktop all the way to 16 GB. I had enough years of having to watch the amount of RAM in use with the old Compaq's 3 GB..... Now, I can do whatever I want, secure in the knowledge that I'm never getting anywhere near the system's limits.

I may yet go to 32 GB. Haven't decided yet...

Try loading 16GB or more of video files into /root and see if you can pick the exact point when the pup clags up.
Then repeat the process after plugging in a fast 16GB usb stick formatted as swap and see if this allows you to treat the system as if it had double the RAM.

I would be keen to see the result.
(Obviously usb swap is slow - but at least it should stop the total clag-out of memory processes).

I think i will need to make it a habit to plug in extra swap on-the-fly when i notice my partview going orange.

[taersh]

I have also 16 GB of RAM. Plus 11 GB of Swap partition. When the RAM usage is at around 15 GB and the swap is used at several GB Puppy gets very slow. Even simple tasks like moving the mouse pointer doesn't work properly any longer.

From my experience swap is a fake. Ok, it may save the computer needing to be shut off by pressing the reset button, but any useful use isn't possible then until the RAM usage gets much lower. When the RAM returns to default usage by a few GB around 4GB the swap doesn't return to zero. It keeps several hundred MB up to 1.5 GB.

[garnet]

When the RAM returns to default usage by a few GB around 4GB the swap doesn't return to zero. It keeps several hundred MB up to 1.5 GB.

Code: Select all

echo 10 > /proc/sys/vm/swappiness

(⌒‿⌒)

[greengeek]

As far as I know, there are no OS's that can guarantee data integrity if you remove removable media without dismounting first or using the "Safely remove device" option.

I get caught by the deception of the rox "copy" dialog disappearing before the data is actually written out. In part this is because the data is sometimes only being written to RAM (and the auto sync will occur later), but sometimes if the file is big enough the rox dialog closes half way through the actual data transfer.

I believe there are some Pup modes where data writing is not delayed - it starts as soon as the copy dialog is begun.

I hope i can access that functionality without experiencing the super slow data transfer i saw when i tried Garnets suggestion of the sync mount.

(A bit of Googling suggests sync mount is pretty much never used - so i doubt that is the mechanism Puppy uses when in a pupmode that writes immediately to disk)

[taersh]

I have a script running at boot that already sets the swappiness to 10.

Code: Select all

sysctl vm.swappiness=10

Is that different to

Code: Select all

echo 10 > /proc/sys/vm/swappiness

???

[garnet]

I get caught by the deception of the rox "copy" dialog disappearing before the data is actually written out. In part this is because the data is sometimes only being written to RAM (and the auto sync will occur later)

Not "sometimes", but "always", unless one uses "sync" mount option.

I hope i can access that functionality without experiencing the super slow data transfer i saw when i tried Garnets suggestion of the sync mount.

You do get the guarantee that your data is on the stick, though (⌒‿⌒)

(A bit of Googling suggests sync mount is pretty much never used - so i doubt that is the mechanism Puppy uses when in a pupmode that writes immediately to disk)

PUPMODE=5 ?

I have a script running at boot that already sets the swappiness to 10.

Code: Select all

sysctl vm.swappiness=10

Is that different to

Code: Select all

echo 10 > /proc/sys/vm/swappiness

???

Same (⌒‿⌒)

[greengeek]

You do get the guarantee that your data is on the stick, though (⌒‿⌒)

True. And that is the important outcome. But is that the only impact i wonder? It seems to me that the sync mount must have some other impact that causes a slower data transfer rate to the usb stick. I want to investigate that further on other more standard pups. May have just been a temporarily anomaly.

[greengeek]

I thought i would try to make some accurate comparisons copying to my usb stick using both sync mount and async mount - using a different puppy and a different method to my previous test.

In order to avoid any contributing factors from Rox-Filer i wrote the following script:

Code: Select all

#! /bin/bash
gxmessage "   Click ok to start copy now.
   You will receive a confirmation when copy is finished.
   You will also be advised of start and stop times."
STARTDATE=$(date)
cp -f /root/test.iso /mnt/sdd1
sync
ENDDATE=$(date)
gxmessage "   Copy is finished.
   Start time was $STARTDATE and end time was $ENDDATE
   Subtract start time from end time to calculate elapsed time for copy"

I set it up to copy a 100MB file (i used tazpuppy-5.0-beta-30.iso renamed as test.iso) to sdd1, and i used it after mounting sdd1 with each of the following commands:

# mount -o sync /dev/sdd1 /mnt/sdd1
and
# mount -o async /dev/sdd1 /mnt/sdd1

The sync mount copy took 3 minutes and 28 seconds.
The async mount copy took only 11 seconds.

In each case the usb stick LED flashing stopped at the same time as the script ended - so i believe these figures measured accurately the time taken to transfer the data.

I am not sure what exactly is happening with sync mount. Keen to hear if other peoples pups and usb sticks show similar characteristics.

[greengeek]

Just want to post a link to a comment on the old forum by Flash - explaining that "copyfast" uses rsync and supposedly keeps up a window monitoring the copy until it is actually finished.

http://murga-linux.com/puppy/viewtopic. ... 57#1034582

This sounds like something i could benefit from so i gave Don570's rightclick version of copyfast a try and it is promising - although instead of a copy window it seems to pop up a yafsplash or similar for the duration of the copy - but that message disappears behind other windows and drops out of the users view.

Maybe i can modify it to stay on top.

[garnet]

I found this article about disk caching. I hope it helps.
better-linux-disk-caching-performance-vm-dirty_ratio

[Flash]

Have you tried Copy-Fast? It uses rsync, I think. Anyway, it adds an item to Rox's right-click menu that immediately copies to the destination without any caching or such.

[Jafadmin]

There is no magical way to copy files. I/O is I/O. All that can be modified is how the process appears to the user.

Do some benchmarking from the console.

Code: Select all

cp test.file /mnt/sdb1;sync

That will give you the sync blinking cursor until all the buffering is spooled out to disk. Try it without using sync.

You never see the problem copying on fixed disks. It just buffers in the background as you go on your merry way with what you wanted to do. The buffered data finishes writing out in the background.

The problem only becomes an issue with removable media.

[greengeek]

Have you tried Copy-Fast? It uses rsync, I think. Anyway, it adds an item to Rox's right-click menu that immediately copies to the destination without any caching or such.

Hi Flash,
yes i did try copyfast and it definitely reports accurate timing of "copy finished" - please see my post a couple of posts above your one. I think Copyfast and rsync is the answer i needed - although Copyfast uses a yaf-splash advisory dialog that does not stay on top of other windows and therefore risks leaving me in the dark just as much as Rox Filers copy dialog.

If i can solve the issue of the dialog window disappearing from view Copyfast will be a good alternative to Rox.

Tried replacing Copyfast's yaf-splash dialog with gxmessage as an alternative but my gxmessage version does not allow "ontop" functionality so no gain there.

[vtpup]

Very interesting discussion. I've been caught with file corruption on USB drives for the same reason.

Forum console workarounds for data transfer notifications aside, no responsible OS GUI should indicate the un-mounting operation to an external USB device is completed when it isn't. That seems basic.

[Flash]

Have you tried Copy-Fast?

Hi Flash,
yes i did try copyfast and it definitely reports accurate timing of "copy finished" - please see my post a couple of posts above your one.

[vtpup]

Again, the "normal" Puppy unmount incorrectly indicates data transfer is complete when it is not. Are new users expected to somehow know that they should run an alternative undocumented unmounting method; either a program, or through console commands, and that they should avoid the Puppy's intended desktop methods (rt click on drive icon and select unmount, or left click on the mounted dot)?

Is it okay to frustrate them with file corruptions? Do we want people to think this is a usable OS?

This is a basic GUI bug, plain and simple.

[Flash]

Vtpup, I haven't tried to unmount a drive many times while something is being written to it, but, if I remember right, when I've clicked the "unmount" x in the mounted drive icon, it doesn't go away until all the pending operations on that drive are complete.