Puppy Linux Discussion Forum

Posted: **Sun Aug 23, 2020 10:47 am**

One of Puppy's original benefits was the ability to protect USB drives by accumulating write data in memory temporarily rather than writing repeatedly direct to the usb drive - which would potentially shorten the life of the drive.

However - the downside of this has been that the write process has to play "catchup" at inconvenient times - such as at shutdown or when trying to unmount drives etc, which means that the user had to wait for the data to be transferred out of memory - even if it took minutes to transfer a couple of gigs across a slow bus onto a slow usb stick.

I experienced data corruption at times which i attributed to my having pulled usb sticks out of the PC before such data transfers had finished. The problem was that the system would say that the transfer had finished when in fact it was still underway.

If a usb stick had an LED which flashed to indicate the data transfer i could tell that i should wait - but with no LED it was pot luck.

These days I still experience the problem where the data is still transferring long after it appears that the copy has ended - but now i know to use the "sync" command in a terminal to force the data to transfer out of memory - and i wait for the terminal prompt to re-appear before pulling the stick.

What i am wondering is if there is some way to force any given write to begin immediately and tell me truthfully when it has finished?

Let me offer an example: lets say i have a 3GB video file in /root and want to copy it to a usb stick - i can start the copy (rox window to rox window) and hope that the transfer will start immediately but really i have no idea when the system will start to flush the data from memory (which is where /root lives...).

This is especially problematic if i am generating other NEW video files at the same time and filling up /root and approaching the limit of my personal save space (which is a total of system RAM + swap partition).

So my question is: is there any way to overcome the inherent write delay and tell the system that i want a specific copy to start NOW and complete as soon as possible (and that i dont care about preserving usb stick life)?

I dont want to have the uncertainty of thinking that my RAM is running out simply because the system is not planning on emptying RAM writes until it decides that some random timer will run out in x minutes and force the usb writes then.

Hope that makes sense.

I am guessing that there is some PUPMODE that will correct this anomaly - although i really want this to be under my manual control rather than treating my usb sticks as HDDs and writing at all times. I seem to be currently running PUPMODE 5 (system boots from usb and has no savefile).

Posted: **Sun Aug 23, 2020 11:44 am**

I think the smartest solution would be to modify Rox filer (and/or any other file manager) to do the sync immediately after the copy is done AND before the copy dialog/form disappears. So, the guys who already had worked on refining Rox filer would be the people to ask for this.

Posted: **Sun Aug 23, 2020 2:32 pm**

greengeek wrote: ↑Sun Aug 23, 2020 10:47 amSo my question is: is there any way to overcome the inherent write delay and tell the system that i want a specific copy to start NOW and complete as soon as possible (and that i dont care about preserving usb stick life)?

Use the "sync" mount option.

Code: Select all

mount -o sync /dev/sdb2 /mnt/sdb2

Assuming your usb stick is /dev/sdb2.

Posted: **Sun Aug 23, 2020 3:23 pm**

I seem to be currently running PUPMODE 5 (system boots from usb and has no savefile

I am a little confused.
Is this the way you always run?

Pupmode 5 only uses a ramdisk for save.

After you actually make a save file/folder and boot using the save.
Normally, a USB install will be running in pupmode 13.

How much RAM is in the computer?

I wonder if your issues, are trying to do too much, with too little RAM.

I hate to say it, but Puppy is not that good at warning about running out of RAM.

Also, if you have a swap file or partition.
Running in pupmode 5 is going to cause use of the swap, if the ramdisk is running out of RAM.
There is also the setting that controls when swap is used. At what percent level of RAM usage.

Also, you could have a large usage of RAM cache going on.
The cache would have to clear, to give more free RAM, for the ramdisk to use.

Posted: **Sun Aug 23, 2020 4:56 pm**

bigpup wrote: ↑Sun Aug 23, 2020 3:23 pm
I seem to be currently running PUPMODE 5 (system boots from usb and has no savefile
<snip>....How much RAM is in the computer?

I wonder if your issues, are trying to do too much, with too little RAM.

I hate to say it, but Puppy is not that good at warning about running out of RAM....</snip>

Tell me about it..! And this is one of the primary reasons I've upped the original 4 GB in this new HP desktop all the way to 16 GB. I had enough years of having to watch the amount of RAM in use with the old Compaq's 3 GB..... Now, I can do whatever I want, secure in the knowledge that I'm never getting anywhere near the system's limits.

I may yet go to 32 GB. Haven't decided yet...

(*shrug*)

Mike.

Posted: **Mon Aug 24, 2020 12:17 am**

As far as I know, there are no OS's that can guarantee data integrity if you remove removable media without dismounting first or using the "Safely remove device" option.

As a matter of course, I always issue the 'synch' command and allow it to complete before removing storage devices.

Posted: **Mon Aug 24, 2020 2:02 am**

Jafadmin wrote: ↑Mon Aug 24, 2020 12:17 am As far as I know, there are no OS's that can guarantee data integrity if you remove removable media without dismounting first or using the "Safely remove device" option.

DOS ヽ(♡‿♡)ノ

Posted: **Mon Aug 24, 2020 2:29 am**

garnet wrote: ↑Sun Aug 23, 2020 2:32 pm Use the "sync" mount option.
Code: Select all
mount -o sync /dev/sdb2 /mnt/sdb2
Assuming your usb stick is /dev/sdb2.

Thanks - just tried this idea and got unexpected results.

If i mount async it takes about 10 seconds to copy a 100MB file to the stick (and the copy dialog disappears halfway through the actual copy)

If i mount using sync it takes about 3 or 4 minutes to do the copy - but the copy dialog stays open until the writes have actually finished and the LED stops flashing.

Not sure what to make of this - more testing required.

Posted: **Mon Aug 24, 2020 2:35 am**

taersh wrote: ↑Sun Aug 23, 2020 11:44 am I think the smartest solution would be to modify Rox filer (and/or any other file manager) to do the sync immediately after the copy is done AND before the copy dialog/form disappears.

Yes that would be a significant benefit.

Posted: **Mon Aug 24, 2020 2:52 am**

bigpup wrote: ↑Sun Aug 23, 2020 3:23 pm I am a little confused.
Is this the way you always run?

Pupmode 5 only uses a ramdisk for save.

After you actually make a save file/folder and boot using the save.
Normally, a USB install will be running in pupmode 13.

Yes this is how my system has been running for the last 5 years. (It is probably a distraction if i explain how my pup is set up but basically i used jrb's technique of swapping the zdrv with the main pup.sfs so that my personalisations are in the zdrv and as a result i don't need a savefile)

How much RAM is in the computer?

I have 2GB of RAM - but a 6GB swap partition. What i find is that the system uses the swap partition as if it was RAM - so that i can copy a 3GB video file into /root and manipulate it quite happily with ffmpeg or similar (which can potentially double the RAM requirements and runs close to stalling the pup once the swap is all used up...). Seems as if i could happily add 20GB of swap rather than trying to double the internal RAM - so i never bothered to increase the internal.

I wonder if your issues, are trying to do too much, with too little RAM.

Yes the problem with running out of free RAM is caused by having too little memory - i know that i can get past that problem by adding more sodimms, or by adding more swap partition space, or by limiting the size of video files that i am manipulating. But the problem with puppy not writing data OUT of /root as soon as i start a copy is a different problem. Sometimes i just want to ditch some data out of root and i want it gone now rather than waiting for it to decide it is ready to run a sync and flush the requested "copy" out of memory.
(I can manually force it with the "sync" command but i would like to make it happen automatically as soon as i command Rox to do the copy)

I hate to say it, but Puppy is not that good at warning about running out of RAM.

I agree. I read in various places that "Linux is good at managing swap functions" - yet i don't see that on my system. Never any warnings, just dead in the water once the bottom rung of the "partview" icon starts flashing red.
But in all fairness - i should have noticed the partview icon stepping down to orange - and got my other "root file dump" copies out of the way sooner.

Also, if you have a swap file or partition.
Running in pupmode 5 is going to cause use of the swap, if the ramdisk is running out of RAM.
There is also the setting that controls when swap is used. At what percent level of RAM usage.

Swap usage has always been an oddball thing to me. It almost never looks as if swap is in use - but then if i copy a massive file to /root (or other non-HDD directory) it seems happy to use the swap to hold that file as if it was just extended RAM. Seems all or nothing. Like swap is "shut for business" until the exact moment when puppy is about to die - then the swap door opens and all is good. (Till swap hits 99% full and puppy goes fully tits up).

Also, you could have a large usage of RAM cache going on.
The cache would have to clear, to give more free RAM, for the ramdisk to use.

Got me there - cache is a mystery to me.

Posted: **Mon Aug 24, 2020 2:56 am**

greengeek wrote: ↑Mon Aug 24, 2020 2:29 am If i mount using sync it takes about 3 or 4 minutes to do the copy - but the copy dialog stays open until the writes have actually finished and the LED stops flashing.

Isn't that what you want?

Posted: **Mon Aug 24, 2020 3:00 am**

garnet wrote: ↑Mon Aug 24, 2020 2:56 am
greengeek wrote: ↑Mon Aug 24, 2020 2:29 am If i mount using sync it takes about 3 or 4 minutes to do the copy - but the copy dialog stays open until the writes have actually finished and the LED stops flashing.
Isn't that what you want?

Well i do want the copy dialog to stay open until the copy is finished - but i don't want the copy to take ten times as long in syncmount as it did in asyncmount.
(The actual data transfer took ten times as long using sync mount)

Posted: **Mon Aug 24, 2020 3:09 am**

greengeek wrote: ↑Mon Aug 24, 2020 3:00 am Well i do want the copy dialog to stay open until the copy is finished - but i don't want the copy to take ten times as long in syncmount as it did in asyncmount.
(The actual data transfer took ten times as long using sync mount)

What is the filesystem on the usb stick? I hope it is not ext3 or ext4?
How much is the total time to: copy in asyncmount + run "sync" in terminal and waits until the "sync" is completed? I think this total should be the same as copy in syncmount, no?

Posted: **Mon Aug 24, 2020 3:09 am**

mikewalsh wrote: ↑Sun Aug 23, 2020 4:56 pm I've upped the original 4 GB in this new HP desktop all the way to 16 GB. I had enough years of having to watch the amount of RAM in use with the old Compaq's 3 GB..... Now, I can do whatever I want, secure in the knowledge that I'm never getting anywhere near the system's limits.

I may yet go to 32 GB. Haven't decided yet...

Try loading 16GB or more of video files into /root and see if you can pick the exact point when the pup clags up.
Then repeat the process after plugging in a fast 16GB usb stick formatted as swap and see if this allows you to treat the system as if it had double the RAM.

I would be keen to see the result.
(Obviously usb swap is slow - but at least it should stop the total clag-out of memory processes).

I think i will need to make it a habit to plug in extra swap on-the-fly when i notice my partview going orange.

Posted: **Mon Aug 24, 2020 3:26 am**

I have also 16 GB of RAM. Plus 11 GB of Swap partition. When the RAM usage is at around 15 GB and the swap is used at several GB Puppy gets very slow. Even simple tasks like moving the mouse pointer doesn't work properly any longer.

From my experience swap is a fake. Ok, it may save the computer needing to be shut off by pressing the reset button, but any useful use isn't possible then until the RAM usage gets much lower. When the RAM returns to default usage by a few GB around 4GB the swap doesn't return to zero. It keeps several hundred MB up to 1.5 GB.

Posted: **Mon Aug 24, 2020 3:38 am**

taersh wrote: ↑Mon Aug 24, 2020 3:26 am When the RAM returns to default usage by a few GB around 4GB the swap doesn't return to zero. It keeps several hundred MB up to 1.5 GB.

Code: Select all

echo 10 > /proc/sys/vm/swappiness

(⌒‿⌒)

Posted: **Mon Aug 24, 2020 3:39 am**

Jafadmin wrote: ↑Mon Aug 24, 2020 12:17 am As far as I know, there are no OS's that can guarantee data integrity if you remove removable media without dismounting first or using the "Safely remove device" option.

I get caught by the deception of the rox "copy" dialog disappearing before the data is actually written out. In part this is because the data is sometimes only being written to RAM (and the auto sync will occur later), but sometimes if the file is big enough the rox dialog closes half way through the actual data transfer.

I believe there are some Pup modes where data writing is not delayed - it starts as soon as the copy dialog is begun.

I hope i can access that functionality without experiencing the super slow data transfer i saw when i tried Garnets suggestion of the sync mount.

(A bit of Googling suggests sync mount is pretty much never used - so i doubt that is the mechanism Puppy uses when in a pupmode that writes immediately to disk)

Posted: **Mon Aug 24, 2020 3:42 am**

I have a script running at boot that already sets the swappiness to 10.

Code: Select all

sysctl vm.swappiness=10

Is that different to

Code: Select all

echo 10 > /proc/sys/vm/swappiness

???

Posted: **Mon Aug 24, 2020 4:02 am**

greengeek wrote: ↑Mon Aug 24, 2020 3:39 am I get caught by the deception of the rox "copy" dialog disappearing before the data is actually written out. In part this is because the data is sometimes only being written to RAM (and the auto sync will occur later)

Not "sometimes", but "always", unless one uses "sync" mount option.

I hope i can access that functionality without experiencing the super slow data transfer i saw when i tried Garnets suggestion of the sync mount.

You do get the guarantee that your data is on the stick, though (⌒‿⌒)

(A bit of Googling suggests sync mount is pretty much never used - so i doubt that is the mechanism Puppy uses when in a pupmode that writes immediately to disk)

PUPMODE=5 ?

taersh wrote: ↑Mon Aug 24, 2020 3:42 am I have a script running at boot that already sets the swappiness to 10.
Code: Select all
sysctl vm.swappiness=10
Is that different to
Code: Select all
echo 10 > /proc/sys/vm/swappiness
???

Same (⌒‿⌒)

Posted: **Mon Aug 24, 2020 5:03 am**

garnet wrote: ↑Mon Aug 24, 2020 4:02 am You do get the guarantee that your data is on the stick, though (⌒‿⌒)

True. And that is the important outcome. But is that the only impact i wonder? It seems to me that the sync mount must have some other impact that causes a slower data transfer rate to the usb stick. I want to investigate that further on other more standard pups. May have just been a temporarily anomaly.

Posted: **Mon Aug 24, 2020 8:29 am**

I thought i would try to make some accurate comparisons copying to my usb stick using both sync mount and async mount - using a different puppy and a different method to my previous test.

In order to avoid any contributing factors from Rox-Filer i wrote the following script:

Code: Select all

#! /bin/bash
gxmessage "   Click ok to start copy now.
   You will receive a confirmation when copy is finished.
   You will also be advised of start and stop times."
STARTDATE=$(date)
cp -f /root/test.iso /mnt/sdd1
sync
ENDDATE=$(date)
gxmessage "   Copy is finished.
   Start time was $STARTDATE and end time was $ENDDATE
   Subtract start time from end time to calculate elapsed time for copy"

I set it up to copy a 100MB file (i used tazpuppy-5.0-beta-30.iso renamed as test.iso) to sdd1, and i used it after mounting sdd1 with each of the following commands:

# mount -o sync /dev/sdd1 /mnt/sdd1
and
# mount -o async /dev/sdd1 /mnt/sdd1

The sync mount copy took 3 minutes and 28 seconds.
The async mount copy took only 11 seconds.

In each case the usb stick LED flashing stopped at the same time as the script ended - so i believe these figures measured accurately the time taken to transfer the data.

I am not sure what exactly is happening with sync mount. Keen to hear if other peoples pups and usb sticks show similar characteristics.

Posted: **Tue Aug 25, 2020 8:32 am**

Just want to post a link to a comment on the old forum by Flash - explaining that "copyfast" uses rsync and supposedly keeps up a window monitoring the copy until it is actually finished.

http://murga-linux.com/puppy/viewtopic. ... 57#1034582

This sounds like something i could benefit from so i gave Don570's rightclick version of copyfast a try and it is promising - although instead of a copy window it seems to pop up a yafsplash or similar for the duration of the copy - but that message disappears behind other windows and drops out of the users view.

Maybe i can modify it to stay on top.

Posted: **Thu Sep 03, 2020 8:54 am**

I found this article about disk caching. I hope it helps.
better-linux-disk-caching-performance-vm-dirty_ratio

Posted: **Thu Sep 03, 2020 6:27 pm**

Have you tried Copy-Fast? It uses rsync, I think. Anyway, it adds an item to Rox's right-click menu that immediately copies to the destination without any caching or such.

Posted: **Thu Sep 03, 2020 8:08 pm**

There is no magical way to copy files. I/O is I/O. All that can be modified is how the process appears to the user.

Do some benchmarking from the console.

Code: Select all

cp test.file /mnt/sdb1;sync

That will give you the sync blinking cursor until all the buffering is spooled out to disk. Try it without using sync.

You never see the problem copying on fixed disks. It just buffers in the background as you go on your merry way with what you wanted to do. The buffered data finishes writing out in the background.

The problem only becomes an issue with removable media.

Posted: **Fri Sep 04, 2020 7:38 am**

Flash wrote: ↑Thu Sep 03, 2020 6:27 pm Have you tried Copy-Fast? It uses rsync, I think. Anyway, it adds an item to Rox's right-click menu that immediately copies to the destination without any caching or such.

Hi Flash,
yes i did try copyfast and it definitely reports accurate timing of "copy finished" - please see my post a couple of posts above your one. I think Copyfast and rsync is the answer i needed - although Copyfast uses a yaf-splash advisory dialog that does not stay on top of other windows and therefore risks leaving me in the dark just as much as Rox Filers copy dialog.

If i can solve the issue of the dialog window disappearing from view Copyfast will be a good alternative to Rox.

Tried replacing Copyfast's yaf-splash dialog with gxmessage as an alternative but my gxmessage version does not allow "ontop" functionality so no gain there.

Posted: **Fri Sep 04, 2020 12:53 pm**

Very interesting discussion. I've been caught with file corruption on USB drives for the same reason.

Forum console workarounds for data transfer notifications aside, no responsible OS GUI should indicate the un-mounting operation to an external USB device is completed when it isn't. That seems basic.

Posted: **Fri Sep 04, 2020 6:08 pm**

greengeek wrote: ↑Fri Sep 04, 2020 7:38 am
Flash wrote: ↑Thu Sep 03, 2020 6:27 pm Have you tried Copy-Fast?
Hi Flash,
yes i did try copyfast and it definitely reports accurate timing of "copy finished" - please see my post a couple of posts above your one.

Posted: **Sat Sep 05, 2020 2:43 am**

Again, the "normal" Puppy unmount incorrectly indicates data transfer is complete when it is not. Are new users expected to somehow know that they should run an alternative undocumented unmounting method; either a program, or through console commands, and that they should avoid the Puppy's intended desktop methods (rt click on drive icon and select unmount, or left click on the mounted dot)?

Is it okay to frustrate them with file corruptions? Do we want people to think this is a usable OS?

This is a basic GUI bug, plain and simple.

Posted: **Sat Sep 05, 2020 2:57 am**

Vtpup, I haven't tried to unmount a drive many times while something is being written to it, but, if I remember right, when I've clicked the "unmount" x in the mounted drive icon, it doesn't go away until all the pending operations on that drive are complete.

Puppy Linux Discussion Forum

Overcoming USB copy delay

Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay

Re: Overcoming USB copy delay