Problem copying a folder/subfolders to another drive

[Governor]

Code: Select all

Distro Name                BookwormPup64 10.0.6 (x86_64)
 Woof-CE Build              testing;22250aa6d;2024-03-04 13:57:41 +0000
 Woof Version               9
 Binary Base                debian
 Distro Base                bookworm
 Distro Date                Mar 2024
 User Session               x11
 Window Manager             JWM v2.4.3
 Desktop Start              xwin jwm

I have a Download folder with subfolders on a nvme ext4 partition.

I want to copy all of it to an external usb exfat drive.
I understand there will be a problem copying to a non ext drive if there are symlinks, but I get this:

Error copying a folder-subfolders to another drive.jpg (50.01 KiB) Viewed 1269 times

It looks to me like the problem is a filename is already copied, and the OS is attempting to copy a folder with the same name, and there is a clash.

How do I solve this?
Thanks!

[MochiMoppel]

It looks to me like the problem is a filename is already copied

Correct. A script file - probably named 'mullvadbrowser' - was already copied.

and the OS is attempting to copy a folder with the same name, and there is a clash.

Yes, and this folder is named 'MullvadBrowser'. The clash happens because while file names in ext4 are case sensitive, in FAT they are not. This means that in FAT you can not have 2 files with the same name in the same folder, Just a different capitalization doesn't make them different.
If you let ROX overwrite, the existing script file would be replaced by the folder.

How do I solve this?

Change one of the conflicting names

BTW: This is not an "error" as your thread title and screenshot comment suggest.

[Governor]

It looks to me like the problem is a filename is already copied

Correct. A script file - probably named 'mullvadbrowser' - was already copied.

and the OS is attempting to copy a folder with the same name, and there is a clash.

Yes, and this folder is named 'MullvadBrowser'. The clash happens because while file names in ext4 are case sensitive, in FAT they are not. This means that in FAT you can not have 2 files with the same name in the same folder, Just a different capitalization doesn't make them different.
If you let ROX overwrite, the existing script file would be replaced by the folder.

How do I solve this?

Change one of the conflicting names

BTW: This is not an "error" as your thread title and screenshot comment suggest.

Thanks!
I changed 'error' to 'problem'. It is a problem. I need to be able to access documents, images and files from a Windows computer, which is why I used exFAT

[Governor]

It looks to me like the problem is a filename is already copied

Correct. A script file - probably named 'mullvadbrowser' - was already copied.

and the OS is attempting to copy a folder with the same name, and there is a clash.

Yes, and this folder is named 'MullvadBrowser'. The clash happens because while file names in ext4 are case sensitive, in FAT they are not. This means that in FAT you can not have 2 files with the same name in the same folder, Just a different capitalization doesn't make them different.
If you let ROX overwrite, the existing script file would be replaced by the folder.

How do I solve this?

Change one of the conflicting names

BTW: This is not an "error" as your thread title and screenshot comment suggest.

Ok, thanks!
I changed 'error' to 'problem'. It is a problem. I need to be able to access documents, images and files from a Windows computer, which is why I used exfat on the external drive.

[esos]

I think, you have to use archive manager.
Direct copy will fail.

[bigpup]

Usually in Linux files are named with all lower case letters.

I would try and name the file to using all lower case letters.

Hopefully whatever uses this file will still work with the name being in all lower case letters.

[Governor]

I think, you have to use archive manager.
Direct copy will fail.

Sounds promising. The folder is 140 GB, and I can't imagine going through it and changing files individually if there are many instances.
Where do I find the archive manager, and can I use that for regular backups?
Is that Xarchiver?
Thanks!

[fredx181]

@Governor Just a simple suggestion (for next year ):
IF you don't know exactly what you're doing, do not copy files/folders that are on NTFS filesystem to a Linux filesystem (ext*) (and not the other way around too) , also not FAT > Linux, Linux > FAT etc..
(but if it are all just e.g. text files, video/audio files, that's often ok of course)

Wish you a great 2025 !

[MochiMoppel]

I can't imagine going through it and changing files individually if there are many instances.

If there are only few or no instances at all, it can't be that hard.

I made a little one-liner that will list all instances of duplicate names that differ only in their capitalization.
The example tests the directory /usr. Adapt to your needs:

Code: Select all

find /usr/ | sort | uniq -i --all-repeated=separate

Output in case of BW64:

Code: Select all

/usr/bin/flsynclient
/usr/bin/flSynclient

/usr/include/X11/bitmaps/stipple
/usr/include/X11/bitmaps/Stipple

/usr/lib/firmware/radeon/bonaire_ce.bin
/usr/lib/firmware/radeon/BONAIRE_ce.bin

/usr/lib/firmware/radeon/bonaire_mc.bin
/usr/lib/firmware/radeon/BONAIRE_mc.bin
...

[Governor]

I can't imagine going through it and changing files individually if there are many instances.

If there are only few or no instances at all, it can't be that hard.

I made a little one-liner that will list all instances of duplicate names that differ only in their capitalization.
The example tests the directory /usr. Adapt to your needs:

Code: Select all

find /usr/ | sort | uniq -i --all-repeated=separate

Output in case of BW64:

Code: Select all

/usr/bin/flsynclient
/usr/bin/flSynclient

/usr/include/X11/bitmaps/stipple
/usr/include/X11/bitmaps/Stipple

/usr/lib/firmware/radeon/bonaire_ce.bin
/usr/lib/firmware/radeon/BONAIRE_ce.bin

/usr/lib/firmware/radeon/bonaire_mc.bin
/usr/lib/firmware/radeon/BONAIRE_mc.bin
...

This is great. I used to write lots of these in my msdos days. Back then, I was using about two hours a day on coding. I am not in that headspace anymore, and linux is far more complex. I adjusted your command to my needs. How would you put this into a 'batch' file? I would call it: "dupfilenames.bat". I don't know what the extension should be. The target directory should be required and passed as a parameter from the command line, and a message echoed to the screen (prompting the user) if no parameter is given. If I can see how it is done with this command, I am sure I can make others myself.

Code: Select all

# find [b]/mnt/nvme0n1p5/Downloads/[/b] | sort | uniq -i --all-repeated=separate
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.6/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.6/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.9/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.9/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-14.0/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-14.0/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/XZ/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/XZ/mullvad-browser/Browser/MullvadBrowser

BTW, I did not create these duplicates, it was the mullvad people. I wonder why they do that.
Thanks!

[Governor]

@Governor Just a simple suggestion (for next year ):
IF you don't know exactly what you're doing, do not copy files/folders that are on NTFS filesystem to a Linux filesystem (ext*) (and not the other way around too) , also not FAT > Linux, Linux > FAT etc..
(but if it are all just e.g. text files, video/audio files, that's often ok of course)

Wish you a great 2025 !

I tried to paste ASCII art here within code brackets, but for some reason it gets trashed anyway. Back in the day, people used to post ASCII art and it did not get trashed. I guess that's progress, right.
A belated Merry Christmas and Happy New Year to all.

Merry Christmas.jpg (88.24 KiB) Viewed 1095 times

[fredx181]

@Governor

BTW, I did not create these duplicates, it was the mullvad people. I wonder why they do that.

I wouldn't call it duplicates, probably from Linux view these are different files (or one is a file, the other a folder).

How would you put this into a 'batch' file? I would call it: "dupfilenames.bat". I don't know what the extension should be. The target directory should be required and passed as a parameter from the command line, and a message echoed to the screen (prompting the user) if no parameter is given. If I can see how it is done with this command, I am sure I can make others myself.

This should do, put it in a text file, give it some name, no extension needed, and... make the file executable. (e.g. chmod +x /path/to/myscript)

Code: Select all

#!/bin/bash

DIR="$1"
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi

find "$DIR" | sort | uniq -i --all-repeated=separate

[esos]

The folder is 140 GB, and I can't imagine going through it and changing files individually if there are many instances.

Now you are talking about 140 gb folder.
There are many tools you can use:"imaging tool" in window system and linux as well.
Basically just create an image file and then extract it some where else(make sure 100% your system support exfat).

[Governor]

@Governor

BTW, I did not create these duplicates, it was the mullvad people. I wonder why they do that.

I wouldn't call it duplicates, probably from Linux view these are different files (or one is a file, the other a folder).

How would you put this into a 'batch' file? I would call it: "dupfilenames.bat". I don't know what the extension should be. The target directory should be required and passed as a parameter from the command line, and a message echoed to the screen (prompting the user) if no parameter is given. If I can see how it is done with this command, I am sure I can make others myself.

This should do, put it in a text file, give it some name, no extension needed, and... make the file executable. (e.g. chmod +x /path/to/myscript)

Code: Select all

#!/bin/bash

DIR="$1"
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi

find "$DIR" | sort | uniq -i --all-repeated=separate

I made a file, but it did not work. I gave it permissions and the OS says it is a shell script, but it still won't run. Not from rox and not from the command line. From rox, I get nothing. From the command line, I get: command not found.
Any suggestions? Also can a give it a .bat extension and have it run? This will enable me to differentiate between homemade scripts and non-homemade scripts.
Thanks!

[d-pupp]

@Governor
To run a bash script from the command line it either has to be in the path or cd into the directory where the script is. Type ls (list files) to confirm you are in the correct directory. Then type ./script_name. The ./ tells bash to look in this directory for the command.

[fredx181]

Yes, or type the full path in terminal, if it's inside /root then e.g.: /root/myscript /path/to/mydir

[MochiMoppel]

Also can a give it a .bat extension and have it run? This will enable me to differentiate between homemade scripts and non-homemade scripts.

You can, but you shouldn't.
To distinguish between homemade scripts and non-homemade scripts Puppy provides the directory /root/my-applications/bin. When you put your script into this directory you can run it from the command line just by its name, without entering the full path. No other application will ever write into this directory, so it's exclusively there to hold your stuff.

[Governor]

@d-pupp @esos

Also can a give it a .bat extension and have it run? This will enable me to differentiate between homemade scripts and non-homemade scripts.

You can, but you shouldn't.
To distinguish between homemade scripts and non-homemade scripts Puppy provides the directory /root/my-applications/bin. When you put your script into this directory you can run it from the command line just by its name, without entering the full path. No other application will ever write into this directory, so it's exclusively there to hold your stuff.

Thanks for the answers. I have put the script into /root/my-applications/bin. I am using the script provided by @fredx181.
It works in the terminal from any location! But not from rox, where set run action is grayed out.
Can I get it to run from rox and xfe?

Does Linux use errorlevels?
How do I echo to the screen: "No duplicates found", to show that the command worked correctly when no duplicates are found?
Thanks!

#!/bin/bash

Code: Select all

DIR="$1"
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi

find "$DIR" | sort | uniq -i --all-repeated=separate

[d-pupp]

@Governor

I am a bash script beginner and someone may have a better idea. However my understanding is that the script will run when clicked on in ROX, you just won't see the output because it's running in the background.
To get output when the script completes successfully I would add && echo "Script completed".
You can make the text whatever you want but you need the quotes "" if you have any spaces. This text will show if the script ran successfully even if there is no other output.

Code: Select all

find "$DIR" | sort | uniq -i --all-repeated=separate && echo "Script completed"

[fredx181]

@Governor

It works in the terminal from any location! But not from rox, where set run action is grayed out.

Not run action, you need to right-click on *some* directory (as it's for scanning a directory), e.g. /usr (as I did , see pic) then "Customise Menu" , further see .gif picture below:
(I named the script "fndups", probably different from yours)

Screenshot(2).gif (286.83 KiB) Viewed 874 times

New script: (runs now in urxvt terminal (otherwise no output will show when running from rox) and gives info when no dups found)

Code: Select all

#!/bin/bash

export DIR="$1"

fnddups () {
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi

find "$DIR" | sort | uniq -i --all-repeated=separate | tee /tmp/fnd_dups
[ -z $(cat /tmp/fnd_dups) ] && echo "No duplicates found"  # if /tmp/fnd_dups is empty
rm -f /tmp/fnd_dups       # comment this out if you want to see the output in that text file
bash
}; export -f fnddups

urxvt -T "Scan for Duplicate names" -si -sb -fg white -bg SkyBlue4 -geometry 80x20 -e bash -c fnddups

EDIT: you can also comment out rm -f /tmp/fnd_dups so becomes # rm -f /tmp/fnd_dups (disabled then) for to see the output in that text file.

[geo_c]

Can I get it to run from rox and xfe?

How do I echo to the screen: "No duplicates found", to show that the command worked correctly when no duplicates are found?"

If your script has executable permissions, then xfe will run it as long as the 'Don't execute scripts' option is not enabled in the General menu of Preferences. I uncheck that.

But xfe has one more option that I do enable, which is in the Dialogs menu of Preferences. It's the "Confirm execute text files" option. I check that one, because with that option enabled, double clicking an exectuable script will give you a choice of executing it in the background, or running it with a console output of the script as it's running that stays open until you close the dialog. That's how I run my scripts from a file manager. Xfe is great for running scripts. With "Confirm execute text files" checked you get the option of executing in the background, executing with a terminal output, editing the text file, or cancelling. So it's a good way to execute scripts from the file manager safely.

[Governor]

@geo_c @fredx181 @MochiMoppel @esos

@Governor

I am a bash script beginner and someone may have a better idea. However my understanding is that the script will run when clicked on in ROX, you just won't see the output because it's running in the background.
To get output when the script completes successfully I would add && echo "Script completed".
You can make the text whatever you want but you need the quotes "" if you have any spaces. This text will show if the script ran successfully even if there is no other output.

Code: Select all

find "$DIR" | sort | uniq -i --all-repeated=separate && echo "Script completed"

This is great. Very useful. Thanks!
I have forgotten most of what I knew, but when I was writing scripts many years ago, I loved documenting events for the user. I was meticulous with checking everything so the chances of error were nonexistent to minimal. For example:
Check if the temp directory exists, and is writable, check if there is enough space in the temp directory, check if external commands used in the script are present and working, etc.
A user could be running a script on a locked drive; the temp variable could be set wrong, an external program that ships with the OS could be corrupt or missing, drive could be full, etc.
I included a built-in help in most of the scripts I wrote - I thought that was cool, and it left no doubt about how to use the script, since it is very limited how descriptive a filename can be with only eight characters. I ended up changing to the %temp% directory within the scripts, running all the commands there and then changing back to the original directory before ending. I realize most people would not go through so much trouble, but I liked doing it. Most people back then just assumed everything was in order and the current directory was writable and had enough space. Sorry about the rambling.

One of the worst things for me is to run a program or script and not be able to tell if it is working or if it is stuck. Or it fails without giving a very specific reason.

Code: Select all

#!/bin/bash

DIR="$1"
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi
echo Please wait... duplicates found will show on screen.
find "$DIR" | sort | uniq -i --all-repeated=separate && echo "Script completed"
echo

,

[Trapster]

Remember to quote your echo's

You could also send the results to a file. then have the file open.

Code: Select all

find "$DIR" | sort | uniq -i --all-repeated=separate > /root/find_results.txt
geany /root/find_results.txt

Code: Select all

#!/bin/bash

DIR="$1"
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi
echo Please wait... duplicates found will show on screen.
find "$DIR" | sort | uniq -i --all-repeated=separate && echo "Script completed"
echo

[d-pupp]

@Governor I agree it is fun leaning and writing simple scripts. I started with setup scripts. After a new frugal install I would just run a script or two and most of my personalization was done.

BTW there is a second method for testing if a command worked or not. The exit code. Most command have an exit code of 0 if they worked ok. Check the man page to verify. It is stored in the variable $? and is overwrote by the next command that runs. You can check it with echo $?

[fredx181]

@Governor I agree it is fun leaning and writing simple scripts. I started with setup scripts. After a new frugal install I would just run a script or two and most of my personalization was done.

BTW there is a second method for testing if a command worked or not. The exit code. Most command have an exit code of 0 if they worked ok. Check the man page to verify. It is stored in the variable $? and is overwrote by the next command that runs. You can check it with echo $?

But in case of find (edit: this case with the pipes) it seems to return always value 0
Check this and it still echoes "Script completed" (but also gives "find: '/BLAHBLA': No such file or directory", as it cannot find the directory)

Code: Select all

DIR=/BLAHBLA
find "$DIR" | sort | uniq -i --all-repeated=separate && echo "Script completed"

Code: Select all

find: '/BLAHBLA': No such file or directory
Script completed

EDIT: but without the pipes to sort uniq etc... it gives error code 1

Code: Select all

DIR=/BLAHBLA
find "$DIR" && echo "Script completed"
find: ‘/BLAHBLA’: No such file or directory
echo $?
1

EDIT2: So need to use PIPESTATUS :

Code: Select all

DIR=/BLAHBLA
find "$DIR" | sort | uniq -i --all-repeated=separate
[ $PIPESTATUS -eq 0 ] && echo "Script completed" || echo "There was an error" 

output:

Code: Select all

find: '/BLAHBLA': No such file or directory
There was an error

But that may only give exit status 1 if the path provided to find doesn't exist (as in this case), not sure though.

[d-pupp]

Hello @fredx181
I agree it's simple and needs more error checking. I'm just a beginner so I just offered what I would do quickly to get some idea if the script worked or not. Then I would work on making it better just to see if I could.

[Governor]

Remember to quote your echo's

You could also send the results to a file. then have the file open.

Code: Select all

find "$DIR" | sort | uniq -i --all-repeated=separate > /root/find_results.txt
geany /root/find_results.txt

This was elegant!
I find the single line commands much easier to begin to understand, as Bash is unfamiliar to me with the all of the symbols, and Linux commands in general are unfamiliar and super complex. I also have so far not been able to really grasp regular expressions, and I therefore find .htaccess files daunting. Cobol and dos batch are more like plain English.

Anyway, using the script further below, I get this:

Code: Select all

# dupfilenames /mnt/nvme0n1p5/Downloads/
Please wait... duplicates found will show on screen.
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.6/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.6/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.9/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-13.5.9/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-14.0/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/Browsers/mullvad-browser-linux-x86_64-14.0/mullvad-browser/Browser/MullvadBrowser

/mnt/nvme0n1p5/Downloads/XZ/mullvad-browser/Browser/mullvadbrowser
/mnt/nvme0n1p5/Downloads/XZ/mullvad-browser/Browser/MullvadBrowser

.mp4
.mp4
Script completed

I could not see where the two .mp4 are coming from at the end. How do I locate the cause of that?

Code: Select all

#!/bin/bash

DIR="$1"
if [ ! -d "$DIR" ]; then
echo "Please provide a directory as first argument"
exit
fi
echo Please wait... duplicates found will show on screen.
find "$DIR" | sort | uniq -i --all-repeated=separate && echo "Script completed"
echo ""

[d-pupp]

Files that start with a . are hidden in Linux by default.
If you are using ROX filer and I remember correctly a left click on the eye should show hidden files.

[MochiMoppel]

Remember to quote your echo's

You could also send the results to a file. then have the file open.

Code: Select all

find "$DIR" | sort | uniq -i --all-repeated=separate > /root/find_results.txt
geany /root/find_results.txt

This was elegant!

Maybe even more elegant, because you wouldn't need to create (and later delete) a temporary file:

Code: Select all

find "$DIR" | sort | uniq -i --all-repeated=separate | leafpad

The unelegant part of both solutions is that you will stare at a blank Geany/Leafpad document when no duplicates were found. It needs more code to prevent this.

I could not see where the two .mp4 are coming from at the end. How do I locate the cause of that?

I suspect that you have 2 files with linefeeds in their names:
mullvadbrowser
.mp4
and
MullvadBrowser
.mp4
This would cause the output you posted.
You could use the find command to hunt them down:

Code: Select all

find /mnt/nvme0n1p5/Downloads/ -iname 'mullvadbrowser*mp4'

If there are indeed linefeeds in the names they may be represented by a questionmark in output.
Please note that even without duplicates of such funny names you would not be able to copy such files to a Windows filesystem!

[some1]

.mp4
.mp4
Script completed

I could not see where the two .mp4 are coming from at the end. How do I locate the cause of that?

You probably have a newline in a downloaded filename.
Try this:

Code: Select all

DIR=/mnt/nvme0n1p5/Downloads                      #i.e. your upper-path
find "$DIR" -name *.mp4|grep -n -B2 -E '^.mp4'
exit

Alternatively list all mp4s for eyeballing:

Code: Select all

DIR=/mnt/nvme0n1p5/Downloads                      #i.e. your upper-path
find "$DIR" -name *.mp4
exit

the culprits will probably show up with a questionmark .i.e ? where the newline (cruft ) is
If there is an embedded newline in the path/filename -the line breaks going through the pipe -
and sort will see several distinct lines consisting of the parts.
----------
MochiMoppels/Freds code will probably work when we speak about {a-z] and [A-Z] but...