Is there a limit to removing files with xargs?

[fredx181]

Hi, I'd like to know if there's a limit for xargs to execute removing files from list, e.g. this command:

Code: Select all

xargs -d '\n' -a /tmp/filelist rm

The file /tmp/filelist contains 6000 lines (paths to files) and above works OK, it's removing all listed files.
But what if it contains, let's say 6000000 lines of files to be removed, will it still work ?

Thanks,

Fred

[williams2]

I created a file named "x" with 8388608 lines, each line being the char "x"

Code: Select all

# wc x
 8388608  8388608 16777216 x
#

Code: Select all

# cat /tmp/x | xargs -d "\n" | wc
    129 8388608 16777216
#

So xargs echoed "x" 8388608 times. using the default echo command 129 times.
The -l switch would force the echo (or rm) command to be used 8388608 times vs. 129 times.

Code: Select all

# cat /tmp/x | head -n 5 | xargs -t -d "\n" | wc
echo x x x x x 
      1       5      10
# cat /tmp/x | head -n 5 | xargs -t -l -d "\n" | wc
echo x 
echo x 
echo x 
echo x 
echo x 
      5       5      10
#

The rm command should work the same way.

To make the test file:

Code: Select all

echo x > x
cat x >> y ; cat x >> y ; mv y x ; wc x
cat x >> y ; cat x >> y ; mv y x ; wc x
cat x >> y ; cat x >> y ; mv y x ; wc x
etc etc

[MochiMoppel]

The file /tmp/filelist contains 6000 lines (paths to files) and above works OK, it's removing all listed files.
But what if it contains, let's say 6000000 lines of files to be removed, will it still work ?

As williams2's test shows it will still work.
xargs reads filepaths from filelist until it reaches a limit of roughly 131k, upon which it passes these filepaths to rm. Then xargs reads the next 131k chunck and passes it to rm and so on until all lines are processed. What puzzles me: How does xargs determine the maximum commandline length?

[puppy_apprentice]

What puzzles me: How does xargs determine the maximum commandline length?

Code: Select all

xargs --show-limits

Shows system limits.

[MochiMoppel]

Thanks

It says:
"Size of command buffer we are actually using: 131072"
Question remains: How does xargs come up with this figure?
[Edit] Seems to be a fixed kernel thing and not changeable. If so, let's forget it. Sorry for the interruption.

[puppy_apprentice]

Some answers i hope:

stackexchange.com
https://unix.stackexchange.com/question ... its-output
https://unix.stackexchange.com/question ... tem-define

Bash Xargs Command the Hard Way by Example
https://linuxhint.com/bash_xargs_command_by_example/

[fredx181]

Thanks @williams2 , @MochiMoppel and all for the info.
The xargs way is great, incredibly fast compared to using a "while read ..." loop to read from a file.

Fred

[MochiMoppel]

The xargs way is great, incredibly fast compared to using a "while read ..." loop to read from a file.

xargs is useful even when not reading from file.
Let's say you have an insane number of mp3 files in /root/mysongs and you want to delete them all.
The command rm /root/mysongs/*.mp3 will certainly fail with an error because bash will try to use rm only once with all filepaths as arguments (rm /root/mysongs/file1.mp3 /root/mysongs/file2.mp3 /root/mysongs/file3.mp3 etc ). That's too long.
The danger of hitting the 131k wall can be reduced by a cd /root/mysongs , followed by rm *.mp3 , but the risk to fail still remains.
Using xargs would eliminate this risk: echo *.mp3 | xargs rm

[fredx181]

xargs is useful even when not reading from file.
Let's say you have an insane number of mp3 files in /root/mysongs and you want to delete them all.
The command rm /root/mysongs/*.mp3 will certainly fail with an error because bash will try to use rm only once with all filepaths as arguments (rm /root/mysongs/file1.mp3 /root/mysongs/file2.mp3 /root/mysongs/file3.mp3 etc ). That's too long.
The danger of hitting the 131k wall can be reduced by a cd /root/mysongs , followed by rm *.mp3 , but the risk to fail still remains.
Using xargs would eliminate this risk: echo *.mp3 | xargs rm

Tested echo *.mp3 | xargs rm on a folder full of mp3's, and failed because of filenames containing spaces.
This, for example, should work in that case:
cd /root/mysongs; ls *.mp3 | xargs -d '\n' rm
Edit: Or, to delete all (not just mp3's)
cd /root/mysongs; ls | xargs -d '\n' rm

[some1]

printf is probably faster than ls

Using touch instead of rm,try something like this:

Code: Select all

cd /root/mysongs;
time printf "%s\0" *.mp3 | xargs -0 touch -f
time ls *.mp3 | xargs -d '\n' touch -f

[MochiMoppel]

printf's format specifier %q is designed to fix the problem of spaces in file names.

This should work too:
printf '%q ' *.mp3 | xargs rm <= mind the space after %q
or
printf '%q\n' *.mp3 | xargs rm