Error: Process Substitution: Pass File Descriptor to Bash Function

[s243a]

The following works:

Code: Select all

cat <(echo a)

However, the following procuces an error:

Code: Select all

set -- <(echo a)
[root@Dpupbuster ~] $ cat $1
cat: /dev/fd/63: No such file or directory

Why? Are there any workarounds? The closes, that I think of is a named pipe, which in my opinion isn't really process subsitution and requires an extra line (or two) of code and something almost like a temporary file.

Some info on process substition:
https://tldp.org/LDP/abs/html/process-sub.html

BTW, I see now file called "/dev/fd/63". Does the collowign code, "cat <(echo a)" briefly create such a file descriptor or should I create it?

[s243a]

As a side note, the following symbolic link exists on my sytem

Code: Select all

ln -s /proc/self/fd /dev/fd

https://askubuntu.com/questions/1086617 ... -directory

I didn't create this link and it should be there. It looks like a directory and has file descriptors up to fd9. As I noted above, I presume that higher file descriptors are created only temporarily.

[step]

It breaks because /dev/fd/63 only lives on the line you entered <(). Bash creates and destroys /dev/fd/63 for you; you have no saying.

Work-around (works in other shells => it's more portable)

Code: Select all

cat << EOF
$(echo a)
EOF

It's also more flexible

Code: Select all

cat << EOF
$(echo a)
and this line too
more...
EOF

You can use it in a pipe and other command list, with redirections too...

Code: Select all

cat << EOF |
$(echo a)
and this line too
EOF
wc

output (that's wc printing it)

Code: Select all

      2       5      20

I prefer this form to bash's <(). Where bash has an edge is when you need to feed multiple process substitutions to the same command, such as:

Code: Select all

diff <(echo a) <(echo b)

output

Code: Select all

1c1
< a
---
> b

[s243a]

I have a workaround although not as flexible as the techniques that @step mentioned above. I created a lazy cat function that returns the path to a named pipe:

./lcat

Code: Select all

#!/bin/bash
tmp_pipe="$(realpath .)/$(mktemp -u tmppipeXXXX)"
mkfifo "$tmp_pipe"
nohup cat "$@" >$tmp_pipe &
echo "$tmp_pipe"
exec 1>/dev/null

Send commands to named pipe:

Code: Select all

bla="$(./lcat some_file|head -n 1)"

read from named pipe:

Code: Select all

cat ""$bla"

bla="$(./lcat sub|head -n 1)"

The "head -n 1" is necessary to tell the command substitution that we have reached the end of file because otherwise command substitution might wait for all subprocesses to finish before exiting. In some cases though you can supposedly get around this by redirecting stdout to null.

See:

Bash run command in background inside subshell

I will study @steps post above to see if I can learn some more general techniques.

[s243a]

It breaks because /dev/fd/63 only lives on the line you entered <(). Bash creates and destroys /dev/fd/63 for you; you have no saying.

I wonder why the cat command can get around this limitation but a shell script can't. One could probably write a "c" function to redirect /dev/fd/63 to a named pipe.

[step]

I'm not sure I understand your question. The cat command doesn't need to get around anything. As far as cat's is concerned, it's simply reading from a file as it always does (unless its inputs is redirected). When you write cat <(echo a) bash processes the line, it notices the <(), it creates a temporary file named /dev/fd/63, executes "echo a" with output redirected to /dev/fd/63, runs cat pretending that the original line was cat /dev/fd/63, and finally destroys /dev/fd/63. Any links to /dev/fd/63 will be left dangling. My description is oversimplified to keep it simple. The missing part is that the "file" that bash creates is a fifo and not a regular file, and bash runs echo and cat at the same time so that the former can keep feeding the latter through the fifo, as if the command was echo a | cat.

Note for the purist: when you factor in the second part of my description you will notice that bash must start cat before echo otherwise the pipe would hang. It's an important implementation detail, and we can be sure bash got the implementation right

"named pipe" and "fifo" are two names for the same thing.

[s243a]

I'm not sure I understand your question. The cat command doesn't need to get around anything.

I suppose I just lack intuition about when the file descriptor will be destoryed. I didn't think that the following will work but it seems to work (at least today):

Code: Select all

[root@Dpupbuster ~] $ function catw(){ cat "$1"; }
[root@Dpupbuster ~] $ catw <(echo a)
a

[step]

Perhaps you are confusing /dev/fd/63, which is simply a string whose characters spell out a path, with the file or pipe that provides the data you want to read. As a string, /dev/fd/63 is not the data. Neither as a path. In order to read the data cat really needs the file/pipe to exist while cat is reading.

In you first post, your first example: cat <(echo a)

For that bash creates a file, names it "63" located in /dev/fd, and gives cat a string "/dev/fd/63" to chew. Cat chews the string by opening the file that string refers to, etc.

Then your second example:
set -- <(echo a)
cat $1 => error file not found

For that bash creates a file ... and gives set -- the strings "/dev/fd/63" to chew. Set chews the string by storing it as positional parameter $1. At this point the line is fully executed, so bash destroy the file--not the string "/dev/fd/63" but the file.
On the next line, before starting cat, bash expands $1 to the string it is, "/dev/fd/63", then it gives cat that string to chew. Cat chews the string by opening the file that string refers to... But wait, there is no such file anymore because bash destroyed it one line ago. So cat reports an error: cat: /dev/fd/63: No such file or directory.

In your last post, your example:
function catw(){ cat "$1"; }
catw <(echo a)

bash rewrites it to
cat <(echo a)

[s243a]

Perhaps you are confusing /dev/fd/63, which is simply a string whose characters spell out a path, with the file or pipe that provides the data you want to read. As a string, /dev/fd/63 is not the data. Neither as a path. In order to read the data cat really needs the file/pipe to exist while cat is reading.

In you first post, your first example: cat <(echo a)

For that bash creates a file, names it "63" located in /dev/fd, and gives cat a string "/dev/fd/63" to chew. Cat chews the string by opening the file that string refers to, etc.

Then your second example:
set -- <(echo a)
cat $1 => error file not found

For that bash creates a file ... and gives set -- the strings "/dev/fd/63" to chew. Set chews the string by storing it as positional parameter $1. At this point the line is fully executed, so bash destroy the file--not the string "/dev/fd/63" but the file.
On the next line, before starting cat, bash expands $1 to the string it is, "/dev/fd/63", then it gives cat that string to chew. Cat chews the string by opening the file that string refers to... But wait, there is no such file anymore because bash destroyed it one line ago. So cat reports an error: cat: /dev/fd/63: No such file or directory.

In your last post, your example:
function catw(){ cat "$1"; }
catw <(echo a)

bash rewrites it to
cat <(echo a)

Thanks for the summary. It is interesting However, in my opinion bash violates the principle of least astonishment because to me it isn't obvious why bash should do different things in all these cases and while there might be a good reason for different behavior in these cases; there seems to be lots of stumbling blocks here for people learning the language. That said bash has been a very successful language despite non-obvious behavior so it must be doing something right.

Anyway, I've updted my lazy cat function and I might have addressed the previews performance issues I've had with it by redirecting the output of nohup to null:

Code: Select all

#!/bin/bash
cd "$(dirname "$0")"
source ./ppm-reduce-functions
  tmp_pipe="$(realpath .)/$(mktemp -u tmppipeXXXX)"
  mkfifo "$tmp_pipe"
  nohup bash -c "cat \"$@\" >\"$tmp_pipe\"" &>/dev/null &
  echo "$tmp_pipe"
  exec 1>/dev/null

https://github.com/s243a/ppm_reduce/blo ... azy_cat.sh

I'm using this function as part of my ppm_reduce project (see thread).

P.S. another thing that isn't obvious to me is when bash expands variables vs treating them as a string. For this reason I probably quote things and use the realpath function more than I need to.

[step]

You asked, how long does /dev/fd/63 live in a process substitution? Perhaps this command-line experiment can further your current understanding.
Here we have two verbose (-x) processes: bash -xc "..." left, and <(...) right. Left-bash waits a little then starts reading from <() via filepath $f, which is /dev/fd/63. <() waits a little then checks filepath /dev/fd/63 then writes something to stdout, then waits a little and exits.

First run. Notice how /dev/fd/63 exists for bash -c only.

Code: Select all

bash-4.4# t1=5 t2=3 t3=3; bash -xc 'f="$1"; sleep '$t1'; while read line; do echo "$line"; done < "$f"' left-bash <(set -x; sleep $t2; ls /dev/fd/63; echo hi; sleep $t3; exit)
++ sleep 3
+ f=/dev/fd/63
+ sleep 5
++ ls /dev/fd/63
ls: cannot access '/dev/fd/63': No such file or directory
++ echo hi
++ sleep 3
+ read line
+ echo hi
hi
+ read line
++ exit

Now with different timeouts. Notice how bash -c starts reading while <() is still sleeping but the end result is the same.

Code: Select all

bash-4.4# t1=3 t2=5 t3=3; bash -xc 'f="$1"; sleep '$t1'; while read line; do echo "$line"; done < "$f"' left-bash <(set -x; sleep $t2; ls /dev/fd/63; echo hi; sleep $t3; exit)
++ sleep 5
+ f=/dev/fd/63
+ sleep 3
+ read line
++ ls /dev/fd/63
ls: cannot access '/dev/fd/63': No such file or directory
++ echo hi
++ sleep 3
+ echo hi
hi
+ read line
++ exit
bash-4.4#

You can change timeouts, instructions, etc.
The reason I use a while loop instead of cat to read inside bash -c is instructional. Since the while loop is line-buffered, you can see what comes in as soon as it's read. With cat this wouldn't be possible because cat isn't line-buffered (although it can be made so).