Buffering Issues w/ Tee & stderr

For discussions about programming, and for programming questions and advice


Moderator: Forum moderators

Post Reply
s243a
Posts: 501
Joined: Mon Dec 09, 2019 7:29 pm
Has thanked: 90 times
Been thanked: 37 times

Buffering Issues w/ Tee & stderr

Post by s243a »

I wrote a script to clean private data from a save file for sandbox

The private data could be relative to the save file path or relative to the /cont folder within the save file. The /cont folder might for instance contain a chrooted iron browser. In my case the data in "/cont" is duplicated in "/" via hardlinks (see post).

To test the script I created a logger function, that copies the output of both stdout and stderr to a file.

Code: Select all

    start)
      [ "$TRACE" = true ] && SET_X=true
      exec &> >(tee -a "$LOGFILE") 

To stop the logger your restore the stderr and stdout to the default value and sop trading. There are two ways to do this. One is to first clone stdout and stderr prior to starting the logger.

Code: Select all

    case "$1" in
    init)
      [ "$TRACE" = true ] && SET_X=true
      [ ! -f "$LOGFILE" ] && rm "$LOGFILE"
      exec 6>&1           # Link file descriptor #6 with stdout.
      exec 7>&2
      [ ! -f "$LOGFILE" ] && touch "$LOGFILE"
      ;;
...
    stop)
      exec 2>&7                     
      exec 1>&6           # Link file descriptor #6 with stdout.
      ;;

Alternatively you can restore stdout and stderr as follows:

Code: Select all

    stop)
      exec 1>&6
      exec 2>/dev/tty
      ;;

The problem is that because the output of stderr is buffered (possibly due to the tee command), it takes a long time to restore. This means that the terminal might try to read the input before it prints the prompt. The solution is to add a "sync" command to the following function.

Code: Select all

function maybe_remove(){

	#log stop	
	#echo "remove \"$1\"? (y/n)" >&2
	sync
	#fsync /dev/tty
	read -p "remove \"$1\"? (y/n)" answer
	if [ "$ASK" = no ] || [[ "$answer" = y* ]] ||   [[ "$answer" = Y* ]]; then
	  return 0
	else
	  return 1
	fi
	#log start
}

However, on media with slow disk speed (e.g. USB 2.0). The sync command is very very slow! I won't see the prompt for several seconds after the terminal has been waiting for input. This is with testing on LXTerminal on dpupup Buster.

Stopping and starting logging were commented out because this technique wasn't working for me but that was prior to somewhat resolving my issues via the sync command. Here is an example of where I called this function:

Code: Select all

          maybe_remove "$a_path" && {
            log start       
            if [ -d "$a_path" ]; then
              rm -rf "$a_path"          
            elif [ -f "$a_path" ]; then
              rm -f "$a_path"
            fi
          } || log start

The syntax is a bit unorthodox. The reason for the above syntax is that I was trying to remove as much things as possible that might cause the might prevent standard error from being printed. For instance will standard error print before a command substitution exits? In the above example, I even removed the test command (i.e. [] or [[]]) and the if command. All this may have been unnecessary.

Post Reply

Return to “Programming”