Need a command to delete certain entries in a text file (Solved)

[amethyst]

Scenario: I have a directory, let's call it A. In this directory I have a lot of text files, each text file lists the paths of a user installed package. Let's say text file 1 contains the following entries:
/usr
/usr/bin
/usr/bin/eat
/usr/sbin
/usr/sbin/eating
/usr/documents
/usr/documents/egg.pdf

Okay, the idea is to keep only the entries with the full paths of actual files and delete the rest. In this example I only want to keep the following entries:
/usr/bin/eat (this is an executable)
/usr/sbin/eating (this is a script)
/usr/documents/egg.pdf

So, a command that checks the contents of all the entries of the text files in directory A and delete all entries except the entries with full paths to the actual files (like indicated in the example).

Thanks

[fredx181]

Assuming that the files are in /root/A/ , this should do it:
(it checks every line for if it's an existing directory, if true, delete the line using sed -i)

Code: Select all

DIR="/root/A"    # change as desired

for f in $(ls -d "$DIR"/*); do
  while read i; do
  [ -d "$i" ] && sed -i '\|^'"$i"'$|d' "$f"
  done < "$f"
done

[Burunduk]

change as desired

Almost identical variant only without ls and sed:

Code: Select all

#!/bin/bash

dir=A
tmpfile=`mktemp`

for file in "$dir"/*; do
  while read -r line; do
    [ -f "$line" ] && printf "%s\n" "$line" >>"$tmpfile"
  done < "$file"
  mv "$tmpfile" "$file"
done

[amethyst]

Assuming that the files are in /root/A/ , this should do it:
(it checks every line for if it's an existing directory, if true, delete the line using sed -i)

Code: Select all

DIR="/root/A"    # change as desired

for f in $(ls -d "$DIR"/*); do
  while read i; do
  [ -d "$i" ] && sed -i '\|^'"$i"'$|d' "$f"
  done < "$f"
done

Thanks, appreciated.

[MochiMoppel]

(it checks every line for if it's an existing directory, if true, delete the line using sed -i)

...but it would keep any non-existing files and directories.
Why not change the logic and keep only paths of existing files, deleting everything else?

It's possible to process all textfiles in a directory /root/A with a single sed call, eliminating the need for loops:

Code: Select all

sed -i 's/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' /root/A/*

[stemsee]

.but it would keep any non-existing files and directories.
Why not change the logic and keep only paths of existing files, deleting everything else?

It's possible to process all textfiles in a directory /root/A with a single sed call, eliminating the need for loops:

Code: Select all

sed -i 's/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' /root/A/*

Please annotate that command...

[some1]

eliminating the need for loops:

..but not cloaked loops,tmpfiles,shell-calls
Is it faster than Burunduks suggestion?

[amethyst]

Didn't check the others but Mochi's effort does not work for me when the target directory ONLY has entries of paths to actual files (it hangs and the working of my application is halted). So far Fred's effort has worked in all the scenarios I want to use it for. But I'm just reporting what I'm experiencing, you programming guys will know better.

[Trapster]

.but it would keep any non-existing files and directories.
Why not change the logic and keep only paths of existing files, deleting everything else?

It's possible to process all textfiles in a directory /root/A with a single sed call, eliminating the need for loops:

Code: Select all

sed -i 's/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' /root/A/*

Please annotate that command...

I just plugged this into chat.openai.com/chat and this was the response:

what does "sed -i 's/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' /root/A/*" do

This command uses the sed utility to perform a search and replace operation on each file specified by the /root/A/* glob.

The -i option tells sed to edit the files in place, rather than writing the output to standard output.

The s/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' part is the search and replace expression. It searches for the whole line (. matches any character, * repeats the preceding character or pattern zero or more times, so .* matches any number of characters on a line) and replaces it with the string [ -f "&" ] && echo "&". The & in the replacement string refers to the whole matched string.

The /e flag tells sed to execute the replacement as a command. The /d flag tells sed to delete any lines that are empty (that is, lines that contain only a newline character).

This command will loop through each file in /root/A/, and for each file it will check if the file exists ([ -f "&" ]) and if it does, it will print the name of the file (echo "&"). If the file does not exist or if the line is empty, it will be deleted.

It's worth noting that this command will not work as written if the filenames contain spaces or other special characters, as the shell will interpret those characters and may not pass the correct filenames to sed

[some1]

#!/bin/bash

dir=A
tmpfile=`mktemp`

for file in "$dir"/*; do
while read -r line; do
[ -f "$line" ] && printf "%s\n" "$line" >>"$tmpfile"
done < "$file"
mv "$tmpfile" "$file"
done

This
[ -f "$line" ] && printf "%s\n" "$line" >>"$tmpfile"
is inefficient
change to
[ -f "$line" ] && printf "%s\n" "$line"
and
change
done < "$file"
to
done < "$file" >"$tmpfile"
Probably reduces runtime with something like 50% .

The glob will grab eventual directories in /root/A -
so a test may be needed
[ ! -d "$file" ] && while ....

[amethyst]

It's worth noting that this command will not work as written if the filenames contain spaces or other special characters, as the shell will interpret those characters and may not pass the correct filenames to sed

Interesting. Afterwards (after I got Fred's and Mochi's responses), I was about to post another reply exactly highlighting this issue (thinking it will be an issue). So I wanted to present an example of a list with a lot of gibberish symbols and crap. However, I fell on my own sword as Fred's effort STILL worked without any issues when I tested it (maybe I didn't use enough outlandish crap, I didn't use spaces either). So I didn't post the follow up query...

[some1]

The text of the response Trapster posted has some clarification
of what MochiMoppels code is cloaking.

However - its worth noting that we are talking about 2
different kinds of filenames:
1) those that arrive via the glob /root/A/*
2) the names/paths LISTED in the files under 1)

The response - as I read it - does not seem to reflect that.

Happy NewYear

[MochiMoppel]

I posted the sed code in response to Fred's code, to demonstrate that sed is capable to do the job without multiple calls. This does not mean that it is suitable for all scenarios. I expect it to fail in case file names contain certain unusual characters, however the examples given in the first post suggested that we are dealing with reasonably normal names. If these examples are inadequate, they should be corrected.

This command will loop through each file in /root/A/, and for each file it will check if the file exists ([ -f "&" ]) and if it does, it will print the name of the file (echo "&"). If the file does not exist or if the line is empty, it will be deleted.

This is wrong. It does not check if files in /root/A/ exist. Maybe he/she means the right thing, but the explanation is wrong.

It's worth noting that this command will not work as written if the filenames contain spaces or other special characters, as the shell will interpret those characters and may not pass the correct filenames to sed

Also wrong. Filenames in /root/A/* are interpreted correctly even if they contain spaces. Only if the source files are contained in a folder like '/root/A B C', then the source of sed needs quoting, such as '/root/A B C'/*

This was my test scenario. Source dir is '/root/A B C'. A textfile (name contains spaces!) with paths to 4 existing files, 1 existing folder and 2 completely bogus entries. The code succeeds if the paths in the textfiles are reduced to the 4 existing file paths.

1) Fred's code failed. Requires additional quotes in
for f in "$(ls -d "$DIR"/*)"; do
After that only the directory was removed but the bogus files remained - as expected, because the purpose of his code is different from the one in Burunduk's an my code.

2) Burunduk's code worked as expected. With minor changes this would be the code I suggest for productive use. Could even be tweaked into using no external tools, only bash built-ins.

3) My code also worked as expected. I also changed the textfile to contain only existing file paths. This also worked, i.e. nothing was changed.

I expected that filesnames with quotation marks or backslashes would cause problems, but bash and ROX didn't allow me to create such files in the first place, so I didn't bother.

acid_files.png (44.7 KiB) Viewed 2373 times

[some1]

Thanks!
So - as I understand you -
when sed executes the shell-command -
the filename -(line in a text-file) is safely
evaluated by the shell and printed by sed ?
Allways ?
(If the file exist)

Edit:
Ok - judging from the pic - LISTED filenames
seem to be safe with spaces etc.

[MochiMoppel]

So - as I understand you -
when sed executes the shell-command -
the filename -(line in a text-file) is safely
evaluated by the shell and printed by sed ?

Normally

Allways ?

No. File names containing quotation marks (not allowed in FAT32, that's why I had trouble to create them) will not work without extra precautions.
[ -f "&" ] in sed is not the same as [ -f "$line" ] in bash . Bash evaluates $line first but sed sees only a string containing a quotation mark, so sed will fail while bash (as in Burunduk's code) will succeed. Backslashes are OK though.

[some1]

tyvm

Very nicely explained.
Thanks!!

[amethyst]

Checked Mochimoppel's code again. This is what happens for me:
If the file that needs to be changed is in my target directory named /tmp/packages/ydrv_files (ie: /tmp/packages/ydrv_files/pete_files), then the code works and the contents are changed correctly.
If the same file is in my target directory /tmp/packages (ie: /tmp/packages/pete_files, the code does not work for me. The contents are not changed. So, don't know what the probablem is here, in both cases the target directory is definitely stipulated correctly (I can see the file listed when running my application so I can select it). And as mentioned, I do not have this issue when using Fred's code. Probably something simple that I'm missing (a / issue maybe)?

PS: Burunduk's code also works for me. Between Fred's and Burunduk's code, which one would the programmers recommend for me to use for my purposes? Burunduk's code not involving ls and sed so seems simpler achieving the same result? I've also tested some1's suggestions for the changes to Burunduk's code, it also works. Should I go with Burunduk's code, altered with some1's suggestions? Anyways - thanks to all for their input here, very informative.

[Trapster]

This command will loop through each file in /root/A/, and for each file it will check if the file exists ([ -f "&" ]) and if it does, it will print the name of the file (echo "&"). If the file does not exist or if the line is empty, it will be deleted.

This is wrong. It does not check if files in /root/A/ exist. Maybe he/she means the right thing, but the explanation is wrong.

It's worth noting that this command will not work as written if the filenames contain spaces or other special characters, as the shell will interpret those characters and may not pass the correct filenames to sed

Also wrong. Filenames in /root/A/* are interpreted correctly even if they contain spaces. Only if the source files are contained in a folder like '/root/A B C', then the source of sed needs quoting, such as '/root/A B C'/*

This was my test scenario. Source dir is '/root/A B C'. A textfile (name contains spaces!) with paths to 4 existing files, 1 existing folder and 2 completely bogus entries. The code succeeds if the paths in the textfiles are reduced to the 4 existing file paths.

Please Note:
As stated in my post, I quoted Openai.chat.com.
The comments were NOT MY analysis of your code.

[Burunduk]

Checked Mochimoppel's code again. This is what happens for me:
If the file that needs to be changed is in my target directory named /tmp/packages/ydrv_files (ie: /tmp/packages/ydrv_files/pete_files), then the code works and the contents are changed correctly.
If the same file is in my target directory /tmp/packages (ie: /tmp/packages/pete_files, the code does not work for me.

/tmp/packages/ probably still contains the /tmp/packages/ydrv_files sub-directory. This breaks the code. You need to eliminate directories from the command. Use a more precise glob pattern if possible, or find /root/A -maxdepth 1 -type f ... My script prints some error messages in this situation but works anyway.

File names containing quotation marks (not allowed in FAT32, that's why I had trouble to create them) will not work without extra precautions.

This should work with filenames containing double quotes (leafpad won't start if a filename contains "; leafpad "):

Code: Select all

sed -i 's/[$"`]/\\&/g; s/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' /root/A/*

It seems that sed starts a new shell for each command. It's very slow on my computer.

A test: 101 files 1000 lines per file (500 real and 500 dummy file paths)

Scripts:

1. improved by some1

   Code: Select all

   #!/bin/bash
   
   dir=A
   tmpfile=`mktemp`
   
   for file in "$dir"/*; do
     while read -r line; do
       [ -f "$line" ] && printf "%s\n" "$line"
     done <"$file" >"$tmpfile"
     mv "$tmpfile" "$file"
   done
   

   time ...: 100%
   strace -fc ...: 790k calls total

2. my original variant

   time ...: 133%
   strace -fc ...: 1294k calls total

3. MochiMoppel's one-liner

   time ...: 12066%
   strace -fc ...: 44696k calls total

Edit: It's about 6 minutes. Don't know why it's so slow. I've compiled sed 4.9 - it's not faster.

[some1]

@Burunduk: Thanks.Nice work.

I will read your post closely,later.

The code below has some condiions:
1) ensures that only regular files are sought read
2) pruned files may become empty -
if so - the code removes them
These conditions may be relevant or not in the y_drv-territory -
I dont really know.
Amethyst can decide.

Code: Select all

!/bin/bash
                                      
tmpf=`mktemp`
dir=/root/A

for xfile in "$dir"/*;do
[ ! -f "$xfile" ] && continue

while read -r line; do
[ -f "$line" ] && printf "%s\n" "$line"
done <"$xfile">"$tmpf"

[ -s "$tmpf" ] && mv "$tmpf" "$xfile" || rm  "$xfile"
done

Edit: found a lost $

[MochiMoppel]

/tmp/packages/ probably still contains the /tmp/packages/ydrv_files sub-directory. This breaks the code.

A /tmp/packages/ydrv_files sub-directory alone wouldn't be a problem because sed ignores directories with a harmless error message like sed: read error on /tmp/packages/ydrv_files: Is a directory. It still would process the textfiles.

This should work with filenames containing double quotes (leafpad won't start if a filename contains "; leafpad "):

Code: Select all

sed -i 's/[$"`]/\\&/g; s/.*/[ -f "&" ] \&\& echo "&"/e; /^$/d' /root/A/*

It does. That's what I meant with "will not work without extra precautions". Another way would be to outsource the whole replace section into a function, which should avoid the need to escape individual characters, but then the whole point to use sed becomes questionable. In the end everything sed does can be done with bash.

It seems that sed starts a new shell for each command.

Yes, it's calling the shell for every line it finds. Did I mention that this solution is not built for speed ? Probably the slowest kid on the block.

[Burunduk]

A /tmp/packages/ydrv_files sub-directory alone wouldn't be a problem because sed ignores directories with a harmless error message like sed: read error on /tmp/packages/ydrv_files: Is a directory. It still would process the textfiles.

Not so harmless here. It stops on this error:

Code: Select all

root# for c in a b c e f; do echo $c >$c; done; mkdir d
root# sed '' a b c d e f
a
b
c
sed: read error on d: Is a directory
root# echo $?
4

[MochiMoppel]

Not so harmless here. It stops on this error:

You are right. I tested with directory name ydrv_files, and as this name came last in alphabetical list created by /root/A/* the error didn't matter.

[amethyst]

I'm going to use the code below but in addition I want the process to DELETE any DIRECTORIES and FILES in the target directory which do not contain any entries to full paths of actual files (ultimately). Thanks.

Code: Select all

D=/tmp/.packages
tmpfile=`mktemp`

for file in "$D"/*; do
  while read -r line; do
    [ -f "$line" ] && printf "%s\n" "$line"
  done <"$file" >"$tmpfile"
  mv "$tmpfile" "$file"
done

[some1]

I'm going to use the code below

Yes - thats efficient

but in addition I want the process to DELETE any DIRECTORIES and FILES in the target directory which do not contain any entries to full paths of actual files (ultimately)

What do you mean ????
"target-directory": is that "$D" in your code-adaption?
Do you have DIRECTORIES in "$D",which contains FILES,
which contains LISTINGS?

Please explain - f.x. draw a tree like in your first post
with some comments

Have you tried my code,understood how it works ?

[amethyst]

I'm going to use the code below

Yes - thats efficient

but in addition I want the process to DELETE any DIRECTORIES and FILES in the target directory which do not contain any entries to full paths of actual files (ultimately)

What do you mean ????
"target-directory": is that "$D" in your code-adaption?
Do you have DIRECTORIES in "$D",which contains FILES,
which contains LISTINGS?

Please explain - f.x. draw a tree like in your first post
with some comments

Have you tried my code,understood how it works ?

After this code is run, there may be directories and text files (especially text files) that will still be there but are now empty with no entries (which is the desired result after the code). However, I want these empty files to be removed because they still show up in my application's list of packages (although the file has no more entries). Example: there are text files that do not contain any full paths to actual files (but have other contents). The code will remove these contents but the (empty) file will still be there."$D" is the target directory which has files as well subdirectories with files. So in effect - we want to delete all empty text files.

[some1]

After this code is run, there may be directories and text files (especially text files) that will still be there but are now empty with no entries (which is the desired result after the code).

My code intentds to remove a textlife - which has become
empty - because none of the original listed files exists on the actual
system.
Run my code - and see if it works on the toplevel of text-files,
My code is NOT the code you have adapted.

As I understand you - you also have SUBDIRS and in the subdirs you have
FILES with listings.
So - if a FILE becomes empty because it has no items which
exist on the system-you want the FILE removed.
if a SUBDIR becomes empty - because all FILES becomes removed,
you want the SUBDIR removed.
Is that your scenario /wish -now ???

If so - then you have changed the playing field - to something more complicated.

-------
The code you have adapted only finds files on the toplevel.
The code will throw an error if it meets a subdir in the toplevel.
The code will NOT remove a file - which becomes empty.
The code will NOT recurse into SUBDIRS.'

My code is not much better - from your point of view -
because it does NOT recurse into subdirs.
Instead it deliberately skips subdirs - to avoid errors.
However - its intended to remove FILES at the toplevel -
which has become empty.
Try to put a text-file containing bogus-filepaths in the toplevel
- and run my code.
The code should remove the text-file -
and if so - and if thats what you want at the toplevel and in
lower levels -please confirm,comment.

How many dirs and files do you think you have/will have under the toplevel - roughly -before the pruning starts?
The size of the sinkhole may matter.

------
If my script works -and nobody wants to go there -
you *could* try to figure out how to run the script
successively - with each subdir as a toplevel .
That way you will prune all listings,delete files
which becomes empty.
Then you can delete empty subdirs - by hand or a script.

Think it over,consider your feedback.

[amethyst]

@some1

Code: Select all

D=/tmp/.packages
tmpfile=`mktemp`

for file in "$D"/*; do
  while read -r line; do
    [ -f "$line" ] && printf "%s\n" "$line"
  done <"$file" >"$tmpfile"
  mv "$tmpfile" "$file"
done

The above code works perfectly for me as far as pruning the files correctly but it still leaves empty text files in "$D"/* which I want removed automatically (with code, not manually). So simply - there must be no empty text files (ie. files with no entries "inside") left anywhere in "$D"/* . Don't know how to put this any clearer.

Your code posted neither prunes nor deletes text files in "$D"/* which do not have any entries of full paths to actual files...in my results, anyway.

[Burunduk]

If your /tmp/.packages directory has sub-directories and you want to process all the files there recursively, this may work for you:

Code: Select all

#!/bin/bash

dir=/tmp/.packages
tmpfile=`mktemp`

while read -r file; do
  while IFS= read -r line; do
    [ -f "$line" ] && printf "%s\n" "$line" 
  done <"$file" >"$tmpfile"
  mv "$tmpfile" "$file"
done < <(find "$dir" -type f)

find "$dir" -mindepth 1 -empty -delete

Notes:

• The first find command searches for regular files in the /tmp/.packages recursively. The process substitution is used as in the example here but it's not the only option. You can use find's -fprint and another tempfile, a globstar or whatever if you prefer.

• The second find removes empty files and directories inside /tmp/.packages Change it to find "$dir" -type f -empty -delete if you want to remove just the files.

• IFS= is here to deal with strange filenames that contains trailing spaces. If you expect a weird characters (like newline) in the filenames, you probably should use a null separator.

About symbolic links: the code preserves entries pointing to symlinks to real regular files.

[amethyst]

If your /tmp/.packages directory has sub-directories and you want to process all the files there recursively, this may work for you:

Code: Select all

#!/bin/bash

dir=/tmp/.packages
tmpfile=`mktemp`

while read -r file; do
  while IFS= read -r line; do
    [ -f "$line" ] && printf "%s\n" "$line" 
  done <"$file" >"$tmpfile"
  mv "$tmpfile" "$file"
done < <(find "$dir" -type f)

find "$dir" -mindepth 1 -empty -delete

Notes:

• The first find command searches for regular files in the /tmp/.packages recursively. The process substitution is used as in the example here but it's not the only option. You can use find's -fprint and another tempfile, a globstar or whatever if you prefer.

• The second find removes empty files and directories inside /tmp/.packages Change it to find "$dir" -type f -empty -delete if you want to remove just the files.

• IFS= is here to deal with strange filenames that contains trailing spaces. If you expect a weird characters (like newline) in the filenames, you probably should use a null separator.

About symbolic links: the code preserves entries pointing to symlinks to real regular files.

Thanks, I'll try that solution which looks to be watertight.
In the meantime, I did run the code below which gave me the result I wanted

Code: Select all

cd /"$D"
find . -type f -empty -print -delete