How to use wildcards with sed -i?

[stemsee]

How to use wildcards with sed -i ? I tried the [^']* form but no success.
I want to replace an unknown string following a known string

Code: Select all

B0='asdofnnadovnsoAFGSagFsbFbFBaFBAFD
ASDFAFDSfaSDFsfDSFSdgWREg4tSFB=='

with

Code: Select all

B0='AEGHETZHGRARegsfdbgSGFGAefsgwHaevagre
wERAHAGfwafvDSFadvgaerbrvgagr'

in the form

Code: Select all

SELECTED=B0
EDITS="AEGHETZHGRARegsfdbgSGFGAefsgwHaevagre
wERAHAGfwafvDSFadvgaerbrvgagr"
DOC='/root/test.txt'
sed -i "s/$SELECTED='.*'/$SELECTED='$EDITS'/" "$DOC"

[some1]

As I understand your presentation:
Was
sed -i "s/$SELECTED='.*'/$SELECTED='$EDITS'/" "$DOC"
try

Code: Select all

sed -i 's/'$SELECTED'=.*/'$SELECTED'='$EDITS'/' "$DOC"

or (shorter)

Code: Select all

sed -i 's/\('$SELECTED'=\).*/\1'$EDITS'/' "$DOC"

(double-quote the variables i.e.... '"$EDITS"'...if needed by data)

[stemsee]

Thanks for the suggestion however it doesn't honour the '' in B0='$EDITS'

xscreenshot-20230426T202745.png (17.72 KiB) Viewed 1447 times

It adds to it without replacing all of it

xscreenshot-20230426T202941.png (20.06 KiB) Viewed 1447 times

I also tried double quoting. It cannot simply replace 'Status:' with 'Status:YES' because it will be encoded in base64 before placing in the file.

[Burunduk]

You've asked this question before in viewtopic.php?t=7371 but haven't responded anything. Have you tried the awk code? It should work. The record separator can be set to ^$ too: RS='^$'

It's possible to do it with sed if you definitely prefer sed over awk. It's a bit more tricky with multiline patterns (two lines in your example). If a replacement contains literal newlines, your code generates the "unterminated `s' command" error. To fix this you need to replace newlines with \n. Also s command separators, ampersands and backslashes should be escaped. And just like awk, sed works with lines. For multiline patterns, more lines should be read into the sed's pattern space before attempting to replace. This should work if there are no single quotes inside the original string and in the replacement:

Code: Select all

#!/bin/bash

inputfile="in.txt"
outputfile="out.txt"
pattern='A1'
replacement='Every literal newline should be escaped with "\\"\
as well as these symbols: "\&" "\\" "\/" \
Or replace them automatically before use this variable in the sed command'

# escape newlines if not escaped already in the assignment statement above
#replacement="${replacement//$'\n'/\\n}"

sed "
  /^$pattern=/{              # for multi-line patterns do
  :a
  s/'[^']*'/'$replacement'/  # try to replace
  t     # start the new cycle if the replacement succeeds
  N     # append the next line
  ba    # and try again
  }
  " "$inputfile" >"$outputfile"


# the short sed command:
#sed "/^$pattern=/{:a;s/'[^']*'/'$replacement'/;t;N;ba}" "$inputfile" >"$outputfile"

It cannot simply replace 'Status:' with 'Status:YES' because it will be encoded in base64 before placing in the file.

I don't understand this. If the string and the replacement are encoded in base64, they can be on a single line and a very simple sed's s command will work. Just choose a different separator for it because "/" is used in base64.

[stemsee]

hi @Burunduk I'm so Sorry !! I forgot about that! Seems ignorant of me. Yes, and I found another solution at that time, But I can not use my solution in this new context.

I recall thinking your code was too difficult for me to learn to use, as I was in a hurry, and I was going to get back to you!

Now I am ready to learn! Thanks for your patience!

Let me test it, and learn it.

my base64 encodings seem to conform to 80 cols limit, so occupies multiline. But anyway the important thing is to locate the single quotes pair after a search term, even if the quotes are N lines apart. Some indexes will be plain text. So one sure method for any multiline strings between single quotes.

BTW two hours with ChatGPT efforts were no better than my first few attempts with sed. In the end I gave ChatGPT my previous solution, which i cannot use in the new context.

[stemsee]

Incredible!! It works absolutely perfectly!! And so compact and precise.
Thank you @Burunduk

[Burunduk]

It works absolutely perfectly!!

Except when it doesn't! With records encoded in base64, the text file can contain something like this:

Code: Select all

A0='blablabla...
A1='
A1='...

Unfortunately it's not possible to distinguish those "A1=" without parsing.

This tries to read all records:

Code: Select all

#!/bin/bash

inputfile="in.txt"
outputfile="out.txt"
pattern='A1'
replacement='Every literal newline should be escaped with "\"
as well as these characters: "&" "\"
The next commands escape them. (It could be a single command in the recent bash.)'

# escape newlines and other characters
replacement="${replacement//\\/\\\\}"
replacement="${replacement//\&/\\\&}"
replacement="${replacement//$'\n'/\\n}"

sed "
  /^\w\w*='/{                            # if a record starts in this line
  :a                                     # read it into the pattern space line by line
  /^$pattern='/s&'[^']*'&'$replacement'& # if it is THE record, try to replace it
  /'[^']*'/!{                            # if it is not the whole record
  N                                      # append the next line
  ba                                     # and try again
  }}
  " "$inputfile" >"$outputfile"


# the short sed command:
#sed "/^\w\w*='/{:a;/^$pattern='/s&'[^']*'&'$replacement'&;/'[^']*'/!{N;ba}}" "$inputfile" >"$outputfile"

[stemsee]

Except when it doesn't! With records encoded in base64, the text file can contain something like this:

Code: Select all

A0='blablabla...
A1='
A1='...

Unfortunately it's not possible to distinguish those "A1=" without parsing.

Agreed. The search pattern needs to include the '=' in case A1 also occurs in the text. Probably I could make the indices (variable terms) more unique.

My previous solution was to append the new definition echo -e "A1=\'$EDITS\'" >> "$DOC1", then 'source "$DOC1"' and regenerate a new DOC, because 'source' keeps the last occurence of the variable, at least in my limited tests.

Code: Select all

for i in {A..Z}{0..5}
do
echo -e "$i='${!i}'" >> $DOCS
done
mv "$DOCS2" "$DOC1" 

but now the function checks for 'Status:YES' and if found aborts! Because that means no edits allowed. That's why i need to use sed to replace single defined entries.

[Burunduk]

The search pattern needs to include the '=' in case A1 also occurs in the text. Probably I could make the indices (variable terms) more unique.

The pattern includes "=" already: /^$pattern='/ - this regex matches A1=' at the start of the line but is still ambiguous because a split base64 string can contain A1=' at the start of a line too. This is why I've changed the sed script to read all the records one by one. It "consumes" A1= inside single quotes before it tries to match the pattern and shouldn't confuse indices with the data. The only limitation - no single quotes inside single quotes.

[stemsee]

The search pattern needs to include the '=' in case A1 also occurs in the text. Probably I could make the indices (variable terms) more unique.

The pattern includes "=" already: /^$pattern='/ - this regex matches A1=' at the start of the line but is still ambiguous because a split base64 string can contain A1=' at the start of a line too. This is why I've changed the sed script to read all the records one by one. It "consumes" A1= inside single quotes before it tries to match the pattern and shouldn't confuse indices with the data. The only limitation - no single quotes inside single quotes.

But if the single quotes inside single quotes are base64 encoded sed will not see them, right? So that's only for plain text, so I can use both commands in a case statement.

[Burunduk]

But if the single quotes inside single quotes are base64 encoded sed will not see them, right?

This is not supported: A1='It'\''s wrong!' and this A1='It\'s wrong too.' What's encoded is encoded, only literal single quotes should be avoided.

So that's only for plain text, so I can use both commands in a case statement.

Sorry, I don't understand, what are those both commands? sed and awk? The updated sed command should work now. If there are no \ and & in the data (or they are escaped: \\ and \&), then it can be a one-liner (it contains ! so to test it in the terminal disable the history expansion):

Code: Select all

pattern=A1
replacement='cHVwcHls
aW51eAo='
sed -i "/^\w\w*='/{:a;/^$pattern='/s&'[^']*'&'${replacement//$'\n'/\\n}'&;/'[^']*'/!{N;ba}}" "$textfile"

The awk script hasn't been updated, I'll see what I can do.

Edit: the awk's regex engine is basically the same as in sed. Not very powerful. It's hard to fix the script and keep it compact. It would be easier with PCRE:

Code: Select all

inputfile="in.txt"
outputfile="out.txt"
# export is needed for perl
export pattern='A1'
export replacement='Perl will use this variable as is.
There is no need to escape newlines or other characters here.
Test: & $ \ / # " - OK, but still no single quotes'

perl -p0e "s/^(?!\$ENV{pattern}=)\w*='.*?'(*SKIP)(*F)|(^\w*=)'.*?'/\$1'\$ENV{replacement}'/ms" "$inputfile" >"$outputfile"

[stemsee]

I've been looking at the various base64 schemes, bash's base64 uses only

• Bash supports numeric literals in Base64. Bash uses the alphabet "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ@_".

only Uenencoding and BinHex 4 use the single quote character, so not a problem.https://en.wikipedia.org/wiki/Base64

What's of interest to me is base64 is commonly used to encode binary data such as images.

[Burunduk]

Bash supports numeric literals in Base64

This is a bit different story. Bash can do some math: echo $(( 2 * 5 )). The numbers here are decimal but they can be octal (base8), hexadecimal (base16) or even base64: echo $(( 55#100 - 64#q )). What you probably need is the base64 command that can encode and decode strings. It uses a slightly different alphabet.

Code: Select all

A1=$(echo "the puppy linux" | base64)
echo "$A1" | base64 -id

---

I've rewritten the script to add a limited support for single quotes. It needs testing. It seems to work, but I'm not sure about using nested ungreedy and greedy quantifiers together and I've never used the match reset feature before. It looks interesting. Maybe I could replace (*SKIP) with it.

Code: Select all

#!/bin/bash
inputfile="in.txt"
outputfile="out.txt"
export pattern='A1'
export replacement='Perl will use this variable as is.
There is no need to escape newlines or other characters here.
Test: & $ \ / # " - OK, it should be possible now
to have a single quote here in a form that bash can interpret,
so it can be sourced: '\'' - but it can'\''t be at start or end
of the string unless with redundant single quotes: '\'''

perl -p0e "\$ENV{replacement}=~s/'/'\\\''/g;s/^(?!\$ENV{pattern}=)\w*=('(.*?('(\\\')+')?)*')(*SKIP)(*F)|^\w*=\K(?1)/'\$ENV{replacement}'/ms" "$inputfile" >"$outputfile"

---

How big your database is going to be? If it's big you should consider something like SQLite. But if it's relatively small and has a simple structure, you could use a bash associative array. This is a very basic example script (or a function) that can store values in an array and save it to a file:

Code: Select all

#!/bin/bash

# Simple bash database draft
# It can be a script or a function
# Usage:
# to add or update: <this_script> <key> <value>
# to read: <this_script> <key>
#

dbfile="bashdb.txt"
declare -A DB

# read the database from a file
[ -f "$dbfile" ] && . "$dbfile"

case $# in 
  1) # read
    printf "%s\n" "${DB["$1"]}"
    ;;
  2) # write
    DB["$1"]="$2"
    # save the database to a file
    declare -p DB >"$dbfile"
    ;;
esac

Now to store or retrieve values (the script is called sbdb here):

Code: Select all

root# ./sbdb A1 'recipe: pie
> status: ready'
root# ./sbdb A2 'comment: delicious'
root# ./sbdb A1
recipe: pie
status: ready
root# ./sbdb A1 'recipe: pie
> status: eaten'
root# ./sbdb A1
recipe: pie
status: eaten
root# 

[stemsee]

To estimate the size of a database, estimate the size of each table individually and then add the values obtained. The size of a table depends on whether the table has indexes and, if they do, what type of indexes

Image result for how to quantify the 'size' of a database
The most common way is to simply calculate the space which the database files physically consume on disk: select sum(bytes)/1024/1024/1024 size_in_GB from dba_data_files;

My present system is a homegrown pseudo database. The limits to it's size depend on the users and limitations of 'yad --form --field=:txt' and the set limit of viewable text. Each 'Table' is a text file, or indexed.book as i call them. Indexes start at [A0..Z0] and upto [A0..Z9] presently. Parsed fields presently include 'Title: Icon: Comment: Status: Password: Url:' other fields are not limited 'Note: Code: Objective: Date: ' these can be user defined. The entire index is displayed and edited manually. See here viewtopic.php?p=87607 So as the front end is limited and the data is base64 encode using a database entry system is not convenient without a full rewrite. Because of the parsed fields each index can be locked against editing, each table/book can be locked against deletion, and each index can be password protected. Implementing that at this stage using a regular database is beyond the scope of my app

xscreenshot-20230502T075408.png (191.87 KiB) Viewed 1339 times