Bash echo within awk hex conversion?

[blumenwesen]

Hi, can echo in awk be used to convert a hexadecimal number into a character?
These are my four attempts that work without the variable indentation from $0.

Code: Select all

echo "61" | awk '{print | "echo \\\x"$0""}'
# echo "61" | awk '{print | "echo \\\x61"}'
echo "61" | awk '{system("echo \\\x"$0"")}'
# echo "61" | awk '{system("echo \\\x61")}'
echo "61" | awk '{print | eval "echo \\\x"$0""}'
# echo "61" | awk '{print | eval "echo \\\x61"}'
echo "61" | awk '{system("bash -c '\''echo \\\x"$0"'\''")}'
# echo "61" | awk '{system("bash -c '\''echo \\\x61'\''")}'

echo "61" | awk '{eval "read a<<<$(echo -e \\\x"$0")"; print $a}'

The last would be a redirect but I don't know how the variable can be routed from awk and back in.

[step]

Hi, welcome to the forum. if I understood your question, here's one way to echo '61' to awk and get '=' on output

Code: Select all

 echo 61 | awk '{printf "%c", $0; exit}'

If you want to do the same without awk, just with shell commands, here's one way

Code: Select all

printf "\x$(printf %x 61)"

[blumenwesen]

Thanks, I wanted to limit this script as oneliner within awk.
However, wanted to reach whole only with echo.

Code: Select all

a=$(zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); print $2}}')
for z in $a; do
    echo -e \\x"$z $z" 
done

[MochiMoppel]

here's one way to echo '61' to awk and get '=' on output

Code: Select all

 echo 61 | awk '{printf "%c", $0; exit}'

I think blumenwesen needs to convert a hexadecimal number Your solution would work only with decimal 61 (character '='), but not with hex 61 (character 'a').

@blumenwesen
Try

Code: Select all

a=$(zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); system("echo -e \\\\x"$2" "$2)}}')
echo "$a"

Your NR values seem a bit off (or my UTF-8.gz is different), but it should give you an idea how to do conversion with echo in awk.

[blumenwesen]

Echo doesn't get the conversion, I think it has to go from awk out to echo and then back to awk.
Also don't get the function within awk right, but maybe it works with getline.

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2);}} END function fun(s){system("exec bash -c '\''echo \"\x"$1"\"'\''"); return}{fun($0); print}'

[MochiMoppel]

Echo doesn't get the conversion

It does. Here the code I posted returns exactly the same output as your preceding code containing the for loop. The only difference is that it doesn't need a loop because conversion is done within awk, using echo, and that's what you asked for, didn't you? Please post any error messages.

In principle the hex2char conversion is very close to your 3rd example in your initial post, which does not work because it contains 2 syntax errors:

Code: Select all

echo "61" | awk '{system("echo \\\x"$0"")}'

It would work if you change it to

Code: Select all

echo "61" | awk '{system("echo -e \\\\x"$0"")}'

[blumenwesen]

I get:

Code: Select all

echo "61" | awk '{system("echo -e \\\\x"$0"")}'
# -e \x61

echo "61" | awk '{system("echo \\\x"$0"")}'
# awk: Command line:1: Warning: In the "\x" escape sequence are no hexadecimal numbers
# x61

[MochiMoppel]

@blumenwesen Are you using something other than GNU awk? I'm using GNU awk, symlinked to gawk

Code: Select all

# awk --version
GNU Awk 3.1.8
Copyright (C) 1989, 1991-2010 Free Software Foundation.

Reason why I suspect an awk incompatibility: I'm getting a slightly different error message:

Code: Select all

# echo "61" | awk '{system("echo \\\x"$0"")}'
awk: warning: no hex digits in `\x' escape sequence
x61

[blumenwesen]

have Awk 5.1.0 Copyright © 1989, 1991-2020

[williams2]

Code: Select all

# awk --version
GNU Awk 4.1.4
#
# echo 61 | awk '{printf "%X",$1}'
3D# 
# echo 61 | awk '{printf "%X\n",$0}'
3D
#

[step]

Code: Select all

# echo 61 | busybox awk '{printf "%c\n",0+sprintf("%d","0x"$0)}'
a
# echo 61 | gawk -Wposix '{printf "%c\n",0+sprintf("%d","0x"$0)}'
a
# echo 61 | mawk '{printf "%c\n",0+sprintf("%d","0x"$0)}'
a

edit: added -Wposix

[misko_2083]

Code: Select all

for i in {61..67}; do echo $i; done | awk '{ x = $0; cmd = "printf %x"x""; cmd;  close(cmd, "to"); cmd |& getline out;  printf "%c ", out;  close(cmd);}'
= > ? @ A B C

[williams2]

Are you trying to convert hex to decimal?

Code: Select all

# echo fe | gawk -Wposix '{printf("%d\n","0x" $1)}'
254
# echo 0a | gawk -Wposix '{printf("%d\n","0x" $1)}'
10
# echo 0b | gawk -Wposix '{printf("%d\n","0x" $1)}'
11
#

[misko_2083]

Are you trying to convert hex to decimal?

No.
Actually yes.

Code: Select all

# for i in {61..67}; do echo $i; done | awk '{ x = $0; cmd = "printf %d 0x"x""; cmd;  close(cmd, "to"); cmd |& getline out;  printf "%c ", out;  close(cmd);}'
a b c d e f g 

I think gawk default shell is dash

Code: Select all

# sh -c "echo  -e \\\x61"
-e \x61
bash -c "echo  -e \\\x61"
a

This works only in gawk

Code: Select all

echo 61 | awk '{ printf "%c\n", strtonum( "0x"$1 )}'

He wants to do this

Code: Select all

# echo 61 | awk  '{ cmd = "echo $((0x"$0"))";cmd;close(cmd, "to"); cmd |& getline out;  printf "%c\n", out;  close(cmd);}'
a
# zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); cmd = "echo $((0x"$2"))";cmd;close(cmd, "to"); cmd |& getline out; printf "%c %s\n", out, $2; close(cmd);}}'

[MochiMoppel]

have Awk 5.1.0 Copyright © 1989, 1991-2020

???
I suspect that you are not using Puppy. If you do, which version?

Your earlier posts suggests that

Code: Select all

# echo -e \\x61
a

works for you. This would be bash.

However it appears that in your case awk's system command doesn't call bash but a different shell, maybe dash (which doesn't support echo -e). I don't know how to enforce bash with 'bash -c' without getting in trouble with nested quotations. One easy workaround would be to use busybox echo, if busybox is installed and you give up your insistence on using bash.

In any case bash echo (or printf) is the way to go when you need hex2char or char2hex conversions. From bash 4.2 echo supports the -U option \U escape sequence, converting to any Unicode character:

Code: Select all

# echo -e \\U61        
a
# echo -e \\U20AC
€

[misko_2083]

However it appears that in your case awk's system command doesn't call bash but a different shell, maybe dash (which doesn't support echo -e).

That's what I wrote but not in so many words.

For simple commands awk can always go back to the shell.

Code: Select all

echo | awk '{var="'"`echo $0`"'"; print var}'
bash

@blumenwesen
char "a" dec 97 hex 61
char "=" dec 61 hex 3d
You probably wanted to limit the characters that can't be displayed, like non-printing, 2-byte characters, 3 byte-characters and 4-byte characters.

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); cmd = "echo $((0x"$2"))";cmd;close(cmd, "to"); cmd |& getline out; printf "char \"%c\" dec %s hex %s\n", out, out, $2; close(cmd);}}'
char " " dec 32 hex 20
char "!" dec 33 hex 21
char """ dec 34 hex 22
char "#" dec 35 hex 23
char "$" dec 36 hex 24
char "%" dec 37 hex 25
char "&" dec 38 hex 26
char "'" dec 39 hex 27
char "(" dec 40 hex 28
char ")" dec 41 hex 29
char "*" dec 42 hex 2a
char "+" dec 43 hex 2b
char "," dec 44 hex 2c
char "-" dec 45 hex 2d
char "." dec 46 hex 2e
char "/" dec 47 hex 2f
char "0" dec 48 hex 30
char "1" dec 49 hex 31
char "2" dec 50 hex 32
char "3" dec 51 hex 33
char "4" dec 52 hex 34
char "5" dec 53 hex 35
char "6" dec 54 hex 36
char "7" dec 55 hex 37
char "8" dec 56 hex 38
char "9" dec 57 hex 39
char ":" dec 58 hex 3a
char ";" dec 59 hex 3b
char "<" dec 60 hex 3c
char "=" dec 61 hex 3d
char ">" dec 62 hex 3e
char "?" dec 63 hex 3f
char "@" dec 64 hex 40
char "A" dec 65 hex 41
char "B" dec 66 hex 42
char "C" dec 67 hex 43
char "D" dec 68 hex 44
char "E" dec 69 hex 45
char "F" dec 70 hex 46
char "G" dec 71 hex 47
char "H" dec 72 hex 48
char "I" dec 73 hex 49
char "J" dec 74 hex 4a
char "K" dec 75 hex 4b
char "L" dec 76 hex 4c
char "M" dec 77 hex 4d
char "N" dec 78 hex 4e
char "O" dec 79 hex 4f
char "P" dec 80 hex 50
char "Q" dec 81 hex 51
char "R" dec 82 hex 52
char "S" dec 83 hex 53
char "T" dec 84 hex 54
char "U" dec 85 hex 55
char "V" dec 86 hex 56
char "W" dec 87 hex 57
char "X" dec 88 hex 58
char "Y" dec 89 hex 59
char "Z" dec 90 hex 5a
char "[" dec 91 hex 5b
char "\" dec 92 hex 5c
char "]" dec 93 hex 5d
char "^" dec 94 hex 5e
char "_" dec 95 hex 5f
char "`" dec 96 hex 60
char "a" dec 97 hex 61
char "b" dec 98 hex 62
char "c" dec 99 hex 63
char "d" dec 100 hex 64
char "e" dec 101 hex 65
char "f" dec 102 hex 66
char "g" dec 103 hex 67
char "h" dec 104 hex 68
char "i" dec 105 hex 69
char "j" dec 106 hex 6a
char "k" dec 107 hex 6b
char "l" dec 108 hex 6c
char "m" dec 109 hex 6d
char "n" dec 110 hex 6e
char "o" dec 111 hex 6f
char "p" dec 112 hex 70
char "q" dec 113 hex 71
char "r" dec 114 hex 72
char "s" dec 115 hex 73
char "t" dec 116 hex 74
char "u" dec 117 hex 75
char "v" dec 118 hex 76
char "w" dec 119 hex 77
char "x" dec 120 hex 78
char "y" dec 121 hex 79
char "z" dec 122 hex 7a
char "{" dec 123 hex 7b
char "|" dec 124 hex 7c
char "}" dec 125 hex 7d
char "~" dec 126 hex 7e
char "" dec 127 hex 7f

It's interesting to see how UTF-8 works https://stackoverflow.com/questions/44 ... s#44568131

[blumenwesen]

Sorry, Echo runs normally for me and I also get "a" from "echo -e \\U61" but I was hoping that I can use echo instead of prinf in awk.

[misko_2083]

^Does it works if you use the full path to echo?
e.g. /usr/bin/echo

Code: Select all

# echo "61" | awk '{system("/usr/bin/echo -e \\\\x"$0"")}'

[MochiMoppel]

That's what I wrote but not in so many words.

Yeah, and because you used few words you finished your post while I was still editing. Can make slow posters like me look pretty stupid when they appear to repeat what a previous poster said already.

I think gawk default shell is dash

Firstly, @blumenwesen isn't using gawk (=GNU awk), and secondly my gawk is using bash. Dash isn't even installed. That's why I find it important that blumenwesen tells us which Puppy distro he is using. I hope he will.

I was hoping that I can use echo instead of prinf in awk.

I think you are confusing 3 different printf commands, none of them seems usable for you.
1) Awk has its own builtin printf command but it can't do hex2char conversion, so not useful for your purpose
2) Bash builtin printf command can do the conversion but your awk would prefer dash instead of bash
3) Dash builtin printf command can't do the conversion (judging from the manual, I never used dash so I may be wrong)
There is a fourth printf that probably does work for you. See below

^Does it works if you use the full path to echo?
e.g. /usr/bin/echo

Code: Select all

# echo "61" | awk '{system("/usr/bin/echo -e \\\\x"$0"")}'

If with "full path" you mean symlink to busybox:That's what I already proposed to blumenwesen. Busybox echo and printf both work for me:

Code: Select all

# echo "61" | awk '{system("busybox echo -e \\\\x"$0"")}' 
a
# echo "61" | awk '{system("busybox printf \\\\x"$0"")}' 
a

[misko_2083]

That's what I wrote but not in so many words.

Yeah, and because you used few words you finished your post while I was still editing. Can make slow posters like me look pretty stupid when they appear to repeat what a previous poster said already.

https://youtube.com/clip/UgkxOVhHbzsiVs ... zH3IljZtjv

[blumenwesen]

Yes it worked well I just noticed it now.
Thanks for solving the problem.

[some1]

@blumenwesen:

???

What worked well?
What distro do you use ?

[blumenwesen]

Have installed 6.4 but then completely replaced it with current ones, and this both echo commands works very well.

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); system("/usr/bin/echo -e \\\\x"$2"")}}'
zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); system("busybox echo -e \\\\x"$2"")}}'

[some1]

Thanks!
The codepieces are slow -
lets speed them up with a bit extra code.

Some timings.
The codepieces are one-liners in production -
but may be easier reads this way.
Copy code into terminal -pres enter.

I am on
#Fatdog 812
#Bash 4.4.18
#gawk 5.1

The codepieces for /bin/echo and busybox echo
should work for you.

#For reference

Code: Select all

#GAWK strtonum
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	printf "%c\t%s\n", strtonum( "0x" d),d;next};{exit}'

real 0m0.011s
user 0m0.013s
sys 0m0.005s
#

Code: Select all

#TOO MANY MISKO
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<138) {gsub(/\/x/, "", $2); cmd = "echo $((0x"$2"))";cmd;close(cmd, "to"); cmd |& getline out; printf "%c %s\n", out, $2; close(cmd);}}'

real 0m0.332s
user 0m0.260s
sys 0m0.104s
####################################
####################################

Code: Select all

#TOO MANY (bash) echo -e
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	str="\\\\x" d "\t" d;
	system("echo -e " str);next};
    {exit}'

real 0m0.318s
user 0m0.248s
sys 0m0.081s

Code: Select all

#TOO MANY /bin/echo -e
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	str="\\\\x" d "\t" d;
	system("/bin/echo -e " str);next}
    {exit}'

real 0m0.511s
user 0m0.388s
sys 0m0.133s

Code: Select all

#TOO MANY busybox echo -e
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	str="\\\\x" d "\t" d;
	system("busybox echo -e " str);next}
    {exit}'

real 0m0.547s
user 0m0.380s
sys 0m0.176s
###############################
###############################
Speed-stripes coming up:

Code: Select all

#BULK (bash) echo -e 
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	 str=str "\\\\x" d "\t" d "\n";next};	
     {exit}END {cmd="echo -en " "\"" str "\"";system(cmd);}'

real 0m0.015s
user 0m0.017s
sys 0m0.008s
#

Code: Select all

    
#BULK /bin/echo -e 
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	 str=str "\\\\x" d "\t" d "\n";next};	
     {exit}END {cmd="/bin/echo -en " "\"" str "\"";system(cmd);}'

real 0m0.015s
user 0m0.018s
sys 0m0.007s

Code: Select all

 
#BULK busybox echo -e 
time zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR < 42){next};((NR < 138) && ( $2 ~ /\/x/ )){d=substr($2,3);
	 str=str "\\\\x" d "\t" d "\n";next};	
     {exit}END {cmd="busybox echo -en " "\"" str "\"";system(cmd);}'

real 0m0.015s
user 0m0.017s
sys 0m0.009s
#

[MochiMoppel]

UTF-8.gz is a file that contains about 38000 Unicode characters, but so far the code is limited to single byte characters.
This variant can convert hex strings to any of the Unicode characters:

Code: Select all

# zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '{if(NR>7400 && NR<7413) {gsub("/","\\\\"); system("busybox echo -e "$2)}}'        
⇳
⇴
⇵
⇶
⇷
⇸
⇹
⇺
⇻
⇼
⇽
⇾

The same with Unicode names:

Code: Select all

# zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '{if(NR>7400 && NR<7413) {gsub("/","\\\\"); system("busybox echo -ne "$2); $1=""; $2=""; print}}'
⇳  UP DOWN WHITE ARROW
⇴  RIGHT ARROW WITH SMALL CIRCLE
⇵  DOWNWARDS ARROW LEFTWARDS OF UPWARDS ARROW
⇶  THREE RIGHTWARDS ARROWS
⇷  LEFTWARDS ARROW WITH VERTICAL STROKE
⇸  RIGHTWARDS ARROW WITH VERTICAL STROKE
⇹  LEFT RIGHT ARROW WITH VERTICAL STROKE
⇺  LEFTWARDS ARROW WITH DOUBLE VERTICAL STROKE
⇻  RIGHTWARDS ARROW WITH DOUBLE VERTICAL STROKE
⇼  LEFT RIGHT ARROW WITH DOUBLE VERTICAL STROKE
⇽  LEFTWARDS OPEN-HEADED ARROW
⇾  RIGHTWARDS OPEN-HEADED ARROW

[blumenwesen]

Is it possible from 138 subsequent characters (⇳, ֆ, ⇵, փ, ...) also to show?

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '/\/x/ {if(NR>41 && NR<1138) {gsub(/\/x/, "", $2); system("/usr/bin/echo -e \\\\u"$2" "$3" "$4" "$5" "$6" "$7" "$8" "$9)}}'
zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '{if(NR>42 && NR<1413) {gsub("/","\\\\"); system("/usr/bin/echo -e \u"$2" "$2" "$3" "$4" "$5" "$6" "$7" "$8" "$9)}}'

I only get "\ud6/2 d6/xa2", don't know how I can adapt and cut off the system/echo output in the second one, if possible this should be visible.
A U0041 /x41 LATIN CAPITAL LETTER A
B U0042 /x42 LATIN CAPITAL LETTER B
...
₮ U20AE /xe2/x82/xae TUGRIK SIGN
₯ U20AF /xe2/x82/xaf DRACHMA SIGN
...
EDIT: wrong code

[MochiMoppel]

I only get "\ud6/2 d6/xa2"

Doing what? Running the code you posted? I have no clue what your problem is.

[misko_2083]

That line is working fine here.
+ I can use yad and xclip to select and copy emojis to clipboard.

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '{if(NR>42702 && NR<44490) {gsub("/","\\\\"); system("busybox echo -ne "$2); $1="\n"; $2=""; print}}'| yad --width=400 --height=800 --list --column="#" --column="Name" --select-action='bash -c ": ${CLIP:=%s}; echo "${CLIP%%" "*}" | xclip -selection clipboard"'

or other characters

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '/SEGMENTED/{gsub("/","\\\\"); system("busybox echo -ne "$2); $1="\n"; $2=""; print}'| yad --width=300 --height=500 --list --column="#" --column=Name --select-action='bash -c ": ${CLIP:=%s}; echo "${CLIP%%" "*}" | xclip -selection clipboard"'

Note if it's not working with your version of yad change --select-action to --dclick-action

[blumenwesen]

I've done it now.

Code: Select all

zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '{if(NR>42 && NR<1135) {gsub(/\//, "\\\\"); a=system("/usr/bin/echo -ne "$2); gsub(/\\\\/, " ", $2); print " "$2" "substr($1, 3, 4)" "$3" "$4" "$5" "$6" "$7" "$8" "$9}}'

zcat /usr/share/i18n/charmaps/UTF-8.gz  | awk '{if(NR>42 && NR<1135) {gsub(/\//, "\\\\"); system("/usr/bin/echo -ne "$2); gsub(/>/, "", $1); gsub(/\\\\/, " ", $2); gsub(/<U|[ ][ ]/, " "); print}}'

zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '(NR<42){next}; ((NR<48649)&&($2~/\/x/)){z=$2; gsub(/\//, "\\\\", z); system("/bin/echo -e \""z" "$1" "$2" "$3" "$4" "$5" "$6" "$7" "$8" "$9"\""); next}'

Is it impossible to converse with "\u1" with /usr/bin/echo?

[MochiMoppel]

EDIT: wrong code

Was correct, now it's wrong. At least explains your error complaint.

Is it impossible to converse with "\u1" with /usr/bin/echo?

Yes, impossible!

I've done it now.

The 3rd code comes close to what you asked for, but you should correct the max NR. Even if you set it to NR<1135 it is very slow. Here it takes 13sec.

If you give up your idea to run echo inside awk and instead run it outside the code will execute much, much faster because you don't need the inefficient system() function and the busybox workaround anymore.

Instead of 13sec only 0.3sec, piped to leafpad for better readability

Code: Select all

echo -e "$(zcat /usr/share/i18n/charmaps/UTF-8.gz | awk '{if(NR>43 && NR<1135) {a=$0; gsub("/","\\"); print $2"\t"a}}')" | leafpad

Using the "\U" escape character allows even simpler code:

Code: Select all

echo -e "$(zcat /usr/share/i18n/charmaps/UTF-8.gz | awk -F"[<> ]+" '{if(NR>43 && NR<1135) print "\\"$2"\t"$0}')" | leafpad

Lastly using sed instead of awk:

Code: Select all

echo -e "$(zcat /usr/share/i18n/charmaps/UTF-8.gz | sed -rn '44,1134 s@<(.+)>(.+)@\\\1\t\1\2@p')" | leafpad