IQ4sh - Calculator for CLI or scripts -Testers wanted

[JakeSFR]

Code: Select all

M_int=${M_int#${M_int%%[^0+-]*}}

That's very nice indeed -something I hadn't really looked into enough. Just tried that under bash, zsh, dash and zsh, where it works.

Which dash are you using?
It doesn't work in Fatdog's dash-0.5.9.1:

Code: Select all

# dash
# M_int="000000000000001.23"; echo ${M_int#${M_int%%[^0+-]*}}
000000000000001.23
# 

I also built the latest (v0.5.11.5) with the same result.

What works though (which I discovered recently) is this:

Code: Select all

# dash
# M_int="000000000000001.23"; echo ${M_int#${M_int%%[!0+-]*}}
1.23
# 

Greetings!

[amigo]

Hey, that's great. That works for posh also. I had edited above asking about reading a single leading/trailing digit, can you guys come up with aything?
I just checked and that works here too, under posh/dash (ubuntu /bin/sh):

Code: Select all

M_int="000000000020001.23"; echo ${M_int#${M_int%%[!0+-]*}}
and:
M_frac=100000000000000000 ; echo  M_frac=${M_frac%${M_frac##*[!0]}

[JakeSFR]

I had edited above asking about reading a single leading/trailing digit, can you guys come up with aything?

Hmm, this maybe?

Code: Select all

# a="firstandlast"; echo ${a%${a#?}*};  echo ${a#*${a%?}}
f
t
#

Greetings!

[amigo]

Yeah, that's the same pattern I'm using -for chunking also.
I can't confirm speed advantage yet for the very neat solutions above, but even if they are slower I'll use them to tidy a few places. And I'll of course do more testing on that to better check speeds.
MM, are you normally using bash, or other?

[MochiMoppel]

Hmm, this maybe?

Code: Select all

# a="firstandlast"; echo ${a%${a#?}*}
f

Or maybe this:

Code: Select all

# a="firstandlast"; echo ${a::1}
f

Trailing characters are a different story.
${a:: -1} works in bash but is not POSIX compatible.
This should be compatible but is a bit ugly: ${a:$((${#a}-1))}. The solution posted by JakeSFR is nicer.

[amigo]

This works only with bash, not even zsh works.

Code: Select all

a="firstandlast"; echo ${a::1}

The only other way I found to read trailing/leading was this really ugly bit:

Code: Select all

case $res in  0*) r=0 ;; 1*) r=1 ;; 2*) r=2 ;; 3*) r=3 ;; 4*) r=4 ;; 
  5*) r=5 ;; 6*) r=6 ;; 7*) r=7 ;; 8*) r=8 ;; 9*) r=9 ;; esac ; res=${res#?*}

I used to use that everywhere -sometimes it seemed that it was faster than the double-masking with very long numbers. But I hated it dearly, and finally got rid of it everywhere -I couldn't stand to see it anymore.
MM, about simplification for div, when I first saw that need/possibility is when I started looking/coding to see about using e-notation/floating-point everywhere. Turned out slower everywhere. So, I left div alone -and forgot to speed up problems like this:

Code: Select all

div -s8 123.678 / 0.00000000001 

which is fixed now.

I'm gonna try to push today -everything looks a little different, so you guys need to have these changes. Thanks again to you both.

[MochiMoppel]

This works only with bash, not even zsh works.

Works with busybox ash, otherwise I wouldn't have posted.

[amigo]

Great. I just pushed changes so you both can get up to date. from iq, everything below div has moved to iq+ without changes. All the ai functions are moved out elsewhere. I added an integer^integer pow (bnpow) function like '**' and used the bracket pattern suggested you guys came up with.
It all looks pretty different -a bit more eye-friendly, I hope.

All the pow stuff is unchanged and so, working to whatever degree it was. Still have to work on log and pow. But first I wanna make sure we've squeezed and tweaked and twisted and pushed the +-*/% routines for everything.
See comments at bnpow. As far as I can tell, the only errors I'm letting through are alpha-chars in inputs. The shell comparison/math operators handle that fine. I'm checking null inputs, multi --/++, mixed signs +- or -+, along with the individual function checks.

Does busybox ash support functions and $() syntax?
Edit: quickly updated github as I had left the ksh shebang in after testing.

[amigo]

All the error and input-handling in each function, seems to have leveled the playing field between the various shells, so that response times (for pow/nthrt) are very similar for all shells. At some point, I might create and function for interactive or one-shot usage, that would isolate all the error-handling into a separate function. Then, all the slow-downs could be removed again from the individual functions. But, that is still down the road a bit.

While waiting for any bug/error reports to come in, I will be working on pow and log to cover all special input/answer cases. The various pow functions make up a huge block of code which needs to be dealt with somehow. I also want to bring in more/better trig functions. What finally gets included in iq+ is still debatable -perhaps grouping related functions into loadable modules would work. Any thoughts?

MM, if you are serious about getting iq working under bb-ash, and it supports functions and '$()', then the place to start would be with cmp3w, then add. If those can be made to work, then the rest should/might also work. But, special code for individual shells is not in the works. I tried that very early on with blocks that supported bash, but that is not workable or efficient. The place where you'd think bash would benefit, like using ${var:?:?} and ${var: -1}
actually give no benefit. I tested, under bash, those forms and the alternatives we are using. Bash was no faster when using those very convenient forms. BTW, some of the current slowdowns may be from using the '${rint%%[!0+-]*}' constructions. You showed better times using it, but I found a slight slowdown here, when using it. I do like the 'look' of it better, for sure. Back to work now...

[MochiMoppel]

MM, if you are serious about getting iq working under bb-ash, and it supports functions and '$()', then the place to start would be with cmp3w

I'm never serious about anything, and I'm still not done with div()
I never used bb ash for serious scripts because I find the syntax cumbersome. This here is a good chance to play more with it. Generally faster than bash but sometimes extremely slow. What I still don't understand: You keep a whole zoo of more or less exotic shells but it doesn't include busybox ash. What Puppy are you using? I can only test with what the Puppy devs put on my plate, and so far it was always the same menu: bash and busybox.

The place where you'd think bash would benefit, like using ${var:?:?} and ${var: -1}
actually give no benefit

I wouldn't expect it to be faster, just neater

BTW, some of the current slowdowns may be from using the '${rint%%[!0+-]*}' constructions. You showed better times using it, but I found a slight slowdown here, when using it. I do like the 'look' of it better, for sure.

Don't know what you mean by "current slowdowns". These constructs (I think universally called "Parameter Expansion") are not necessarily fast. I can imagine that they are in fact slower when used for decimals with only 2~3 zeros to be removed, but with increasing number of zeros your loop becomes very slow while my construct remains stable. Want to see a really slow construct? All ${parameter//pattern/string} constructs are slow but VERY slow in ash:
Try

Code: Select all

M=-3.123000000000000000000000000000000000000000000000000000000000000
i=1 ;while [ "$i" -le 1000 ]; do
X=${M//0}
i=$((i + 1))
done; echo $X

In bash: 0m0.238s
In ash: 0m3.101s

[amigo]

Yeah, the '${M//0}' is the other thing I miss from bash, besides the '${var:0:1}'. But I soon realized it would seldom be helpful.
Check what happens if you change -3.1230000... to -301.10020030000
I applied the double-mask [^+-0] and [!0] everywhere and got response times doubled -slower, I mean. Really neat looking, but I have to have all the speedups possible. Also, there are several places where I need to remove all the trainling/leading -except 1, or where I need to count the zeros as well as remove them.
I finally realized why you had seen the improvement in div -the lines you were replacing were not working at all. Fixed now, and the worse 'real' problems like div -s7 220.456 / 0.00000000000000000123.

[MochiMoppel]

Code: Select all

# iq div 0 / 2
/bin/iq: line 346: / 2 : syntax error: operand expected (error token is "/ 2 ")

Correctly returns 0.0 in previous versions

[amigo]

Very sorry, I was in a hurry and something got corrupted there. I've tracked it down and will push to github tomorrow, nice and slowly...

[MochiMoppel]

I've tracked it down and will push to github tomorrow, nice and slowly...

Nothing pushed so far. Is this project dead?

[amigo]

No, not dead -just feeling dead. I've been sick for 2 weeks and the wife had a surgery last week. I have made some progress the last 2 days. Thanks for asking, and I hope to push in a couple of days.

[MochiMoppel]

Sorry to hear that. I hope that you two get well soon. You'd better stay away from the computer until you are fully recovered.
iq add + + =

[amigo]

Doing better, thanks. I've done a significant re-write of div today which is shorter and cleaner and appears to be working correctly for all cases, but need to check more and polish.

[wiak]

Sitting too long at computers encourages us to push up the daisies quicker rather than push to git, so be careful (I say also to myself).

[amigo]

Best wishes to the daisies, but not at my expense just yet.
Guys, I just pushed the new versions of iq and iq+ to github. I also added files with the AI activation-functions and trigonometry functions.
The code was reorganized and help functionality is changed. Also re-named some functions in iq+. I'm optimistic that any glaring errors are finally gone, and thanks for the well-wishes.

[MochiMoppel]

I'm optimistic that any glaring errors are finally gone

Not sure what constitutes a glaring error. Something like this?

Code: Select all

# iq div blabla / blabla
1.0

 
 
Or maybe this?

Code: Select all

# iq div -2 / --3
-6.-66666

This one is interesting because it used to return 0.66666, which wasn't correct either. It was a kind of double fault: Should have been flagged as syntax error, at least, being a triple negative, should have resulted in a negative value.

 
In one of your previous versions this would have produced a syntax error. IMO it still should:

Code: Select all

# iq div ++ / 3            
0.0

 
The first case should produce -1 and was already reported by JakeSFR last month.
The second case is a novelty. Used to produce the same positive 1.0 but now tries to make up for the missed minus signs

Code: Select all

# iq  div 1 / -1  
1.0
# iq  div 1 / --1
--10.0

 
These days I'm more interested in 'mul' than in 'div', so how about this one:

Code: Select all

# iq mul 22 2-                
/bin/iq: line 331: 22 * 2- : syntax error: operand expected (error token is "- ")

# iq mul 222222222222222222 2- 
/bin/iq: line 155: [: +100000000000000002-: integer expression expected
/bin/iq: line 156: [: +100000000000000002-: integer expression expected
44444444444444444.4

# iq mul 2222222222222222222 2-               
444444444444444444.4

Error handling depends on length of factor?

 

I'm afraid there is more, but let's call it a day. I don't want to spoil your recuperation and your holidays with this stuff. Spend your time with your wife and not with your computer. Happy holidays !

[amigo]

I appreciate your engagement. All those errors come under a single class of input, some of which I did handle in iq-1.62. But not all those cases.
The thorough checks needed to cull out nonsense inputs, slow down every function immensely. I think such checks belong in a more user-oriented, interactive front-end, where one might also have an expression evaluator, which would handle stuff like this ( num + ( num / num ) ). In iq+ there is the 'isnum' function which covers some of those input errors, but creative users can easily find ways to fool nearly any amount of checks.

As it is, each public function is a compromise between assuming completely sane, sensible inputs, and accepting sensible inputs which are mostly sane. The alternative was to put the real meat of each function into a sub-function and have a set of front-end functions -which still slows things down -and creates more inter-function dependencies.

A glaring error would be when one of the functions output looked like those inputs, or if you tried "iq div /user/bin/ls / 3 " (or worse), and something actually happened besides seeing: iq: 80: [: Illegal number: /user/bin/ls0. What I'm trying to achieve is real, correct answers to real math problems. For that I need functions that first work correctly and efficiently among themselves. Just that has been a pretty steep hill, as you can see.

Code: Select all

echo $(( blabla / blabla ))
bash: blabla / blabla : division by 0 (error token is "blabla ")

iq returns '1.0' because it checks early if 'mod' equals 'dvsr', other wise you'd see the above error directly from the math unit. Notice that it also doesn't correctly identify the problem.

Code: Select all

echo "blabla / blabla" |bc
Runtime error (func=(main), adr=5): Divide by zero

What I want is stuff like this to work:

Code: Select all

iq+ nroot -s7 1.1569821 10 
1.0146883
iq+ pow -s7 1.0146883 -4.6
0.9351252
#and especially the ai functions:
sigmoid_real -s4 0.935
0.718
sigmoid_tanh -s4 0.935
0.718

And it does work -I'm optimistic that other such valid inputs produce correct results -to the precision requested. I'm under no illusions that iq might become the next bc or awk. But if it could serve as a usable alternative to them, under some circumstances, then I will be happy. The whole goal was to be able to solve:

Code: Select all

sigmoid (x) = 1 / (1 + e^-x)

or similar equations, and their derivatives -with only simple shell language. The fact that it can do so in usable times was a surprise -once it happened. I'm hoping that div is finally working correctly -although I'm still working on the performance.

[MochiMoppel]

Merged iq+ into iq and tried help examples.

Code: Select all

# iq epow   -h              
    'epow' requires 2 inputs: 'epow [-s?,-S,-e?] base [^] integer-exponent' 
    Raises a decimal number to a (+)integer power, with answers in three forms:
    normally-scaled outputs, significant digits or scientific notation.
    Output format is controlled with the three scaling options -s?, -S? or -e?.
    Example usage for normal scaling: 'epow -s7 0.14 5' returns: 0.0000537
    Example for significant digits: 'epow -S7 0.14 5' returns: 0.0000537824
    Example for scientific notation: 'epow -e7 0.14 5' returns: 5.37824e-5
    'epow' is especially useful with e-notation for very long answers.

# iq epow -s7 0.14 5
0.0000005

# iq epow -S7 0.14 5 
0.000000537824

# iq epow -e7 0.14 5
5.37824e-7

Examples worked as advertised in previous version
 
 

Code: Select all

# iq pow -h               
   'pow' requires 2 inputs: 'pow [-s?] base [^] exponent'
    Example usage: 'pow -s5 3.14 ^ -4.12' for:(3.14^-4.12)
    Fractional exponents >3 digits (x^2.3754) are not recommended.

# iq pow -s5 3.14 ^ -4.12 
->div: 3 inputs required: num1 operator(/,%,r) num2
tsst: Invalid operator '2.8'
0.00896

Can't make sense of the error messages, Result seems to be OK.

[amigo]

Strange, I can't duplicate that error here:

Code: Select all

./iq+ pow -s5 3.14 ^ -4.12
0.00896

How are you getting access to pow through iq, since it is now in iq+? Are you sourcing iq+ into iq? Have you sourced iq/iq+ into your shell session?
If you call iq+ directly it should source in iq itself. I usually run these with ./iq??, so I'm checking placing both in PATH as you appear to be doing -and I find a problem in the execution block at
iq+ line 735:
iq+) src=1 . iq ;;
which needs to be:
* iq+) src=1 . iq ;;
I've fixed that here, to go with other changes.
But, that doesn't exactly explain the error you are seeing, either.

[MochiMoppel]

My bad. I pasted the iq+ functions from Github into my iq file because I prefer to have this stuff in one file but didn't copy the header stuff from the code. This left out the constants. With constants (ee/EE are required) everything works.

[amigo]

Yeah, thanks. I applied those fixes to iq+. I also found and fixed an obscure problem in div and applied a speedup, both in the routine for short numbers.
So, I've pushed those changes which were already done yesterday. I spent today on a new exp function, which delivers precision up to ~69 places in certain range(around 0.0002), 24 places around 0.2, 12 place around 1.2 and is 5-10 faster than old routine. Still working on range reduction to get at least 10-15 places within a reasonable range.
Having everything in one file is handy, but make ~1500 lines. Since iq+ is nearly all about pow, it made a nice place to partition the code. Skeptical users might balk at looking over so much code before running, but ~400 in iq alone is not so bad. Also, most of the ai functions don't need anything from iq+, so having pow&co. separated make things faster.
Besides a new exp function, I'm trying to implement a new log/ln. Together, they could replace much of the pow stuff, with faster, more accurate code.
Happy holidays, if you do that sort of thing. I'll be a bit occupied the next few days, but should have more progress by Monday or so.

[MochiMoppel]

I also found and fixed an obscure problem in div and applied a speedup, both in the routine for short numbers.

Talking about speedup: I found an obscure speed problem in mul. Not that I'm a genius, but I could tell you instantly the result when a 1000 digit precision pi is multiplied with, lets say 0 . With iq and my lame Atom processor this job takes longer than my lunch break: 13 minutes
The funny thing is that multiplication by any other number than 0 is faster.
Some rough speed results:

Code: Select all

1st factor     1    200     400   600   1000 digit number
2nd factor     2      2       2     2      2
      time 0.003   1sec    6sec 24sec   2min

1st factor     1    200     400   600   1000 digit number
2nd factor     0      0       0     0      0
      time 0.003   3sec   47sec  4min  13min

I stumbled on this phenomenon because I finally gave up on poking around in your iq. Makes no sense for me to tinker with code that I don't fully understand. Instead I thought that it would be much more fun to roll my own, with the same challenges: Arbitrary precision, only shell commands allowed and - the ultimate torture - POSIX compatible. I figured that multiplication would be less difficult than addition or division. I also remember from my school days how to multiply "by hand", taking one digit at a time, multiply with another digit, remember the carry-over value and add it to the next result. Sounds not too difficult but may be slow as a snail. I clearly have no ambition to compete with your work and I may not even have use for it, but I'm just curious how it goes. Let's see.

Happy holidays, if you do that sort of thing.

Not really, but thanks and the same to you.

[amigo]

Looks like I'm missing a check for x*0 or 0*x. Since you're running with bash, your times are always gonna be long. If you're going to have a go at writing your own, I probably won't hear much from you for a while... I actually started with division, but quickly realized that working with numbers over 19 digits was gonna need a lot of ingenuity. Multiplication seems easy, but the native shell math is gonna limit you to a total digit length of 18 digits -basically multipying two 9-digit numbers, which is not very big. Since everything is addition, you could implement everything with just addition and numeric comparison -that's where a something like 12876451876097020837 / 2 becomes a really big problem - ~6438225938048510418 subtractions, the naive way. That's where all my weird-looking code comes in.
Get yourself a binary of dash to use for development -it's always ~3x faster than bash, or ksh 8-10x faster. I hate to lose you as a tester -perhaps if I explained code sections for you, you'd see why some things are so slow and difficult to speed up without sacrificing accuracy.

[MochiMoppel]

Multiplication seems easy, but the native shell math is gonna limit you to a total digit length of 18 digits -basically multipying two 9-digit numbers, which is not very big.

Yes, that's also the range where shell arithmetic *should* be faster than bc. When testing I found that the actual threshold in iq seems to be 16 digits:
mul 123456789012345 2 is faster than bc. Adding only 1 digit slows down the process significantly and bc wins.

[misko_2083]

Maybe you can speed up with some Vedic math tricks.

[amigo]

welcome @misko_2083 I looked into using vedic techniques, and learned some stuff that has helped. I've looked into anything which might be faster, mental-techniques, CORDIC routines, anything with 'fast code' in it. My basic exponentiation routine was ported from ruby, other stuff from python, C -anything I can half-way read.
@MochiMoppel Interesting that iq is outperforming bc there, one-shot and head-to-head -is that with bash and busybox bc, or...?
All the shells in question have a 19-digit maximum for inputs and outputs. No single number can be longer, and the output can be max 19 digits. The chunking routine in cmp3w uses 18-digit chunks , add uses 17 and mul uses 9. The chunking in div is more accurately called partitioning, I think.
I'm off to fix mul, thanks for flushing out another oversight.