I created a benchmark in C++ for testing and comparing TeaScript and its competitor ChaiScript by computing the Fibonacci number of 25 recursively. As a reference I did the same also in C++. The complete source code of the benchmark is available below.
The result is pretty amazing, since with the used pre-release 0.8.0 of TeaScript I did not profile and optimize yet. This is a task for a future release. The latest release of TeaScript is available in the download section.
(UPDATE 2023-01-17): I created a Github Repository for this and all future benchmarks comparing script languages in and for C++.
(UPDATE 2024-03-08): I created a new benchmark for filling a 32 bit Full HD (or UHD) image buffer pixel by pixel. Read the results here: New Benchmark.
Here is the benchmark result: Computing time of fibonacci( 25 ) measured in seconds. Shorter is better/faster, longer is slower.
Some technical information about the environment:
OS: Windows 10 21H2 (64bit)
CPU: Intel Core i7-11700
RAM: 32 GB
Compiler/Build: Visual Studio 2022 Community (17.4.3), Release/x64 (AVX2), C++20 (settings like in my blog post).
Some notes to Python and Lua:
Python as well as Lua are actually not in competition with TeaScript from the performance point of view since TeaScript and ChaiScript are real script languages (and header only C++ Libraries), but Python and Lua are compiling the source to a bytecode and executing it in a virtual stack machine (Python) or in an even a more complex virtual machine (Lua).
This may change with a later version of TeaScript.
Here is the C++ source code of the benchmark:
/*
* SPDX-FileCopyrightText: Copyright (C) 2023 Florian Thake, <contact |the-at-symbol| tea-age.solutions>.
* SPDX-License-Identifier: MIT
*/
#include <cstdlib> // EXIT_SUCCESS
#include <cstdio>
#include <iostream>
#include <chrono>
#include <teascript/Parser.hpp>
#include <teascript/CoreLibrary.hpp>
#include <chaiscript/chaiscript.hpp>
// recursive fibonacci function in TeaScript
constexpr char tea_code[] = R"_SCRIPT_(
func fib( x ) {
if( x == 1 or x == 0 ) {
x
} else {
fib( x - 1 ) + fib( x - 2 )
}
}
fib(25)
)_SCRIPT_";
// recursive fibonacci function in ChaiScript
constexpr char chai_code[] = R"_SCRIPT_(
def fib( x )
{
if( x == 0 || x == 1 ) {
return x;
} else {
return fib( x - 1 ) + fib( x - 2 );
}
}
fib(25);
)_SCRIPT_";
auto Now()
{
return std::chrono::steady_clock::now();
}
double CalcTimeInSecs( auto s, auto e )
{
std::chrono::duration<double> const timesecs = e - s;
return timesecs.count();
}
double exec_tea()
{
teascript::Context c;
teascript::CoreLibrary().Bootstrap( c );
teascript::Parser p;
auto ast = p.Parse( tea_code );
try {
auto start = Now();
auto teares = ast->Eval( c );
auto end = Now();
std::cout << "value: " << teares.GetAsLongLong() << std::endl;
return CalcTimeInSecs( start, end );
} catch( teascript::exception::runtime_error const &ex ) {
teascript::util::pretty_print( ex );
} catch( std::exception const &ex ) {
puts( ex.what() );
}
return -1.0;
}
double exec_chai()
{
chaiscript::ChaiScript chai;
auto ast = chai.parse( chai_code );
try {
auto start = Now();
auto chres = chai.eval( *ast );
auto end = Now();
std::cout << "value: " << chaiscript::boxed_cast<int>(chres) << std::endl;
return CalcTimeInSecs( start, end );
} catch( chaiscript::Boxed_Value const &bv ) {
puts( chaiscript::boxed_cast<chaiscript::exception::eval_error const &>(bv).what() );
} catch( std::exception const &ex ) {
puts( ex.what() );
}
return -1.0;
}
// recursive fibonacci function in C++
long long fib( long long x )
{
if( x == 0 || x == 1 ) {
return x;
} else {
return fib( x - 1 ) + fib( x - 2 );
}
}
double exec_cpp()
{
try {
auto start = Now();
auto res = fib( 25 );
auto end = Now();
std::cout << "value: " << res << std::endl;
return CalcTimeInSecs( start, end );
} catch( std::exception const &ex ) {
puts( ex.what() );
}
return -1.0;
}
int main()
{
std::cout << std::fixed;
std::cout << std::setprecision( 8 );
std::cout << "Benchmarking TeaScript and ChaiScript in calculating Fibonacci of 25...\n";
std::cout << "... and C++ as a reference ... \n";
std::cout << "\nStart Test C++" << std::endl;
for( int i = 3; i != 0; --i ) {
auto secs = exec_cpp();
std::cout << "Calculation took: " << secs << " seconds." << std::endl;
}
std::cout << "\nStart Test TeaScript" << std::endl;
for( int i = 3; i != 0; --i ) {
auto secs = exec_tea();
std::cout << "Calculation took: " << secs << " seconds." << std::endl;
}
std::cout << "\nStart Test ChaiScript" << std::endl;
for( int i = 3; i != 0; --i ) {
auto secs = exec_chai();
std::cout << "Calculation took: " << secs << " seconds." << std::endl;
}
puts( "\n\nTest end." );
return EXIT_SUCCESS;
}
… and this is the screenshot from the run:
BTW: The download package of TeaScript contains a script (beside others) for calculating Fibonacci Numbers inclusive time measurement.
15 replies on “TeaScript VS ChaiScript Fibonacci Benchmark”
Good day,
I have had some time lately embedding script engines in c++:
Angel, Chai, Lua, Wren and Zen. I encountered your benchmark by chance just today. It caught my attention. Of the 5 engines I mentioned, I am leaning towards Chai. Essentially because it drives the c++ template and type-safety system to a level way past my head.
In this respect, Tea resembles Lua (and others) in as far at it defines 1 function signature for all script to host calls. Parameters are exchanged through some local stack. My main interest is how these engines bind to the host and visa-versa.
So I tried your Chai fibonacci example and to my surprise it took forever! Over 33 secs. The debugger shows that it is going round and round throwing exceptions (caught internally).
I.o.w. fibonacci made it reveal some serious issues.
I tried this in Lua:
Classic implementation: 0.0174811 secs
With tail recursion: 0.0059961 secs
(Latter code is by Michael-Keith Bernard, 2012)
Other than Lua, I am afraid that many of these engines are not any longer supported. But they are all interesting to study.
Cheers,
Conrad Weyns
Hi,
thank you very much for your comment and providing your investigations, results and thoughts.
Its highly welcome and a great value.
In reply to your comment, my personal findings are these:
ChaiScript has its strengths in the great and awesome function overload resolution (TeaScript and many other languages don’t support function overloading at all) and the possibility to use C++ classes and its member functions from inside the script.
But the downsides are the costs for the function calls (and maybe some other things as well).
If you compare the current TeaScript implementation (or ChaiScript) with e.g., Lua or Python, you are comparing 2 different kind of script architectures.
TeaScript and ChaiScript are both “simple” recursive AST Walkers, while Python as well as Lua have an integrated virtual machine and they translating the code into instructions for these machines.
With that Python and Lua reaching a way higher execution speed.
TeaScript will get its own (self implemented) virtual machine with the 0.14 release.
Along with the 0.14 release I will publish some new benchmarks (including a new run with the Fibonacci test).
You will find it here on my page.
BTW, the source code of the Fibonacci Benchmark as well as for the new ones are/will be available on Github:
https://github.com/Florian-Thake/BenchmarkScripts-TeaScript-VS-ChaiScript-VS-Jinx
If you are interested in some more information about embedding script engines in C++ you will find some on my website.
I recommend to read my comparison article:
Script Language comparison for embed in C++
Also, the latest release article of TeaScrcipt is a good starting point for discover some nice features and background information for TeaScript:
Release of TeaScript 0.12.0 🌈
If you give TeaScript a try, I will be very happy about any feedback, suggestions and also about bugs or problems.
Cheers,
Florian
Hi Florian,
thanks for your response.
Suffice it to say that as far as the classic Fibonacci algorithm is concerned, Chai is not a contender for benchmarks. About 35 x slower than Tea and others – something must be seriously wrong!
I have since discovered that by merely adding a few more lines to the code here and there my timings were drastically reduced from over 30 sec down to 2 seconds. This is bad news. But I think that Chai has been abandoned years ago, so what can we expect?
As for the others that I am currently testing:
Angel: 0.0575237
Zet: 0.0596717
Wren: 0.0341915
Increasing the input step-wise from 25 and up is giving consistent results in all, including Lua. As expected, most become pretty useless after passing about 35. Except for the Lua tail recursion algorithm which seems to just go on and not care. The int result overflows and starts showing negative results 🙂
This one at https://www.omnicalculator.com/math/fibonacci is good up to 250 and returns the result in real time as you type in the number!
Sort of puts a few things in perspective 🙂
Have a nice day!
Conrad
Hi Conrad,
thank you for your replay and further content and investigation.
Yes, I think ChaiScript has a serious performance problem – at least for nested function calls. But as long as the call depth is not so deep it should be still usable in a good way.
But since the last years there wasn’t any further (feature) development and also – from my personal view – there are missing some new and advanced language features as they were introduced with Rust, Zig and Go and others.
That was one of the main reasons why I started TeaScript 🙂
Lua is incredible fast. My goal for TeaScript is not to reach the performance of Lua, but come very close or even be better than the others.
Regarding the omnicalculator: I think, they are “cheating”. They just have a lookup table for the first 250 numbers. Then no calculation takes place at all.
Have a nice day as well!
Florian
Hi again,
I now also have Tea and Jinx embedded.
This lives in a Win GUI app.
Results fib( 25 ) iterative:
lua: 0.002683
angel: 0.003569
jinx: 0.003581
zet: 0.004299
wren: 0.004578
chai: 0.008041
tea: 0.010924
Results fib( 155 (*a*) ) iterative:
lua: 0.004978
angel: 0.005772
wren: 0.007221
zet: 0.007547
jinx: 0.012120
chai: 0.026899
tea: 0.034383
Results fib( 25 ) classic – recursive
lua: 0.014360
angel: 0.020397
zet: 0.024893
wren: 0.027548
jinx: 0.402053
tea: 0.787331
chai: 18.585981
Result of fib( 35 ) classic – recursive
wren: 0.787331
lua: 1.535814
zet: 2.533135
angel: 1.912057
tea: 110.719227
chai: 130.154235 (27 only – terminated the app at 28!)
jinx: KABOOM! (*b*) (limit is 34, after which it’s context stack from calling wait is full, over 45Meg)
Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz 48.0 GB
(*a) an int overflows quickly. Most of these can simply use vars initialized with a float number.
Except for Angel that requires real types so int needed to change to double.
(*b) Jinx has made a design decision that I find of great value. It enforces your script to use “wait”.
If you don’t, the recursive fib will terminate almost immediately with a message: “over 10 calls ….”.
The core invocation of the engine looks like this in my c++ code:
auto pBytecode = rt->Compile( pUtf8Txt.get() );
if ( pBytecode )
{
// Create a script with the compiled bytecode
auto pScript = rt->CreateScript( pBytecode, userContext );
// Execute script until finished
bool result = true;
size_t cnt = 0;
do
{
result = pScript->Execute();
if ( !pScript->IsFinished() )
++cnt;
} while ( result && !pScript->IsFinished() );
…….
In the recursive fib( 25 ) ‘wait’ was called: 196392 times!
This allows me to let the main application stay in control and employ techniques to remain user-input-sensitive (a.g a Cancel button clicked).
The separation of Compile and Execute is also important.
Currently the invocation of my Tea script looks like this:
eConfig conf( LevelFull );
Engine eng( conf );
auto valObject = eng.ExecuteCode( script );
Cheers,
/conrad.
nb: I am well in my retirement now and ever closer to expiration date!
I have been wanting to play with this for years. I imagine it is good for the brain…
The Jinx language took me back to the 90′ with Apple script and Hyper script!!
Hi Conrad,
thank you very much for providing further results and input!
What kind of GUI are you using for run the benchmarks? Is it Qt based?
To *b*:
Yeah, I run into the same as I tried Jinx the very first time. 😀
You can see all of the code here:
https://github.com/Florian-Thake/BenchmarkScripts-TeaScript-VS-ChaiScript-VS-Jinx
The technique to return from execution every some time is nice.
TeaScript will be able to do so as well when the integrated Virtual Stack Machine is implemented and ready to use (planned with 0.14 release, luckily before Eastern or some time later).
Also, I am planning to provide a kind of “single step debugging” technique which is in my opinion really important for be able to investigate bigger scripts.
Separated compile and execute is available in TeaScript already via the low level interface.
You can use the Parser for parse the code first (
parser.Parse( content );).Then, in the second step you must evaluate the Abstract Syntax Tree (
ast_root->Eval( context );)But also, you need a valid context (where all the functions and variables are saved), ideally with a bootstrapped CoreLibrary:
Context c; CoreLibrary::Bootstrap( c, config );With the next next TeaScript version (0.14) there will be 3 separate steps available:
It will be also possible to save a ‘compiled’ script and then next time load and execute that instead for saving the parsing and compile.
PS: Great that you found a nice and interesting hobby! 🙂
I don’t know Apple Script, but Jinx on its own is a very interesting and well designed language. Just, it is only procedural and it lacks a supporting core library with some functionality.
Correction to my previous post!
fib( 25 ) Iterative:
tea: 0.004342
————- fib_iterative.tea —————
func fib( n ) {
def a := 0
def b := 1
def cnt := 1
repeat {
def tmp := a + b
a := b
b := tmp
cnt := cnt + 1
if ( cnt > n ) {
stop
}
}
a
}
def num := 1
def result := 0
repeat {
result := fib( num )
print( “%(result), ” )
num := num + 1
if ( num > 25 ) {
stop
}
}
result
…………………………………
I have patch CoreLibrary::PrintStdOut to capture the output as I don’t want any std::out etc..
Also, the actual console outputting is filtered out of the timings. The timer is part of my script invocation system and is paused upon entry in the c++ code.
Mostly I observe consistent result from run to run but once in a while it does surprise me either way.
Essentially, seen from the main application thread, all of these engines are blocking.
/conrad
Great! 🙂
Do you know, many CoreLibrary functions in TeaScript are mutable (All which don’t start with an underscore).
So, you could just overwrite the print and/or the println function with yours.
You must register it in the engine first (RegisterUserCallback), give it a name like “capture_print” and then at the very beginning of the TeaScript code you can just do a
print := capture_printMay I ask which TeaScript version are you using actually?
In the lastet 0.12 the forall loop was added.
If you like, you could try to replace the repeat loop with the forall loop.
It could then look similar this
forall( num in _seq(1, 25, 1) ) { /* more code here... */ }I did not check yet what the effect will be for the benchmark results.
Probably I will add a variant to the Benchmark Github (see my previous reply).
Cheers! 🙂
/Florian
PS: I just added a forall loop variant for the iterative Fibonacci Benchmark to the Benchmark repo:
https://github.com/Florian-Thake/BenchmarkScripts-TeaScript-VS-ChaiScript-VS-Jinx
It seems like with the forall loop TeaScript is on par with ChaiScript now.
Also, if you use Notepad++, I added a syntax highlighting configuration file in the TeaScript Library Github:
https://github.com/Florian-Thake/TeaScript-Cpp-Library
It could be helpful for programming.
Cheers! 🙂
Hi Florian,
– teascript::version::as_str() is 0.12.0
– I had discovered the new forall in Tea. Much nicer syntax and easier to use. Nice!
Slightly faster also:
Fib( 25 ) iter. repeat: 0.005345
Fib( 25 ) iter. forall: 0.004191
forall( num in _seq(1, n, 1) ) {}
I am not a big fun of the underscore in _seq in a scripting language.
No big deal, just an opinion 🙂
– Gui app is 32 bit, my OS is Windows 11 x64.
– Framework is long time discontinued, so called “cross platform”, Starview.
It has been extensively doctored by a small Windows team, me in particular, for over 25 years.
– App dates back to before Win 95 and Visual Studio 6, Metrowerks CodeWarrior on the Mac.
Mac version was stopped ca 20 years ago, memory serving..
Kudos to MS for continuing to support the Win32 API! I started on a Mac Plus in 1985, Motorola 68000 cpu.
Very different story at Apple.
But if you dig into the huge Open Office code base, you’ll recognize StarView code at its core!
To be honest, I find it incredible that more or less same c++ code from back then can still run on today’s hardware with Windows 11!
– The Win app passed its end of live about the same time as my retirement 1.5 years ago.
I just keep a reduced version going for fun. I enjoy making use of the latest c++.
Nowadays, it is Web, html, JS, C# and what not. In the sky off course.
It is all about making users pay a little every months for the rest of their lives 🙂
This is not sarcasm, just realism!
– I will have another look at capturing stdout and sterr.
On the subject, I reckon every single callback from the scripting universe should include the Context param. Also print.
This is where the c++ side can make use of GetPassthroughData. Most script engines seem to have some way of passing an opaque void* around.
Important this else one would have to resort to using file-scope globals and they suck.
– I have since also added Pluto. It is basically Lua with more stuff, not as “lean and mean” so to speak.
Some nice extensions I think. It is fairly configurable so I am not sure I got the best performance out of.
It claims that its vm can be slightly faster than Lua in some circumstances. I don’t observe that.
– Another candidate is LuaJIT. Worth trying I suppose.
– I think that my current scripting tests are far to light-weight and naive for any worth while comparisons.
Next thing is to add several hundreds (very conservative estimate) of callbacks and see what happens.
– DaScript, recently renamed to daslang. I understand why. If you haven’t been there, it’s worth taking a look. Lots of very interesting reading.
I managed to compile and link it but not run it. It just hangs at start-up, before main().
It adds a huge footprint to the app. I suspect it is trying to run things while registering file-scope modules.
All of this happens before main() is called and long before the debugger is usable. So I gave up – for now.
Nothing easier than a good crash right into the ms debugger…
But it is an interesting project. Looks like it started as an embedded engine for gaming and that it has now taken over as a complete language of its own. Who needs c++ 🙂
If anything, this guy is into producing fast code!
– To end on a more philosophical note:
Giving your users the capability to extend the app through an API and some easy to use scripting language is of great worth.
It generates revenues and also new opportunities for 3rd parties.
I have experienced this first hand with our app, a CRM. We added a COM and a VB Script API.
One thing I have learned is this: Your API users couldn’t care less about why something is the way is behind the scenes.
They need to achieve some extra business logic for their customer(s) and they will invariably find a way to get there!
Once you have published an API, you are stuck with it.
Cheers,
Conrad
Hi Conrad,
thank you very much for your detailed reply. Wow, that is a lot of background information!
I never heard about Starview. But, yes, the backward compatibility of Windows is very great. Compile once, use forever. (not always, but it is possible.)
Thank you very much for the positive feedback regarding the forall loop! 🙂
Yes, the underscores can be a little annoying. I am aware of this.
They are there for 2 reasons:
So, all the underscore functions are considered the very core part of TeaScript which cannot (and must not) be changed/modified from within the script.
The big drawbacks are the ugliness and that you must remember which function is with underscore and which is without.
I am planning to add a module support later. I did not decide yet in detail how I will proceed with the CoreLibrary then.
On the one hand I want to stay backward compatible, on the other hand I don’t want to fill the global namespace with all of these names.
Might be that everything will be moved into tea:: or std:: or core:: and then without underscore (Also, I am not sure about the syntax yet.)
I don’t know Pluto yet. Thank you for mention it, I will have a look. 🙂
Jit compile is out of my scope for TeaScript. It’s incredible for optimal performance, but for me it loose the character of a script language then.
To your closing parts:
Yes, an API for customization is very great, especially if scripting is also possible.
I divided TeaScript in low level and high level API – and only the latter shall stay stable in the long term once it become mature.
There is the one nice xkcd comic where a user complains about a bugfix against CPU overheating has destroyed their workflow:
https://xkcd.com/1172/
So, yes, its true, once you release sth. you have to stick/deal with it forever. 😀
Cheers,
Florian
print :=hostPrint
println := hostPrintln
This works, nice!
By using a lambda when registering I get to exploit the capture and drag along my host context. It’s the [] in lambdas that makes them so useful 🙂
f.y.i. your Notepad++ highlighting looks good but not in dark mode.
Conrad.
Yes, that is the way to bring your own arbitrary context with you! 🙂
And thanks for the hint for the dark mode. Since there are thousands of styles in notepad++ it will be impossible to make one syntaxhighlighting fitting for all.
But I will have a look. I thought that the colors are also goo viewable on a dark background, aren’t they?
Hi Florian,
re: “wait”, “suspend” and the like.. since you may get to support this in Tea.
I discovered that Angel script has a better solution. The host app can register a Line Callback where it can itself Suspend().
This means the host can remain in control regardless of what the script is doing and the script developer need not call “wait” all over the place. Much better!
Cheers,
/conrad
Hi Conrad,
thank you very much, I will have a look into it! 🙂
Cheers,
Florian
PS: Hopefully until end of next week (at latest) the 0.13 release will be available…