HotBasic Goes Head-to-Head With Its Competition:
Console Application Math Benchmarks
by Norm Kaiser
Abstract
In console application math benchmark tests, HotBasic compiler performance was found to excel significantly compared to VB .Net, Borland CPP, Dev CPP and REALBasic, considering run-times, application size and ram usage. HotBasic defied conventional wisdom regarding which computer language and compiler produces the most efficient executables.
Introduction
The funny thing about conventional wisdom is how often it's wrong. It's amazing how often people have strong convictions about things -- often to the point of zealotry -- that in the face of facts and science are simply flat wrong. Conventional wisdom holds, for example, that Windows is riddled with security holes while Linux is the paragon of security. However, a 2005 year-end vulnerability summary report by US-CERT (United States Computer Emergency Readiness Team) concluded that Linux/Unix accounted for a jaw-dropping 2,328 vulnerabilities while Windows had just 812!
Ready for an even bigger jaw-dropper? Hold on to your hat! For at least the past twenty years, conventional wisdom has held that if you need to write applications that are small, efficient, and fast, you had to write them in C, C++, or assembly. Sure there are other languages that are easier, but conventional wisdom held that using an easier language came with a very steep tradeoff -- poor performance. As a consequence, C/C++ remained ensconced as the language of choice for demanding professional applications. Well guess what -- conventional wisdom has just been proven wrong again.
You may not have heard the news, but there's a new compiler on the scene that boasts very impressive performance numbers, but the really surprising thing is that this new compiler is based on a programming language dismissed by
conventional wisdom long ago -- BASIC. The compiler is called HotBasic (www.hotbasic.org), and it's set to take the software development world by storm. And believe it or not, it's a serious contender for wresting the performance crown from C/C++.
HotBasic is an all-new compiler built from scratch from the ground up. It was written and is still being developed and supported by an accomplished scientist named Dr. James J. Keene. In 2003, Dr. Keene needed a compiler to write some software for an experiment he was working on, but to his surprise he couldn't find one that met his standards. So he decided to write the compiler he needed himself. Applying sound engineering principles and completely rejecting the "it's good enough" mentality, Keene soon discovered that he had almost by accident written what he saw as the best compiler in the world for generating the fastest, smallest, and most efficient executables. Intrigued by his claims, I decided to put HotBasic to the test.
I've written software for almost twenty-five years. I started with BASIC, then moved up (or down, however you look at it) to assembly. Years later I started writing code in C and C++. My most recent professional projects were written in Visual Basic and Visual C++ .Net. Up until I discovered HotBasic, I was most partial to VB .Net. After discovering HotBasic, I began to wonder if continuing to do my professional development in VB .Net was the right choice. So I decided to put together a simple benchmark.
Methods
I wrote a very simple program that would test the fundamental functionality of all the biggest name compilers on the market. When writing the test program, I asked myself, what do computers do most? The answer, to me, was pretty clear -- number crunching and storing and fetching data. So the program I wrote was simple. It would perform eleven common arithmetic functions -- division, multiplication, trigonometry (COS, ACOS, TAN, SIN, etc.), exponents, logarithms, and square roots and store the results in a DOUBLE variable. The eleven functions would serve as the body of a main loop, and the program would execute the main loop fifty million times. A very simple test, but very logical.
Then I picked the compilers to test HotBasic against. I chose Visual Basic .Net 2005, REALBasic 2006, Bloodshed Dev-C++ 4.9.9.2, and Borland C++ 5.5. I then wrote equivalent versions of the benchmark program for each of the languages of the subject compilers. (The source code for each executable appears in the appendix at the end of this article.) Ultimately I intended to run a full gamut of benchmarks to test and compare a wide assortment of common compiler functionality -- string manipulation functions, file access, visual elements, etc.
For this first test I decided to just keep it simple. Each compiler would generate a console application that simply did number crunching and storing and fetching of results in DOUBLE variables. So now I had five equivalent executables.
I decided to compare each executable in terms of size, the amount of memory the executable consumed while running, and how long it took the executable to execute. Simple, easy, and fair. I then decided to run each executable a total of ten times. I ran the tests on my Compaq Presario notebook, which has a 1.6 GHz AMD Athlon processor and 512MB of RAM. To be sure other processes running on my laptop (antivirus, anti-spam, system utilities, etc.) didn't accidentally skew any of the test results, I ran the tests in a random round-robin.
To get an accurate measurement of how long each executable ran, I wrote a simple timer program. The timer worked by simply calling the subject executable through a shell command. When the executable started running through the shell command, the timer started counting. When the executable terminated, the timer stopped counting and displayed the time.
Results and Discussion
The results really surprised me. Take a look for yourself:
Let's take a look at the data. Remember, I ran each test a total of ten times for each compiler and the execution time shown is an average of those ten runs. So it's not like one compiler's executable just happened to have a bad showing. Let's start by comparing HotBasic to my old stalwart, Visual Basic .Net 2005.
Visual Basic .Net 2005
In terms of execution time, Visual Basic .Net 2005 and HotBasic ran neck-and-neck. That alone surprised me. Think about it. VB .Net is one of the flagship products of the biggest software company on the planet, while HotBasic is the brainchild of one man. Granted, he's clearly a very smart man, but one man all the same. What's really noteworthy here is, not only did HotBasic hold its own with VB .Net, it won! Sure, half a second doesn't seem like all that much, but in demanding graphical and scientific environments where all those fractions of a second accumulate, it can be enormous.
The most compelling facet of the VB .Net vs HotBasic matchup is file size and memory required. The HotBasic executable was less than a fifth the size of the VB executable, and more importantly, it required less than a fourth
the amount of RAM to execute! Compared to HotBasic, VB .Net is a veritable memory hog. In summary, the HotBasic executable was faster, smaller, and consumed less system resources -- clearly making it the winner.
Borland C++ 5.5 and Bloodshed Dev-C++ 4.9.9.2
The rest of the field was even more eyebrow-raising. I deliberately included two popular C++ compilers in the test: Borland C++ 5.5 and Bloodshed Dev-C++ 4.9.9.2, which uses MinGW32 as its actual compiler. I really wanted to see how HotBasic -- a compiler using a very high level language with a very simple syntax -- would compare to C++. Well, the numbers speak for themselves. Borland C++ 5.5 was competitive, but its executable took over ten seconds longer to execute than the HotBasic executable! That's right, ten seconds. Furthermore, HotBasic beat the Borland C++ compiler in the other two categories as well.
Enter the Bloodshed Dev-C++ 4.9.9.2 executable, and in terms of execution time, HotBasic just clobbered it. The Dev-C++ executable took well over twice as long to execute. Its file size was almost a hundred times larger. It did, however, win one category: it needed a bit less memory to execute, 836KB vs 1,044KB.
REALBasic 2006
The final entrant was REALBasic 2006, which appeals to a very similar developer community as HotBasic. The results show, however, that REALBasic was no match for HotBasic. Like Dev-C++, the REALBasic executable took over twice as
long to execute as the HotBasic executable. Its file size was enormous, and the amount of memory it required to run was second only to VB .Net. In fairness, I do want to point out that I used the trial version of the REALBasic compiler for the benchmark, not the registered version. The REALBasic compiler inserts a time limit to each
executable it generates to prevent people from building working applications without purchasing the product. The time limit built in to each executable the compiler generates clearly adds to the executable's bloat. I would love to re-run the benchmark with a full version of the REALBasic compiler (are the folks at REAL Software listening?), but paying its exorbitant price just for a benchmark is impractical. It's worth noting, however, the trial version of the compiler's performance is likely not very far from that of the registered version.
Conclusions
The benchmark test results are compelling. In terms of pure number crunching and storing and fetching values in variables, HotBasic clearly excels. There are more benchmarks to come, of course, but these initial tests suggest that this new kid on the block is a very serious contender that software development professionals can't afford to ignore.
Clearly, HotBasic has become much more than just a pet project of a researcher looking to fill a need. Just as Linus Torvalds single-handedly turned the world of operating systems upside down, Dr. James Keene looks like he is about to do the very same thing to the world of software development.
Appendix
Source code for each of the test applications used in the benchmark appear below.
Contact Me
Want more information about the benchmark? Have comments? Want to run the benchmark for yourself? Would you like to receive copies of the test executables? Just send me an e-mail at falkyr11 at yahoo dot com.
HotBasic
$APPTYPE CONSOLE
DIM X AS DOUBLE
DIM Y AS DOUBLE
DIM I AS INTEGER
Y = .5
For I = 1 To 50000000
Y = Y / 1.1234
Y = Y * 1.1234
X = acos(Y)
X = asin(Y)
X = atan(Y)
X = cos(Y)
X = exp(Y)
X = ln(Y) 'natural log
X = sin(Y)
X = sqr(Y)
X = tan(Y)
NEXT I
PRINT Y
C++
#include {cstdlib}
#include {iostream}
#include {cmath}
using namespace std;
int main()
{
double X, Y;
int i;
Y = .5;
for(i = 1; i <= 50000000; i ++)
{
Y = Y / 1.1234;
Y = Y * 1.1234;
X = acos(Y);
X = asin(Y);
X = atan(Y);
X = cos(Y);
X = exp(Y);
X = log(Y);
X = sin(Y);
X = sqrt(Y);
X = tan(Y);
}
cout<<Y;
}
REALBasic
DIM X AS Double
DIM Y As Double
DIM I As Integer
Y = .5
For I = 1 To 50000000
Y = Y / 1.1234
Y = Y * 1.1234
X=ACOS(Y)
X=ASIN(Y)
X=ATAN(Y)
X=COS(Y)
X=EXP(Y)
X=LOG(Y)
X=SIN(Y)
X=SQRT(Y)
X=TAN(Y)
Next
Print STR(Y)
Visual Basic .Net
Dim X As Double
Dim Y As Double
Dim I As Integer
Y = 0.5
For I = 1 To 50000000
Y = Y / 1.1234
Y = Y * 1.1234
X = Math.Acos(Y)
X = Math.Asin(Y)
X = Math.Atan(Y)
X = Math.Cos(Y)
X = Math.Exp(Y)
X = Math.Log(Y)
X = Math.Sin(Y)
X = Math.Sqrt(Y)
X = Math.Tan(Y)
Next
Console.Writeline(Y)
Article text: Copyright © 2006 Norm Kaiser
Copyright © 2006 James J Keene PhD
HotBasic™ is a trademark of James J Keene
Original Publication: Dec 21, 2006
Back to HotBasic Home Page.
