What does HackerNews think of benchmarks?

Some benchmarks of different languages

Language: Makefile

In all the language comparisons I've found over the years, Python consistently comes out slightly slower, for example:

https://github.com/kostya/benchmarks

Bearing in mind these are probably not even using YJIT, which makes Ruby considerably faster in some scenarios.

According to these benchmarks, pure Python is about 100 times slower than C and other fast languages: https://github.com/kostya/benchmarks
I think the basic idea is to cross energy use of a language with ubiquity of existing deployments.

So take energy use benchmarks from things like this

https://haslab.github.io/SAFER/scp21.pdf https://github.com/kostya/benchmarks

and cross it with popularity?

https://spectrum.ieee.org/top-programming-languages/

> They seem to invest a lot in JavaScript which is not what I’d call a performant language

JavaScript (V8 at least) is extremely performant, near native code performance. Considering how dynamic it is, it's not an easy feat but Google, Apple and Mozilla work a lot on it.

Here's a benchmark where JS is 50x faster than Python: https://github.com/kostya/benchmarks

Note that PyPy does much better.

If anyone is interested to see how Nim fares against some other programming languages, here are some benchmarks: https://github.com/kostya/benchmarks
Are the performance claims of Julia greatly exaggerated?

Julia loses almost consistently to Go, Crystal, Nim, Rust, Kotlin, Python (PyPy, Numpy): https://github.com/kostya/benchmarks

Is this because of bad typing or they didn't use Julia properly in idiomatic manner?

You can find some benchmarks here: https://github.com/kostya/benchmarks

With the caveat of course being that benchmarks don't always reflect real world performance.

Crystal is nearly at 1.0 and seems way faster than Ruby in addition to having a Ruby like syntax, i'd suggest you take a look at it, promising stuff there.

https://crystal-lang.org/

edit: did i win the fastest comment to the bottom on hn? what do I get?

jokes aside, care to explain why folks?

In all seriousness, the results speak for themselves, when comparing to Ruby in terms of speed.

https://github.com/kostya/benchmarks

https://github.com/kostya/crystal-benchmarks-game

https://github.com/the-benchmarker/web-frameworks

https://github.com/kostya/benchmarks

There are a few tests which have Julia and Nim implementations. It would be nice if the benchmarks game maintainer would include some more languages, like Nim, Crystal, and Zig.

This doesn’t have Nim, but is another interesting one. https://modelingguru.nasa.gov/docs/DOC-2783

I'm not familiar with Ruby, but it doesn't seem like TruffleRuby is really competitive with languages known for performance.

These are only simple benchmarks, but do indicate a rough ballpark for TruffleRuby: https://github.com/kostya/benchmarks

As I understand it, Crystal would be a good Ruby alternative if you want performance. This is of course a whole new language designed with performance in mind from the beginning and here is a repeating theme: you need to consider performance at the start, not 20 years later.

Nim is on par with C in a lot of benchmarks, e.g. https://github.com/kostya/benchmarks. Go is the king of concurrency, so whatever you compare to it loses. That said, Nim has async/await for concurrency, and I find threads and threadpools easy to use for paralelism.
Also same source, but more comparison against a wider variety of other languages and general benchmarks of data structures and algorithms: https://github.com/kostya/benchmarks
>'TruffleRuby has high potential in speed, as it is nine times faster than CRuby on optcarrot, a NES emulator benchmark developed by the Ruby Core Team.'

Really? But now the latest results are in apparently and judging by these benchmarks[0], I don't know what to make of this claim given how slow TruffleRuby is when compared to other similar languages or even Ruby. But its clear that its LLVM-based cousin Crystal still blows it out of the ocean in every benchmark if you're really talking about 'high performance'...

[0] https://github.com/kostya/benchmarks

If you are interested ins speed, here is a huge list of similar benchmarks: https://github.com/kostya/benchmarks
Kinda biased benchmarks u just took best of crystal and worst of nim from this xd -> https://github.com/kostya/benchmarks
Nim-lang. Its similar to Python in Syntax but runs at close to native speed. Nim has beautiful syntax.

https://github.com/kostya/benchmarks https://nim-lang.org/

> You can only see allocation advantage in micro benchmarks where the app stops before the GC will start.

The golang gc is tuned for latency at the expense of throughput, meaning if you look at the duration of time spent in GC over the course of the code execution, it would actually be longer compared to a GC tuned for throughput.

If you have a use case that requires high throughput, then you cannot change the GC behavior. Unlike on the JVM, where you have several GCs to choose from. The JVM is also getting two new low latency GCs for use cases that require low latency.

And it's not just microbenchmarks where Java does better than golang, it's especially longer running processes where the JVM's runtime optimizations really kick in. Not to mention that the JVM is getting value types as well to ease the load on the GC when required (it does an excellent job as it is even without value types).

I did a dummy port of the C# version of the Havlak code here[1] to Java, preserving the same behavior and not making any data structure changes. On the machine I tested on, the C# version took over 70 seconds to run, while the Java version took ~17 seconds. In comparison, the C++ version took ~24 seconds, and the golang version took ~30 seconds.

Yes, you could most likely spend much more time tuning the C++ version, avoiding allocations, and so on, but at the expense of readability. This is what the JVM gives you, you write straight-forward, readable code, and it does a lot of optimizations for you.

The brainfuck2 benchmark is especially interesting. Kotlin tops the list, but I was able to get Java to the same performance since Kotlin by writing it in a similar manner as the Kotlin code. Again, Java/Kotlin beat out even C++ when I tested them, and by quite a margin.

[1] https://github.com/kostya/benchmarks

Since you are asking about performance of some languages, you may find this helpful https://github.com/kostya/benchmarks
You can find them although they tend to be out of date and/or piecemeal. I think they're still useful.

For example:

https://github.com/drujensen/fib

https://gist.github.com/sdwfrost/7c660322c6c33961297a826df4c...

https://github.com/kostya/benchmarks

My general sense is that Nim is most competitive as an alternative to Python and/or things like Julia, in that it's as expressive and easy to understand as those, but has performance closer to C or Rust than something like Python. Julia is similar to Nim but I think Nim seems cleaner in its implementation overall, and targets much more general use scenarios.

I've been really impressed by Nim. There are some little things that have irritated me, like case insensitivity, but I wish it got more traction in the community. Right now the only thing that it doesn't seem to have going for it is library support. For the numerical applications I do I'd prefer it over everything else, except everything else has huge resource bases, so something that's already packaged in those languages would have to be done from scratch in Nim, which isn't feasible.

I know you're publicly directly messaging, but as an outsider to the discussion, I don't see any actual mention in your posts why D isn't included. Just that it "used to be."

There are many of benchmark projects that include D and don't appear to be struggling under any kind of massive burden. For example:

https://github.com/kostya/benchmarks

What are your actual reasons for not keeping D... "scala... and Clojure and ..."? The results in the previous link show D as a massive competitor (sometimes 1st place beating C and C++) on both memory and speed. Wouldn't the purpose of benchmarks be... to highlight useful, highly-scoring languages? Isn't that one of the primary reasons people read benchmarks?

(The D implementations are also often smaller in lines of code, as per this benchmark project:

https://togototo.wordpress.com/2013/08/23/benchmarks-round-t...

)

Last fall I tried to figure out why Rust is slower than D in a base64 benchmark (https://github.com/kostya/benchmarks). It turned out that decoding in Rust is really fast but encoding is a bit slow. I looked at the assembly output of both programs and I could more or less follow the steps in the Rust version, but the D version was only about 5 SIMD instructions (without a single bit shift). I also checked the source code, the Rust version is fairly long, because a part of the hot loop is manually unrolled, but the D version looked like the simple, naive implementation in C. It might be a result of newer LLVM version (LDC was already using LLVM 5 and rustc is still on LLVM 4) or rustc still might have issues with optimizer settings (both binaries were compiled for Intel Haswell).
The reason D isn't popular is simple and not a fault of the language itself. It's one of marketting.

- Go is backed by Google.

- Rust is backed by Mozilla.

- C# is backed my Microsoft.

- D is backed by... D users... and the two language architects.

A few schools use it. A few companies use it.(::cough::This little company called Facebook::Cough::)

The other is target market. D appeals to a new, or more advanced kind of programmer. A new niche or category. A programmer that doesn't get produced at most colleges, or demanded by most companies. And if they're not produced or demanded, that category isn't going to fill up with programmers. Nobody would have used Java if schools across the nation didn't tell kids that "Java was cool" and the JVM wasn't something to be afraid of.

There are categories of programmers. No one can say that a LISP programmer is quite like a C programmer.

- C programmers want "nicer assembler." C.

- C++ programmers don't want a GC. (Though D is working toward a GC-less standard lib.)

- Python/Java/C# programmers are (more likely to be) afraid of systems languages that can segfault like crazy if you don't know what you're doing.

D appeals to programmers who aren't afraid of the entire stack. They write, understand or at least appreciate assembler and caches, while also dabble in high-level templates and functional programming.

I can absolutely say I've grown as a programmer after looking through D features and going, "Why don't I know how that works?"

The guys behind D are brilliant, and professional. Watch any of their talks. Andrei Alexandrescu works for Facebook and still writes plenty of C++. He also wrote THE quintessential "Modern C++ Design" book on template meta-programming. And Walter Bright has been writing compilers for decades.

This is not a toy language.

Here's plenty of benchmarks showing D as fast as (and sometimes beating) C/C++ _and_ having low memory usage.

https://github.com/kostya/benchmarks

Meanwhile, the benchmark is misleading. Because not only is generally as fast as C/C++, but it's way more productive to write in.

But back to adoption. It's simple. 1) No corporate backers pushing it, 2) A language for advanced/multi-talented programmers is either scary or "unnecessary" to single-talented programmers.

Most C programmers are afraid of, or think templates are unnecessary. Most C++ programmers are afraid of a garbage collector (even if it's a completely deterministic one that only fires off during specific allocation points).

As a mostly C++ programmer, I was apprehensive of D's GC. But the more I read, the more confident I've been, and, I've had zero actual problems with it. I'm pushing 130 FPS on my netbook just fine, and because D isn't a toy language, I can easily move to static pools and have zero GC allocations.

Crystal is one of the most exciting new languages out there. I have been using it for my one off tasks at Ola and it works pretty well. It is huge plus point that you have safety harness of statically typed languages and speed comparable to java and scala(unscientific benchmark here https://github.com/kostya/benchmarks)
For that project, we had very specific requirements: easily handle SSL/TLS with contexts and control over self-signed vs certificate checking, JSON processing, speed, nice syntax, and one of the most challenging requirements: statically compiled, linkable against musl and libressl, while still supporting mingw_64 for windows. Only a few languages have flexible compilers that can do this; for example, rust can't (afaik).

The experience so far has been outstanding. Nim has functioned flawlessly with a minimum of magic. It seems to work very cleanly and the compiler is cleanly integrated, but still swappable (ie between gcc, clang, ming..) Nicely color-coded, too.

Exceptions are caught with full tracebacks, and pre-compile checks quickly point out exact location of syntax errors. (Good, clear error messages are surprisingly missing from many languages.)

Here's an awesome example; in my first day of coding, I was able to replicate python's "+" string concatenator ("hello" + "world" versus "hello" & "world" in Nim) with a one-liner:

    proc `+` (x, y: string): string = x & y
This is pretty amazing; not only is it readable and concise (and more than a passing similarity to python's lambda, of course) but nim comes with the ability to define new operators right in the language, and the compiler raises an error if operators, procs, types, etc would introduce ambiguity.

Nim compiles quickly and its type inference (where it guesses what type of variable you're working with) makes strong typing mostly painless, and you still get all of the advantages (type safety, speed) of static typing.

There are some trade-offs that are made (obviously), but the language designers seem to make trade-offs in favor of speed and robustness over language features -- but this still leaves a lot of room for features.

I also like how the syntax has a lot of similarities to Python's. The only thing I've missed so far is a nim interpreter, so that I can get up to speed faster on the syntax or try things out quickly. The tutorial on the Nim website is definitely not for beginning coders (who would probably be quickly scared off by words like lexical), but it quickly covers the language syntax for experienced coders and seems to borrow a lot of the best ideas from other languages.

Nim is basically awesome. The few downsides are that the standard library is still pretty light (but that gives you an opportunity to build something great and have it be widely adopted), that there's no interpreter, and that the tooling is still a bit lighter than older languages. All of these will be improved with time.

And, it's fast. Really fast. Compare nim in these benchmarks[1] to any other mid-level (or even low-level) language and it really shines. It's generally much faster than Go, for instance.

1. https://github.com/kostya/benchmarks

I'm surprised you're able to get Turing to run faster, to be honest. I looked at Turing and was impressed by it, but in general my experience with MCMC libraries in Julia is similar to what the OP posted, being much slower than Stan (MAMBA and Turing being the two Julia libraries I've tried--both have seemed nice but have run significantly slower on the tasks I've used them for).

Incidentally, I haven't been able to get Turing to run on Julia 0.6, and am not going to reinstall an earlier version just for Turing. I've been waiting for a little while now.

My experience with Turing and MAMBA have kind of diminished my enthusiasm for Julia. Both libraries kind of represent what I was looking for in Julia (similar to what you mention about using your Julia stack), but the speed was a kind of a rude awakening. I'm kind of coming to the opinion that LLVM-based languages need to demonstrate much more consistent performance before they're really ready to replace C (Rust, Nim, and Crystal seem like they might be on their way, though).

I recommend someone thinking about Julia pay attention to these benchmarks:

https://github.com/kostya/benchmarks

Yes, Julia is really fast with some things, but for other things it's much slower, and those slow parts become the lowest common denominator.

Hopefully things improve, though, because I do like Turing and MAMBA on paper.

Here's a link to benchmarks of different languages including Elixir. https://github.com/kostya/benchmarks
Not trying to be a troll, but...

I was on the Julia train full-force until I started using it more heavily. My sense is that the benchmarks are a little misleading.

These benchmarks give a little flavor of what I'm talking about: https://github.com/kostya/benchmarks

If you look at them, you'll see that Julia is indeed crazy fast in some cases. But in other cases, the performance is kind of middling. The native threads implementation of matmul is obscenely fast, but the "pure Julia" implementation is pretty slow--faster than R or Python for sure, but not in the same ballpark as C.

You could argue "well, why would you not use the native threads implementation in that case?" However, for someone implementing a routine in Julia, the implementation would be the pure one. That's the point.

With something like Rust, or Crystal, or Nim, what is promised is basically what you get: something in the ballpark of C. But with Julia, I feel like it's kind of unpredictable.

The problem is that the slow parts end up being a bottleneck that slows down the rest of the code, and that slow code is slow enough that it's not worth it relative to C or Python/R. That is, if you're writing heavy numerics, when you get to 20x slower than C, you might as well write it in C (or Rust or Crystal or Nim), and wrap Python or R around it, because that 20x is enough to kill it. At that point you might as well go with the established language even though it's 100x slower as an interface, and use something fast for the heavy lifting.

Crystal I can't say much about. It seems great, but the "everything is an object" makes me nervous--I tend to get anxious about inheritance because it drives me nuts. Nim seems more attractive to me. But who knows--it's also not like you can't have more than one language out there.

I wonder if this would be a feasible funding strategy for Crystal (https://crystal-lang.org/) as well. From the scattered benchmarks I could find online†, Crystal seems to be just as fast, if not a bit faster.

https://github.com/kostya/benchmarks, http://blog.seraum.com/crystal-lang-vs-nodejs-vs-golang-vs-j...

Good luck on seeing D on that site. There seems to be a active war between the sites maintainer and D. There used to be D benchmarks on the old site and people tries to release new code to the site, but for some reason the current maintainer has a issue with D.

And he thinks its no unfair that D is not resented when a lot of people look at his site and think: "If your not here as a language, then the language must be bad".

Good luck on changing his mindset because i have seen topics going back years regarding D not being there.

If you want a more impartial benchmark set:

https://github.com/kostya/benchmarks

And where is D on those tests ;)

D is a good C++ replacement language. But it needs more community support.

Its advised to develop in DMD compiler and only compile the release code in LDC/GCC as this can be twice or trippel as fast. Not saying that DMD is slow ( unless you consider Go/Swift slow).

https://github.com/kostya/benchmarks

With the 3 reference compilers:

D DMD also written sometimes as simply D D LDC = LLVM backend D Gdc = GCC backend

The benchmarks are somewhat out of date for D, as it has never versions of Rust, Go etc where as the D LDC is still using the very old 0.15.2-beta1. Now its at 1.3.0 beta1, that support LLVM 3.5 to the latest 4.0 ( what is faster ).

Except that there are other languages like Go, Crystal, D, Nim that all offer memory safety ( thanks to there GC ), with a light GC ( like reference counting ). And based upon the same benchmarks that Rust in participates, they are close or even faster at times, with close or better memory usage.

https://github.com/kostya/benchmarks

https://github.com/kostya/benchmarks

it looks more speedy than ruby, but also is eating a lot of memory, and for some benchmarks is not that fast :)

Will be interesting with substrate vm if the results or memory are the same.

There actually is one high-level Python-like language that really is almost "as fast as C".

http://nim-lang.org

I am using it for years already, and it is really performant, somewhere between C and Rust. I am still wondering why so few people use it.

Benchmark: https://github.com/kostya/benchmarks

Nim vs Rust: http://arthurtw.github.io/2015/01/12/quick-comparison-nim-vs...

Performance discussion: http://forum.nim-lang.org/t/2261

Embedded Nim: https://hookrace.net/blog/nim-binary-size/

Nim on LLVM: https://github.com/arnetheduck/nlvm

There's some benches between various languages, including Crystal as well, for those interested https://github.com/kostya/benchmarks

EDIT: There's also this: https://github.com/nsf/pnoise

All the languages couldn't fit in the talk, but the full benchmarks are here:

https://github.com/kostya/benchmarks

As always, take benchmarks with a graint of salt. We use benchmarks mostly to know how far away we are from C (1x, 10x, 100x, etc.), so any language that has a speed "in the order of" C is good enough for us, small differences don't matter much.

will beautiful like python and faster than Go do? then Nim - http://nim-lang.org will do just fine. Anecdotal benchmarks like this one https://github.com/kostya/benchmarks will show you that Nim comes out at the top (so does Crystal for some, FWIW).
In that case, what he missed was AOT optimization :)

Many Lisps have had compilation to binaries. But having a binary doesn't imply being faster than an interpreted language.

Theoretically, a JIT optimization can end up with faster code than an AOT optimizer can produce, thanks to runtime profiling.

In fact, I'd be interested in seeing benchmarks, which Crystal is probably not ready to share, but may have conducted: https://github.com/manastech/crystal/blob/master/samples/fan...

Edit: this[0] seems to indicate that the optimizations are in the same ballpark as Go and D, which is really impressive.

[0]: https://github.com/kostya/benchmarks

I've been looking at the benchmarks https://github.com/kostya/benchmarks and am pleasantly surprised about the speed. It definitely smokes Ruby, but also is usually faster than Go. I know that all benchmarks are relative but Crystal seems a great language from a performance viewpoint.
Yes. There are some here:

https://github.com/kostya/benchmarks

Here's another one:

https://github.com/nsf/pnoise

And then in the samples directory ( https://github.com/manastech/crystal/tree/master/samples ) there are some more but without time results (but you can try them yourself).