What does HackerNews think of FrameworkBenchmarks?
Source for the TechEmpower Framework Benchmarks project
I used https://github.com/TechEmpower/FrameworkBenchmarks and in the future i will write fortunes and other benchmarks with have more things going on!
Yeah this test basically shows that Python backed by uWS is crazy fast, but is not an direct comparison to uWS to Go.
This test is just an troughput test, with is very useful to measure raw performance.
More tools like caching tools, a better database client etc is need to construct an complete scenario, and i'm working on it! (Maybe in 1 week or 2 weeks will be done)
https://github.com/TechEmpower/FrameworkBenchmarks
https://www.techempower.com/benchmarks/#section=test&runid=1...
The benchmarks are TechEmPower plaintext https://github.com/TechEmpower/FrameworkBenchmarks
preliminary results here: https://www.techempower.com/benchmarks/#section=test&runid=1...
Web developer dont care if the framework is written in native or it is an wrapper, developers just want something that works in a nice way :D
The benchmarks are in https://github.com/TechEmpower/FrameworkBenchmarks
EDIT: some preliminary results https://www.techempower.com/benchmarks/#section=test&runid=1...
The downvotes are likely because it's 20 years later and TechEmpower benchmarks are neutral and open-source: https://github.com/TechEmpower/FrameworkBenchmarks
It's not tutorial-style, but it does contain hundreds of sample web apps (that all do the same thing, but still)
For example, look at a full platform Java Spring Boot example here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
The controller code, the model code and even the repository code are all very reflective of the primitives you'd find in production code.
The only differences would be when you add additional complexity, such as validations and other code into the mix, such as additional service calls etc., but if even the simpler stuff in Python (as an example) would be noticeably slower, then it stands to reason that it wouldn't get magically faster on the basis of less overhead alone.
The exceptions to this which i could imagine would be calls to native code, since the aforementioned Python has an amazing number crunching or machine learning support, but that is hardly relevant to web development in most cases.
Otherwise we risk running into the "No true Scotsman" fallacy: https://en.wikipedia.org/wiki/No_true_Scotsman
In such a case, we'd adamantly claim that these benchmarks are simply not good enough even for educated guesses and we'd have to build out our full system to benchmark it, which no one actually has the resources for - in practice you'd develop a system and then would have to do a big rewrite, as was the case with PHP and Facebook, as well as many other companies.
On the other hand, premature optimization and dogmatic beliefs are the opposite end of the spectrum: you don't need to write everything in Rust if using Ruby would get your small application out the door in a more reasonable amount of time.
There are no easy answers, everything depends on the context (e.g. small web dev shop vs an enterprise platform to be used by millions) and the concerns (e.g. resource usage and server budget vs being the first to market, as well as developer knowledge of tech stacks, for example market conditions of PHP vs Rust).
Regardless, in my eyes, there's definitely a lot of value in these attempts to compare idiomatic interpretations of common logic for web development primitives (serving network requests in a known set of formats).
Of the bunch, by only looking at the code, which of them would you like to get dumped on you 2 weeks before a deadline? Which is the most ergonomic?
Techempower classifies each one. Actix is what they call a "platform"
>a platform may include a bare-bones HTTP server implementation with rudimentary request routing and virtually none of the higher-order functionality of frameworks such as form validation, input sanitization, templating, JSON serialization, and database connectivity
https://www.techempower.com/benchmarks/
The source code is published too (https://github.com/TechEmpower/FrameworkBenchmarks)
$ wrk -t2 -c90 -d9s http://localhost:8080/plaintext
Running 9s test @ http://localhost:8080/plaintext
2 threads and 90 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 831.78us 364.48us 7.56ms 70.19%
Req/Sec 40.55k 3.31k 48.04k 74.44%
726417 requests in 9.01s, 87.98MB read
Requests/sec: 80603.64
Transfer/sec: 9.76MB
and here's dash running with the kdb networking disabled (best of three):
$ wrk -t2 -c90 -d3s 'http://127.0.0.1:8080/?f=204&k=hi&v=1'
Running 3s test @ http://127.0.0.1:8080/?f=204&k=hi&v=1
2 threads and 90 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 787.72us 213.62us 3.49ms 71.85%
Req/Sec 44.82k 3.04k 60.44k 83.61%
271946 requests in 3.10s, 16.08MB read
Requests/sec: 87671.23
Transfer/sec: 5.18MB
And comparing a 100 line C program to hundreds or thousands of lines of go or C or Java is a bit pointless. If the 100 lines of C doesn't do what you want, I'll throw it away and write a different 100 lines. That's what brief programs get you.
NB: I would have tried ulib but it wouldn't even build on my laptop.
For what it's worth, in our work on the TechEmpower Framework Benchmarks [1], we have received a great deal of positive feedback and gratefulness. We really appreciate it since it makes the project so much more fun and meaningful for us when we hear that people have found it useful (or at least interesting). We never cease to be amazed at the generosity of the community. All of this often makes me wish we could spend more time on the project. (We're working on getting Round 15 out soon!)
If you want to make your chosen platform fast you can see what they're doing in the src to get those results. The ones not marked "stripped" and with "full" for ORM are very realistic examples. Not sure where you're going with this? Nobody's real workload looks exactly like benchmarks.
They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
[1] http://www.techempower.com/benchmarks/\n[2] https://github.com/TechEmpower/FrameworkBenchmarks
Speaking of benchmarks, we'd be very happy to receive a pull request [1] with a test implementation if anyone is interested.
Edit: someone created an issue asking for contributors: https://github.com/TechEmpower/FrameworkBenchmarks/issues/45...
Some people here are very dismissive of benchmarking in general, but if you are interested in evaluating programming languages for use in web applications, you would be interested in TechEmpower benchmarks, which are very thorough. [0] You can even send pull requests with updated languages, frameworks, or better implementations of different tests to give more competitive/accurate results. Enjoy.
[0] http://www.techempower.com/benchmarks/ [1] https://github.com/TechEmpower/FrameworkBenchmarks
UPDATE: Fixed link to the benchmarks.
Might I trouble you for a test implementation and pull request [1]? :)
To add to this, I'd encourage you to look at the source code used to implement the tests in the higher performance frameworks. Ryan's third point is very relevant: you may find that modern frameworks leverage both high-performance platforms and much of the pragmatic thinking that was introduced a few years back in the frameworks with which you are more familiar.
Source code link: https://github.com/TechEmpower/FrameworkBenchmarks/
In fact, the ETA for Round 6 is next Thursday, June 20. If you can point out any tuning for us to put in place for the Ruby tests before June 17, we may be able to get pull requests accepted before we kick off the test runs.
https://github.com/TechEmpower/FrameworkBenchmarks
We welcome all pull requests, suggestions and criticisms.
--- edit: found it. https://github.com/TechEmpower/FrameworkBenchmarks
Seems to be using cluster. I'm not surprised that Go and servlets were faster... just surprised to see NodeJS in the middle of the pack position. Though performance isn't the only reason I really like NodeJS
I don't think they have anything against C#, probably no one just has committed C# version to be benched.
(edit: whops, linked fork first)