What does HackerNews think of FrameworkBenchmarks?

Source for the TechEmpower Framework Benchmarks project

Language: Java

#2 in Framework
There might also be a chance you can have your webserver included in the techempower benchmarks for their next round. Their code is submitted by experts and open-source: https://github.com/TechEmpower/FrameworkBenchmarks
Actually the benchmark is on https://github.com/TechEmpower/FrameworkBenchmarks with is basically this with more headers, but you are right, is not enought, i used TechEmPower because is very popular. I have some issues (new features comming) open to create an better JWT token support, database and much more, i will post these in the future!
i did not say that Python is faster than Go, i say that this Python Framework outperforms Golang Fiber framework in troughput.

I used https://github.com/TechEmpower/FrameworkBenchmarks and in the future i will write fortunes and other benchmarks with have more things going on!

Actually uWebSockets and uSockets will perform better than this (2x at least in my local tests), I need to do a lot of copying, instancing and crossing Python GIL.

Yeah this test basically shows that Python backed by uWS is crazy fast, but is not an direct comparison to uWS to Go.

This test is just an troughput test, with is very useful to measure raw performance.

More tools like caching tools, a better database client etc is need to construct an complete scenario, and i'm working on it! (Maybe in 1 week or 2 weeks will be done)



socketify is based on uWebSockets (C++) uWebSockets is comparad to FastHTTP and socketify adds a lot of features on top.

The benchmarks are TechEmPower plaintext https://github.com/TechEmpower/FrameworkBenchmarks

preliminary results here: https://www.techempower.com/benchmarks/#section=test&runid=1...

Python framework because is an framework for python, like most performance focused frameworks for python, this uses a lot of native code. uvicorn uses uvloop that is an wrapper for libuv.

Web developer dont care if the framework is written in native or it is an wrapper, developers just want something that works in a nice way :D

The benchmarks are in https://github.com/TechEmpower/FrameworkBenchmarks

EDIT: some preliminary results https://www.techempower.com/benchmarks/#section=test&runid=1...

That did happen.

The downvotes are likely because it's 20 years later and TechEmpower benchmarks are neutral and open-source: https://github.com/TechEmpower/FrameworkBenchmarks

Are you familiar with https://github.com/TechEmpower/FrameworkBenchmarks ?

It's not tutorial-style, but it does contain hundreds of sample web apps (that all do the same thing, but still)

In the case of the TechEmpower benchmarks, i'd argue that it's actually pretty much the same thing as real world applications: https://github.com/TechEmpower/FrameworkBenchmarks

For example, look at a full platform Java Spring Boot example here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...

The controller code, the model code and even the repository code are all very reflective of the primitives you'd find in production code.

The only differences would be when you add additional complexity, such as validations and other code into the mix, such as additional service calls etc., but if even the simpler stuff in Python (as an example) would be noticeably slower, then it stands to reason that it wouldn't get magically faster on the basis of less overhead alone.

The exceptions to this which i could imagine would be calls to native code, since the aforementioned Python has an amazing number crunching or machine learning support, but that is hardly relevant to web development in most cases.

Otherwise we risk running into the "No true Scotsman" fallacy: https://en.wikipedia.org/wiki/No_true_Scotsman

In such a case, we'd adamantly claim that these benchmarks are simply not good enough even for educated guesses and we'd have to build out our full system to benchmark it, which no one actually has the resources for - in practice you'd develop a system and then would have to do a big rewrite, as was the case with PHP and Facebook, as well as many other companies.

On the other hand, premature optimization and dogmatic beliefs are the opposite end of the spectrum: you don't need to write everything in Rust if using Ruby would get your small application out the door in a more reasonable amount of time.

There are no easy answers, everything depends on the context (e.g. small web dev shop vs an enterprise platform to be used by millions) and the concerns (e.g. resource usage and server budget vs being the first to market, as well as developer knowledge of tech stacks, for example market conditions of PHP vs Rust).

Regardless, in my eyes, there's definitely a lot of value in these attempts to compare idiomatic interpretations of common logic for web development primitives (serving network requests in a known set of formats).

What is the most fun and shocking to me to browse the code https://github.com/TechEmpower/FrameworkBenchmarks to see just how unergonomic and messy the code gets in something like this. 20 something files in 10 directories fiddled with config files isn’t uncommon.

Of the bunch, by only looking at the code, which of them would you like to get dumped on you 2 weeks before a deadline? Which is the most ergonomic?

The source code is available on github: https://github.com/TechEmpower/FrameworkBenchmarks

Techempower classifies each one. Actix is what they call a "platform"

>a platform may include a bare-bones HTTP server implementation with rudimentary request routing and virtually none of the higher-order functionality of frameworks such as form validation, input sanitization, templating, JSON serialization, and database connectivity

Here's fasthttp running on my machine (best of three):

    $ wrk -t2 -c90 -d9s http://localhost:8080/plaintext
    Running 9s test @ http://localhost:8080/plaintext
      2 threads and 90 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
        Latency   831.78us  364.48us   7.56ms   70.19%
        Req/Sec    40.55k     3.31k   48.04k    74.44%
      726417 requests in 9.01s, 87.98MB read
    Requests/sec:  80603.64
    Transfer/sec:      9.76MB
which I got by checking out https://github.com/TechEmpower/FrameworkBenchmarks.git, disabling the mysql connection, and running frameworks/Go/fasthttp's ./server-mysql (which is what the benchmark script seems to do). I thought this would be easier than getting dash running the TechEmpower results.

and here's dash running with the kdb networking disabled (best of three):

    $ wrk -t2 -c90 -d3s ''
    Running 3s test @
      2 threads and 90 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
        Latency   787.72us  213.62us   3.49ms   71.85%
        Req/Sec    44.82k     3.04k   60.44k    83.61%
      271946 requests in 3.10s, 16.08MB read
    Requests/sec:  87671.23
    Transfer/sec:      5.18MB
My laptop isn't a beefy "Dell R440 Xeon Gold + 10 GbE" -- this is just a loopback test, but it's already disinclined me to spend any more time on it; Fasthttp definitely is impressive how close it gets, but dash is still faster.

And comparing a 100 line C program to hundreds or thousands of lines of go or C or Java is a bit pointless. If the 100 lines of C doesn't do what you want, I'll throw it away and write a different 100 lines. That's what brief programs get you.

NB: I would have tried ulib but it wouldn't even build on my laptop.

Yes! We accept PRs from the community at the GitHub repository [1]. We do ask that you submit what you believe is production quality code that would be suitable to run a real web application. We are liberal with how that is interpreted, but we reserve the right to reject code that is too experimental.

[1] https://github.com/TechEmpower/FrameworkBenchmarks

The author is promoting gratefulness, and that is awesome. I try to do my part when I can with hand-crafted emails or messages.

For what it's worth, in our work on the TechEmpower Framework Benchmarks [1], we have received a great deal of positive feedback and gratefulness. We really appreciate it since it makes the project so much more fun and meaningful for us when we hear that people have found it useful (or at least interesting). We never cease to be amazed at the generosity of the community. All of this often makes me wish we could spend more time on the project. (We're working on getting Round 15 out soon!)

[1] https://github.com/TechEmpower/FrameworkBenchmarks

the source to all the benchmark is available https://github.com/TechEmpower/FrameworkBenchmarks

If you want to make your chosen platform fast you can see what they're doing in the src to get those results. The ones not marked "stripped" and with "full" for ORM are very realistic examples. Not sure where you're going with this? Nobody's real workload looks exactly like benchmarks.

It's an open sourced benchmarking comparison. If you can improve them, shoot them a pull request!


It's open source benchmarking. There are instructions on how to submit a language-framework of your choosing.


TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking.

They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".

I'm tempted to tackle this even though I don't currently have the time.

[1] http://www.techempower.com/benchmarks/\n[2] https://github.com/TechEmpower/FrameworkBenchmarks

We'd love to have a Meteor test. Any Meteor developers interested in submitting a pull request with a Meteor test?


Indeed, I've only taken a quick look, but I like what I see!

Speaking of benchmarks, we'd be very happy to receive a pull request [1] with a test implementation if anyone is interested.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/

Can you post a link to the app? Have you considered running the TechEmpower benchmark app? https://github.com/TechEmpower/FrameworkBenchmarks
Looks good. It would be nice to see it on the Web Framework Benchmarks (https://github.com/TechEmpower/FrameworkBenchmarks).

Edit: someone created an issue asking for contributors: https://github.com/TechEmpower/FrameworkBenchmarks/issues/45...

I know you put effort this and do not want to give you a hard time, but a lot of people on HN will give you a hard time with benchmarks, especially as simple as this, gracious concluding paragraph or not.

Some people here are very dismissive of benchmarking in general, but if you are interested in evaluating programming languages for use in web applications, you would be interested in TechEmpower benchmarks, which are very thorough. [0] You can even send pull requests with updated languages, frameworks, or better implementations of different tests to give more competitive/accurate results. Enjoy.

[0] http://www.techempower.com/benchmarks/ [1] https://github.com/TechEmpower/FrameworkBenchmarks

UPDATE: Fixed link to the benchmarks.

Speaking of HHVM, would any PHP subject matter experts be interested in submitting pull requests to add HHVM tests to our project [1]? It would be worthwhile to see raw PHP and one or two mainstream PHP frameworks on HHVM versus the existing nginx/FPM configuration of the same.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/

Oh heck yes, the first annual Alternative Web Framework Day is underway!

Might I trouble you for a test implementation and pull request [1]? :)

[1] https://github.com/TechEmpower/FrameworkBenchmarks

Great response, RyanZAG.

To add to this, I'd encourage you to look at the source code used to implement the tests in the higher performance frameworks. Ryan's third point is very relevant: you may find that modern frameworks leverage both high-performance platforms and much of the pragmatic thinking that was introduced a few years back in the frameworks with which you are more familiar.

Source code link: https://github.com/TechEmpower/FrameworkBenchmarks/

We'd love to receive these test cases as pull requests if you have the time to contribute them. :)


I can't speak to Matt's source code, but you can find ours at GitHub [1].

In fact, the ETA for Round 6 is next Thursday, June 20. If you can point out any tuning for us to put in place for the Ruby tests before June 17, we may be able to get pull requests accepted before we kick off the test runs.

[1] https://github.com/TechEmpower/FrameworkBenchmarks

For those new to these benchmarks, all of the source code is up on github:


We welcome all pull requests, suggestions and criticisms.

Where's the source for the implementations? Of interest to me is the NodeJS instances.. was this single-process, or multi-process via cluster (which is recommended for higher performance). The only source I saw was for the testing framework.

--- edit: found it. https://github.com/TechEmpower/FrameworkBenchmarks

Seems to be using cluster. I'm not surprised that Go and servlets were faster... just surprised to see NodeJS in the middle of the pack position. Though performance isn't the only reason I really like NodeJS

Their repo at https://github.com/TechEmpower/FrameworkBenchmarks is open for pull requests

I don't think they have anything against C#, probably no one just has committed C# version to be benched.

(edit: whops, linked fork first)