How different are the various frameworks here?
Are they all running with SSL? Do they keep track of session cookies? Do they have Content Security Policy headers? etc.. There's probably a million different feature configuration differences between each framework.
> How different are the various frameworks here?
Quite different. We have included test implementations for frameworks that span 26 computer languages. There are an infinite variety of opinions about how to do things in computer programming in general, and our project is no exception.
And on the other hand, fairly similar. Tests should stick to the requirements [1], which are designed to be permissive but sufficiently clear on the expected work load of each test type. The principal goal is that test implementations should be realistic, and we will mark those the community believes are not realistic as "stripped" implementations.
> Are they all running with SSL?
No, not yet. But planned future test types would include SSL/TLS.
> Do they keep track of session cookies?
Generally no. These are intended to exercise anonymous requests. But a future test type could include session management.
> Do they have Content Security Policy headers?
No, that is not a requirement of our tests. We have specified which headers we expect. Others are optional.
> There's probably a million different feature configuration differences between each framework.
Yes. And it can be challenging at times to get all of these opinions to fit into the same box. For example, we've to-date kept SQLite implementations out of the project since those would not incur network costs. And other times the box has to be reshaped a bit to make room for new consensus opinions about what is suitable for "production." We recently made a decision to allow for innovative features in the Postgres protocol that are—in a manner of speaking—analogous to automatic pipelining. That conversation is still ongoing on our discussion forum.
So, anyone can submit a framework here as a docker container? I'd love to send in my custom framework for benchmarking.