You’re basically pitching this as a more complicated version of airflow that does basically the same thing, but slightly differently, and scales better?

… but your core dependencies are a Kafka cluster and an elastic search cluster which are both a pain in the ass to scale; so really, could you run this seriously without a really expensive hosted cloud instance of both of those?

This kind of wording:

> Since the application is a Kafka Stream, the application can be scale infinitely

Is a major turn off to me.

Kafka cannot scale infinitely. Nothing can. In fact, Kafka can be a pain in the ass to scale.

In makes me question some of the other commentary on the project.

I agree with you that Kafka & ElasticSearch can be a pain to scale if you need to have a horizontal and vertical scaling.

On other side, on single machine, it's really has easy to setup. With this, you will have the same scaling than Airflow for exemple since it depend on a non scalable database (mysql or postgres). But the chance you will have with Kestra is that you will be able to scale to multiple node for your backend (as well with kestra that allow scaling all services). When you hit the limit with standard database, you will be stuck.

And yes clearly infinite scale is not a literal statement terms, nothing can scale infinitely but since the architecture is really robust (and scalable), the issues will be on other aspects than Kestra (cloud limit, database overload, ...).

A final point and a more important one, the backend are all pluggagle in Kestra since Kestra is really think as module: Look at the directory here : https://github.com/kestra-io/kestra :

- runner-kafka & runner-memory are 2 implementation of Kestra, you can add a new one that will use Redis, Pulsar, ...

- repository-elasticsearch & repository-memory is the same, you can implement another one, I started one implementation for JDBC that I don't have the time to finish for now : https://github.com/kestra-io/kestra/pull/368