There're so many alternatives to Airflow nowadays that you really need to make sure that Airflow is the best solution (or even a solution) to your use case. There's plenty of use cases better resolved with tools like Prefect or Dagster, but I suppose the inertia to install the tool everyone knows about is really big.

BTW, here's https://github.com/pditommaso/awesome-pipeline a list of almost 200 pipeline toolkits.

I've gone through a large number of these, and I think that Airflow is the best on Kubernetes for managed orchestration. The things I like are:

  * Source control for workflows/DAGs (using git-sync)
  * Tracking/retries with SLAs
  * Jobs run in Kubernetes
  * Web UI for management
  * Fully open source

I also use Argo Workflows, because I like its native handling of Kubernetes objects (e.g. the ability to manage and update a deployment as one of the steps), but it just doesn't have the orchestration/tracking side of things very well managed yet

tiagod

I absolutely love Cadence/Temporal. It can do anything the others can do (and you could implement any other engine with it), without any DSLs.

cpdean

I think this is the link to cadence: https://github.com/uber/cadence

and temporal: https://github.com/temporalio/temporal