One thing Hetzner doesn’t have yet (but it SHOULD!) is Object Storage, and it’s something I’m working on over at NimbusWS[0].

Another awesome thing about Hetzner is that bandwidth internally is free and automatically negotiated (i.e if you send traffic to a Hetzner IP it will flow internally).

One of the hardest things is being able to tell the bandwidth usage and when it’s inside/outside Hetzner.

[0]: https://nimbusws.com

Are you using Ceph for object storage?

I'd be super interested in hearing from someone who has set it up whether it's something an experienced but generalized sys admin could build, or if you need an expert in the matter.

Yup, Ceph is going to be the solution to power it. There are quite a few options in the space but I was basically down to Swift (powers OVH) and Ceph (powers Digital Ocean Spaces) and OpenIO (power's OVH's new experimental offering).

Ceph has much more public resources available (SwiftStack recently got acquired, seemed like they had most of the knowledge in the space), and I've actually set it up a few times now thanks to the excellent Rook[0].

It's definitely a lot easier to set up with Kubernetes (the tradeoff being you need to understand Kubernetes), but it's definitely manageable for a generalized sys admin (albeit one with a bit more experience). I've written about the process:

https://www.google.com/search?q=site%3Avadosware.io+ceph

[0]: https://rook.io

I am in the process of migrating off of Rook Ceph after using it in production for two years. Setting it up is easy thanks to Rook, but wait until Ceph gets under load, then the real fun begins. If you only need object storage, I suggest looking into SeaweedFS[0]. It's a far more lightweight and performant solution.

[0]: https://github.com/chrislusf/seaweedfs/