I wish there was something like tailscale but without a central server.

You might want to consider innernet. It's still got a central server, but it's self-hosted and similarly easy to deploy. Check it out here: https://github.com/tonarino/innernet