“There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.”
Obviously a bit of an over simplification since there have been some advancements in that area but not far off.
This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb.
Look at k8s and etcd for a very widely used example of this approach as well.
I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.
> I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.
As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer).
One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO.
If you do build a plugin system though, you can pass off the work of maintaining the other implementations!
[0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all