I would like to understand why Mastodon requires such a huge amount of hardware for mediocre traffic volumes. Not just the lazy "it's Rails" answer - I know Rails is a resource hog, but that doesn't go far enough to explain the extreme requirements here.

As a point of reference, look at what Stack Overflow is run on. As a caveat, SO is probably more read-heavy than Mastodon, but it also serves several orders of magnitude more volume (on a normal day in 2016 they would serve 209,420,973 HTTP requests[0]). They did this on 4 DB servers and 11 web servers. And in fact, it can (and has) worked serving this volume of traffic on only a single server.

With this setup SO was not even close to maxing out their hardware (servers were under 10% load, approximately). SO also listed their server hardware[1] in 2016. I don't know enough about server hardware to assess the difference, but to my eye they look similar on the web tier with similar amounts of memory, similar disk, etc.

I'm not saying Hachyderm is doing anything wrong, but it makes me wonder if there's a fundamental problem with the design of Mastodon. And to be clear I understand that this particular issue was caused by a disk failure, but that they even had this hardware in place running Hachyderm is surprising to me.

[0] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

[1] https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...

Every post and every reply generates a request to all servers you are federating to: https://aeracode.org/2022/12/05/understanding-a-protocol/

This ultimately polynomial direct-connection approach is clearly fatally unscalable.

The fediverse needs to figure out a hubs-and-spokes or supernodes pattern so that service providers can scale up syncing, indexing etc.

ie, my personal instance should be able to offload most of the message passing to an supernode intermediary that lots of other instances use for federation so that my instance only needs one connection, and the supernodes only need to connect to each other and their local network.

I am also closely following https://github.com/nostr-protocol/nostr to see how they go along, because I am growing weary of the "tech elite" that is moving to Mastodon and is pushing for "moderation by committee". I've gotten myself with discussions already with people who actually want server operators that want only to open federation for those that abide by some "Covenant". This seems rooted in good intentions, but it reeks of something that might lead to a corporate copout of a network which is supposed to be open.