What does HackerNews think of partisan?

> I do wonder if maybe streaming large data chunks over Erlang distribution might be a problem and a secondary data channel (e.g. over udp or sctp) might be worth playing with.

You may want to take a look at the partisan[0] library written in Erlang. It is basically that, a reimagination of distributed Erlang, except that it can be multiplexed over multiple connections.

[0] - https://github.com/lasp-lang/partisan/

You might be interested in Partisan's fault injector. I don't know if @cmeiklejohn is still working on it, but it's really cool nonetheless.

https://www.youtube.com/watch?v=KrwhOkiifQ8

https://github.com/lasp-lang/partisan

I can also mirror this general guideline. I've run 250+ node erlang clusters just fine in the past. There were some caveats to how some built-in OTP libraries behaved but they were easy to replace or workaround in the past.

That was many years ago as well. The distributed erlang story has improved with more recent releases (better performance on remote monitors for example) which might push the number a little higher than 300 if you are careful. Keep in mind the default style of clustering is fully connected so there is some danger in managing that many network connections (quadratically scaling for each node added) during network partitions which can be a problem if you're not tuning things like TCP's TIME_WAIT for local networking conditions.

Even better, these days there are great libraries like partisan (https://github.com/lasp-lang/partisan) which can scale to much larger cluster sizes and can be dropped in for most distributed erlang uses cases w/o much effort.

In large Erlang applications, the "send to multiple peers" broadcast tendencies of its message passing can indeed cause the network to become a bottleneck; so can the default chattiness that a lot of Erlang application developers assume for their app behavior.

Fortunately, there are lots of simple-to-apply patterns and solutions when you get to that point, for example the excellent Partisan library: https://github.com/lasp-lang/partisan

Lasp use Partisan for its distribution, not the default erlang distribution.

It was built exactly to deal with a high level of churn, specifically for edge computing with spotty network. So yes it does.

https://github.com/lasp-lang/partisan

I, too, have a bit of experience with backpressure implementations in both Erlang and Pony. Erlang's "penalize the sender" doesn't work in many cases, so I'm not surprised that it's being removed. Erlang's remote distribution implementation & messaging semantics are Mostly Great but is definitely not Perfect.

1. Head-of-line blocking caused by congestion on the single TCP connection used for message transmission between any two Erlang nodes. This can cause major problems for apps control vs. data plane design, such as Riak and Lasp. Work on Partisan (https://github.com/lasp-lang/partisan) appears to be a substantial improvement.

2. If the single remote distribution TCP connection between two nodes is broken, then the first Erlang process to send a message to the remote node is, hrrm, well, borrowed/co-opted by the BEAM VM to connect the new TCP session. IIRC that process is marked unrunnable by the scheduler until the connection is set up or perhaps there's an error. If that process is really important to the functioning of your app, for example, an important system manager in the control plane of the app, then you have a very difficult-to-diagnose latency Heisenbug to cope with.

-Scott

Instead of rolling your own connection subsystem, there is also this https://github.com/lasp-lang/partisan Although they call it still experimental, it seems to be pretty actively developed.