What does HackerNews think of crdt-benchmarks?

A collection of CRDT benchmarks

Language: JavaScript

Author of Yjs here. I'm all for faster data structures. But only benchmarking one dimension looks quite fishy to me. A CRDT needs to be adequate at multiple dimensions. At least you should describe the tradeoffs in your article.

The time to insert characters is the least interesting property of a CRDT. It doesn't matter to the user whether a character is inserted within .1ms or .000000001ms. No human can type that fast.

It would be much more interesting to benchmark the time it takes to load a document containing X operations. Yjs & Yrs are pretty performant and conservative on memory here because they don't have to build an index (it's a tradeoff that we took consciously).

When benchmarking it is important to measure the right things and interpret the results somehow so that you can give recommendations when to use your algorithm / implementation. Some things can't be fast/low enough (e.g. time to load a document, time to apply updates, memory consumption, ..) other things only need to be adequate (e.g. time to insert a character into a document).

Unfortunately, a lot of academic papers set a bad trend of only measuring one dimension. Yeah, it's really easy to succeed in one dimension (e.g. memory or insertion-time) and it is very nice click-bait. But that doesn't make your CRDT a viable option in practice.

I maintain a set of benchmarks that tests multiple dimensions [1]. I'd love to receive a PR from you.

[1]: https://github.com/dmonad/crdt-benchmarks

So yes, a particularly large document is not the norm but it can happen.

JavaScript CRDTs can be quite performant, see the Yjs benchmarks: https://github.com/dmonad/crdt-benchmarks

If you are interested in CRDTs, take a look at Yjs[0] by Kevin Jahns too, it has bindings for the most popular rich text editors but is also brilliant for general data syncing too. It seems to to more performant than Automerge [1]. For an example of using it for shared state take a look at SyncedStore. [2][3].

I think Yjs probably has more potential than Automerge, the community around it is incredible. They have created an Open Collective to fund development of the connected projects[4].

0: https://github.com/yjs/yjs

1: https://github.com/dmonad/crdt-benchmarks

2: https://syncedstore.org/docs/

3: https://news.ycombinator.com/item?id=29483913

4: https://opencollective.com/y-collective

https://github.com/dmonad/crdt-benchmarks

Not completely up to date I believe but it shows that Yjs significantly outperforms automerge.

Hi! Great questions :)

First of all, SyncedStore does not implement any CRDT algorithms. Credits for this go to Yjs [1] (and its author Kevin), which it uses as underlying CRDT.

Yjs and Automerge are (afaik) the two most commonly used CRDT implementations. Both have their pros and cons, but Yjs has focused a lot on performance [2].

Automerge has a bit friendlier "Immer style" [3] API. I'm not too familiar with @localfirst/state, but it seems to add a Redux style API on top of Automerge.

My approach with SyncedStore was really to provide an API on top of Yjs that's as simple as possible to use in React / Vue / Svelte or plain JS app. I.e.: only use a single React Hook to observe changes, and use regular Javascript assigments to update values. The API is inspired mostly by Reactive Programming libraries such as MobX [4] (from the same author as Immer).

Hope you're still following along :) Maybe it helps to compare the TODO-MVC applications, as both SyncedStore (https://github.com/YousefED/SyncedStore/tree/main/examples) and @localfirst/state (https://github.com/local-first-web/state/tree/main/examples/...) have implemented these as examples!

[1]: https://github.com/yjs/yjs [2]: https://github.com/dmonad/crdt-benchmarks [3]: https://github.com/immerjs/immer [4]: https://mobx.js.org/

I know that it is hard to comprehend why modern CRDT implementations are fast. But the data confirms that they work great. OT seems to be much simpler, but there are real advantages in using CRDTs. The performance problems have been solved through an efficient representation of the CRDT model.

The gist of the below [1] read is that it is impossible for a human to create a document that Yjs can't handle (even in the worst case scenario). But yes, it handles real-world scenarios particularily well.

The concept of "hidden classes" is super old. It has first been implemented in a fork of smalltalk and then became foundational concept of runtime engines for scripting languages. It is implemented in V8, python, ruby, spidermonkey, ..

Yjs does not assume a "real-world scenario" and it is not optimized for any specific runtime engine. It runs fast in any browser. The benchmarks confirm this. [2]

Yjs is being used in practice by several companies (eg Nimbus Notes with >3 million users) for quite some time now. I'm not aware of any performance problems.

[1]: https://blog.kevinjahns.de/are-crdts-suitable-for-shared-edi... [2]: https://github.com/dmonad/crdt-benchmarks

Cool! It'd be interesting to see those CRDT implementations added to Kevin Jahns' CRDT Benchmarks page[1]. The LogootSplit paper looks interesting. It looks like xray is abandoned, and I'm not sure about teletype. Though teletype's CRDT looks to be entirely implemented in javascript[2]? If the authors are around I'd love to see some benchmarks so we can compare approaches and learn what actually works well.

And I'm not surprised these techniques have been invented before. Realising a tree is an appropriate data structure here is a pretty obvious step if you have a mind for data structures.

To name it, I often find myself feeling defensive when people read my work and respond with a bunch of links to academic papers. Its probably totally unfair and a complete projection from my side, but I hear a voice in my head reword your comment to instead say something awful like: "Cool, but everything you did was done before. Even if they didn't make any of their work practical, usable or good they still published first and you obviously didn't do a good enough literature review if you didn't know that." And I feel an unfair defensiveness arise in me as a result that wants to find excuses to dismiss the work, even if the work might be otherwise interesting.

Its hard to compare their benchmark results because they used synthetic randomized editing traces, which always have different performance profiles than real edits for this stuff. Their own university gathered some great real world data in an earlier study. It would have been much more instructive if that data set was used here. At a glance their RAM usage looks to be about 2 orders of magnitude worse than diamond-types or yjs. And their CPU usage... ?? I can't tell because they have no tables of results. Just some hard to read charts with log scales, so you can't even really eyeball the figures. So its really hard to tell if their work ends up performance-competitive without spending a couple days getting their enterprise style java code running with a better data set. Do you think thats worth doing?

[1] https://github.com/dmonad/crdt-benchmarks

[2] https://github.com/atom/teletype-crdt