Tree-sitter is unfathomable to me. This is the grammar for Ruby:

https://github.com/tree-sitter/tree-sitter-ruby/blob/master/...

I find it absolutely amazing that a grammar for something as complicated as Ruby can be so concise. Less than a thousand lines. The corresponding Bison grammar is 13k lines. And I think the tree-sitter one is scannerless so also includes the lexer?! How do they do it?

This is more a function of Ruby than of tree-sitter. The tree-sitter grammars for other languages are hopefully less inscrutable. For Ruby, we basically just ported whitequark's parser [1] over to tree-sitter's grammar DSL and scanner API.

[1] https://github.com/whitequark/parser