What does HackerNews think of Co-dfns?

High-performance, Reliable, and Parallel APL

Language: APL

This always seemed like a very interesting project; we need to get to the point where, if things can run in parallel, they must run in parallel to make software more efficient on modern cpu/gpu.

It won't attract funds, I guess, but it would be far more trivial to make this work with an APL or a Lisp/Scheme. There already is great research for APL[0] and looking at the syntax of HVM-core it seems it is rather easy to knock up a CL DSL. If only there were more hours in a day.

[0] https://github.com/Co-dfns/Co-dfns

There are many styles of APL, not just due to its long history, but also because APL is somewhat agnostic to architecture paradigms. You can see heavily imperative code with explicit branching all over the place, strongly functional-style with lots of small functions, even object-oriented style.

However, given the aesthetic that you express, I think you might like https://github.com/Co-dfns/Co-dfns/. This is hands-down my favorite kind of APL, in which the data flow literally follows the linear code flow.

Not that I am aware of. I think the closest project is co-dfns[1] which is being developed by Aaron Hsu (he did a presentation as well). It aims to compile a subset of APL so that it can be executed on GPUs for instance, possibly with other backends. I imagine an XLA backend could be possible there.

[1] https://github.com/Co-dfns/Co-dfns

Stories please! What did the closures do to you?

Hopefully this won't be seen as too combative, but I feel like there are a few people in the array community giving me some pretty strong conclusions that they don't really have the experience to back up (Aaron wrote[0] 17 lines of array compiler, and says the low-abstraction approach he used is the only way to develop sustainably. Cool. I wrote[1] 350 lines of array compiler following his style, and I disagree[2]). At the same time, my experience only goes so far (there's no way I would have invented the array style compiler!), and clearly you arrived at these conclusions somehow. So is there a chance you'd share the observations that led you that way?

On my end, I was actually introduced to a little object-oriented programming in J when Henry suggested using it for a simulation project. I used it, but I don't think I really got it—just a weird way to organize data. And then in college I had to learn objects-only Java. Not good. But later I worked some with Node.js, and its module system was pretty nice: no name conflicts, easy to share code! Some way into BQN development, I figured out (with some help from a Common Lisp programmer) a way to add modules with an APL-y syntax, and something magic happened. I got objects[3] too! I think I've done about as much OOP in BQN as anywhere else, and I feel like I understand it a lot better now.

So, this is my experience with Lisp-family features and APL. Fits like a glove, programming is easier and more fun. I mix and match array, functional, and object-oriented styles however I want. Did I lose coherence? When I translate my old J code it comes out shorter and cleaner and without exec (".) everywhere. But I still don't get why I should want the language I use to not support mutability rather than just default to immutability. Did I fail to understand something in J when I had the chance?

[0] https://github.com/Co-dfns/Co-dfns

[1] https://github.com/mlochbaum/BQN/blob/master/src/c.bqn

[2] https://mlochbaum.github.io/BQN/implementation/codfns.html

[3] https://mlochbaum.github.io/BQN/doc/oop.html

One of the most memorable and interesting things I've ever seen on HN was an AMA from the creator of the Co-dfns parallel APL compiler. Here is co-dfns [0] and the AMA by the creator [1]. The video may give you some insight and at the very least may be extremely interesting to hear the APL code explained (jump to around 13:30 for the beginning of the explanation behind the structure + what the compiler is doing). He explains it very well - it's quite easy to follow along.

[0] https://github.com/Co-dfns/Co-dfns

[1] https://www.youtube.com/watch?v=gcUWTa16Jc0

The changes to the IR data structures in this release are really neat- whole trees and graphs fit in a small fixed number of flat arrays. This saves on allocations, saves on total memory usage, and makes them trivial to serialize because there are no pointers.

I recently arrived at a similar design for manipulating NFAs as adjacency matrices, as a replacement for more pointer-y adjacency lists, by way of sparse matrix data structures. Rather than Zig's performance angle, I found that it made whole-graph operations much easier to implement and reuse.

I've also seen similar approaches from array languages like APL, for example this project for running this kind of stuff on GPUs: https://github.com/Co-dfns/Co-dfns

And, I've seen it in Rust as a "workaround" for the borrow checker, where that framing tends to make it feel like cheating or settling, which IMO is unfortunate since when people arrive at it for other reasons it seems to have a lot of other benefits!

There are probably more contexts I'm not familiar with- anyone have any good examples from domains they've worked in?

You might be interested in Aaron Hsu's work on co-dfns, which is a compiler that compiles a subset of Dyalog APL to the GPU.

https://github.com/Co-dfns/Co-dfns

A really interesting exploration is Co-dfns [1]. This is indeed a compiler that runs on the GPU, but it's also extremely out of the mainstream.

IMHO this is an area ripe for more exploration. If I were working on it, I might look to linking first before compilation, because the basic link task is more similar to what GPUs are good at (advanced stuff such as LTO is a different story, though).

[1]: https://github.com/Co-dfns/Co-dfns

I don't have the oomph to write a whole big thing this morning but uh, here are some links that are relevant IMO:

http://conal.net/papers/compiling-to-categories/

https://www.categoricaldata.net/

https://terminusdb.com/blog/ (It's a little market-ey, dig around for the meat. Graph DB in Prolog & Rust, "git for data", etc.)

https://github.com/Co-dfns/Co-dfns

edit: Ah it's you agumonkey (I don't check usernames before replying.) Apologies if I sound like a broken record in re: the above topics.

The source code to his data parallel compiler hosted on the GPU is 17 lines of APL!

The co-dfns source I've seen[1] is much longer. Do you know if there's a write-up somewhere of the differences between that and Appendix B from the dissertation?

1: https://github.com/Co-dfns/Co-dfns