What does HackerNews think of WaveFunctionCollapse?

Have a good day!

https://news.ycombinator.com/item?id=34561910

[...stuff about SandSpielStudio and block visual programming languages like Scratch and Snap!, Long Now talk between Will Wright and Brian Eno about Cellular Automata, etc ...]

The other really cool rabbit hole to explore for generating tiles and even arbitrary graph based content (I'm sold: hexagons are the bestagons!) is "Wave Function Collapse", which doesn't actually have anything to do with quantum mechanics (it just sounds cool), but is actually a kind of constraint solver related to sudoku solvers.

https://escholarship.org/content/qt3rm1w0mn/qt3rm1w0mn_noSpl...

Maxim Gumin's work: https://github.com/mxgmn/WaveFunctionCollapse

Paul Merrell's work:

https://paulmerrell.org/model-synthesis/

https://paulmerrell.org/research/

Oskar Stålberg's work:

https://twitter.com/OskSta/status/784847588893814785

https://oskarstalberg.com/game/wave/wave.html

There's a way to define cellular automata rules by giving examples of the before and after patterns, and WFC is kind of like a statistical constraint solving version of that.

So it's really easy for artists to define rules just by drawing! Not even requiring any visual programming, but you can layer visual programming on top of it.

That's something that Alexander Repenning's "AgentSheets" supported (among other stuff): you could define cellular automata rules by before-and-after examples, wildcards and variables, and attach additional conditions and actions with a visual programming language.

AgentSheets and other cool systems are described in this classic paper: “A Taxonomy of Simulation Software: A work in progress” from Learning Technology Review by Kurt Schmucker at Apple. It covered many of my favorite systems.

http://donhopkins.com/home/documents/taxonomy.pdf

Chaim Gingold wrote a comprehensive "Gadget Background Survey" at HARC, which includes AgentSheets, Alan Kay's favorites: Rockey’s Boots and Robot Odyssey, and Chaim's amazing SimCity Reverse Diagrams and lots of great stuff I’d never seen before:

http://chaim.io/download/Gingold%20(2017)%20Gadget%20(1)%20S...

Chaim Gingold has analyzed the SimCity (classic) code and visually documented how it works, in his beautiful "SimCity Reverse Diagrams":

>SimCity reverse diagrams: Chaim Gingold (2016).

>These reverse diagrams map and translate the rules of a complex simulation program into a form that is more easily digested, embedded, disseminated, and and discussed (Latour 1986).

>The technique is inspired by the game designer Stone Librande’s one page game design documents (Librande 2010). If we merge the reverse diagram with an interactive approach—e.g. Bret Victor’s Nile Visualization (Victor 2013), such diagrams could be used generatively, to describe programs, and interactively, to allow rich introspection and manipulation of software.

>Latour, Bruno (1986). “Visualization and cognition”. In: Knowledge and Society 6 (1986), pp. 1– 40. Librande, Stone (2010). “One-Page Designs”. Game Developers Conference. 2010. Victor, Bret (2013). “Media for Thinking the Unthinkable”. MIT Media Lab, Apr. 4, 2013.

https://lively-web.org/users/Dan/uploads/SimCityReverseDiagr...

[... stuff about AgentSheets, KidSim, Lex Fridman interviews of Michael Levin and Steven Wolfram discussing Cellular Automata, CAM6 simulator, etc ...]

More:

https://news.ycombinator.com/item?id=34561910

As motivation, say you're an internet provider, providing internet service to a business. You naturally want to save money, so you perhaps want to compress packets before they go over the wire. Let's say the business you're providing service to also compresses their data, but they've made a mistake and do it inefficiently.

Let's say the business has, incorrectly, determined the probability distribution for their data to be $q(x)$. That is, they assign probability of seeing symbol $x$ to be $q(x)$. Let's say you've determined the "true" distribution to be $p(x)$. The entropy, or number of bits, they expect to transmit per packet/symbol will be $-\sum p(x) lg(q(x))$. Meaning, they'll compress their stream under the assumption that the distribution is $q(x)$ but the actually probability of seeing a packet, $x$, is $p(x)$, which is why the term $p(x) lg(q(x))$ shows up.

The number of bits you're transmitting is just $-\sum p(x) lg(p(x))$. Now we ask, how many bits, per packet, is the savings of your method over the businesses? This is $-\sum p(x) lg(q(x)/p(x))$, which is exactly the Kullback-Leibler divergence (maybe up to a sign difference).

In other words, given a "guess" at a distribution and the "true" distribution, how bad is it between them? This is the Kullback-Leibler distribution and why it shows up (I believe) in machine learning and fitness functions.

As a more concrete example, I just ran across a paper talking [0] about using WFC [1] to asses how well it, and other algorithms, do when trying to create generative "super mario brothers" like levels. Take a 2x2 or 3x3 grid, make a library of tiles, use that to generate a random level, then use the K-L divergence to determine how well your generative algorithm has done compared to the observed distribution from an example image.

[0] https://arxiv.org/pdf/1905.05077.pdf

[1] https://github.com/mxgmn/WaveFunctionCollapse

A few years ago I ran across Max Bittker's beautiful "Sandspiel", which is a delightful cellular automata toy that simulates sand and other rules:

https://sandspiel.club/

A few days ago I saw him tweet some amazing stuff that resonated with me, which then led me to discover what he's been working with Lu Wilson (TodePond): Sandspiel Studio -- user definable rules using a block based visual programming language.

https://twitter.com/maxbittker

"working on goth fungus kidpix":

https://twitter.com/maxbittker/status/1593868837111451649

Lu Wilson (TodePond):

https://twitter.com/TodePond

Sandspiel Studio:

https://studio.sandspiel.club/

Sandspiel introduction:

https://www.youtube.com/watch?v=ecCVor7mJ6o

Sandspiel Studio in 60 seconds:

https://www.youtube.com/watch?v=qOA-lR3Xc34

Rainbow Sand:

https://www.youtube.com/watch?v=PGTsy79wx4U

Huegene:

https://www.youtube.com/watch?v=ltpkO7jcFOY

Flower:

https://www.youtube.com/watch?v=ifyYITDq1oo

TodePond's Spellular Automata:

https://www.youtube.com/watch?v=xvlsJ3FqNYU

We had a great discussion on the Sandspiel Studio Discord server, where I posted some interesting links:

Check out the Long Now Foundation talk by Brian Eno and Will Wright about Generative Systems:

10:39 minute excerpt of Will talking about cellular automata and demonstrating with Mirak's Cellebration: https://www.youtube.com/watch?v=UqzVSvqXJYg

6:48 minute excerpt of Will demonstrating Spore Metaverses: https://www.youtube.com/watch?v=7t0kuvz5x_4

6:29 minute excerpt of Will demonstrating Spore Creature Demo: https://www.youtube.com/watch?v=8PXiNNXUUF8

Entire 1:38:50 hour Long Now Foundation talk: Playing with Time | Brian Eno and Will Wright https://www.youtube.com/watch?v=Dfc-DQorohc

Craig Reynolds said the name "Boids" was inspired by The Producers Concierge scene, so that's how you should pronounce it:

Boids. Dirty, disgusting, filthy, lice ridden Boids. Boids. You get my drift?

https://www.youtube.com/watch?v=aL6mTMShVyk

The other really cool rabbit hole to explore for generating tiles and even arbitrary graph based content (I'm sold: hexagons are the bestagons!) is "Wave Function Collapse", which doesn't actually have anything to do with quantum mechanics (it just sounds cool), but is actually a kind of constraint solver related to sudoku solvers.

https://escholarship.org/content/qt3rm1w0mn/qt3rm1w0mn_noSpl...

Maxim Gumin's work: https://github.com/mxgmn/WaveFunctionCollapse

Paul Merrell's work:

https://paulmerrell.org/model-synthesis/

https://paulmerrell.org/research/

Oskar Stålberg's work:

https://twitter.com/OskSta/status/784847588893814785

https://oskarstalberg.com/game/wave/wave.html

There's a way to define cellular automata rules by giving examples of the before and after patterns, and WFC is kind of like a statistical constraint solving version of that.

So it's really easy for artists to define rules just by drawing! Not even requiring any visual programming, but you can layer visual programming on top of it.

That's something that Alexander Repenning's "AgentSheets" supported (among other stuff): you could define cellular automata rules by before-and-after examples, wildcards and variables, and attach additional conditions and actions with a visual programming language.

AgentSheets and other cool systems are described in this classic paper: “A Taxonomy of Simulation Software: A work in progress” from Learning Technology Review by Kurt Schmucker at Apple. It covered many of my favorite systems.

http://donhopkins.com/home/documents/taxonomy.pdf

Chaim Gingold wrote a comprehensive "Gadget Background Survey" at HARC, which includes AgentSheets, Alan Kay's favorites: Rockey’s Boots and Robot Odyssey, and Chaim's amazing SimCity Reverse Diagrams and lots of great stuff I’d never seen before:

http://chaim.io/download/Gingold%20(2017)%20Gadget%20(1)%20S...

Chaim Gingold has analyzed the SimCity (classic) code and visually documented how it works, in his beautiful "SimCity Reverse Diagrams":

>SimCity reverse diagrams: Chaim Gingold (2016).

>These reverse diagrams map and translate the rules of a complex simulation program into a form that is more easily digested, embedded, disseminated, and and discussed (Latour 1986).

>The technique is inspired by the game designer Stone Librande’s one page game design documents (Librande 2010). If we merge the reverse diagram with an interactive approach—e.g. Bret Victor’s Nile Visualization (Victor 2013), such diagrams could be used generatively, to describe programs, and interactively, to allow rich introspection and manipulation of software.

>Latour, Bruno (1986). “Visualization and cognition”. In: Knowledge and Society 6 (1986), pp. 1– 40. Librande, Stone (2010). “One-Page Designs”. Game Developers Conference. 2010. Victor, Bret (2013). “Media for Thinking the Unthinkable”. MIT Media Lab, Apr. 4, 2013.

https://lively-web.org/users/Dan/uploads/SimCityReverseDiagr...

Agentsheets: Alexander Repenning (1993–)

Interacting agents are embedded and interact within cellular spaces called sheets. Agents are reactive to direct manipulation and have autonomous behavior. Agent Sheets draws upon a similarly spirited broad field of paradigms: artificial life, visual programming, “programmable drawing tools,” “simulation environments”, games, cellular automata, and “spreadsheet extensions.” Repenning draws upon these shared characteristics: visual, spatial notation, dynamic, direct manipulation, and incremental agency. The basic tool palette is also a gallery, defined in simulation terms.

Highlights:

• Kits (“agencies”) describe specific domains. One effect of “|ayered” design is “roles”—end-users vs. scenario designers. Example domains in thesis: Turing machines, circuits, flow, traffic, programs. • Sheet is a cellular 2d space, but agents can be stacked up in a cell. • Incremental refinement of art, behavior, etc… • A highly generalize idea of flow is used for things like neural nets, flow charts, water flow, circuits, system dynamic style models, and traffic. • It also supports ecological style spatial simulations. • User interaction and agent communication is in the same representation. i.e. Anything can do to one another everything the user can.

AgentSheets is still going, and has even gone all 3D like Minecraft! AgentCubes!

https://agentsheets.com/

Another great visual programming language for kids that supported defining cellular automata rules by example and visual programming:

KidSim (later Cocoa, then Stagecast Creator) Smith, David C., Allen Cypher, and James Spohrer (1994) In KidSim graphical simulations are created via graphical rewrite rules, which also enables a kind of programming by demonstration. The creators argue that most people can use editor GUIs (e.g. paint programs), and can give directions, but cannot program. Their solution is to “get rid of the programming language” in favor of a philosophy grounded in GUI design:

• Visibility. Relevant information is visible; causality is clear; modelessness. • Copy and modify, not make from scratch. • See and point, not remember and type. • Concrete, not abstract. • Familiar conceptual model. (“minimum translation distance”).

They choose a symbolic simulation microworld as a domain because it leads to knowing, ownership, and motivation. All objects are agents which have appearances, properties (name value pairs), and rules.

Programming by demonstration extends to using a calculator and dragging properties around to define conditionals. One of the creators of KidSim, David Smith, was also the creator of another graphical programming environment: Pygmalion.

Smith, David C., Allen Cypher, and James Spohrer (1994)

(Then I ran across TodePond's Spellular Automata video and realized I was preaching to the choir! TodePond wrote: "and stagecast creator is a big inspiration to me! I name-dropped it in a demo I did this week :D")

Wow I did not realize I was evangelizing to the choir! This video by TodePond, is exactly what I was talking about, just much more beautiful than I'd imagined possible:

https://www.youtube.com/watch?v=xvlsJ3FqNYU

I was watching Lex Fredman interview Michael Levin, and Lex expressed the same level of fascination about cellular automata that I have:

Michael Levin: Michael Levin: Biology, Life, Aliens, Evolution, Embryogenesis & Xenobots | Lex Fridman Podcast #325

https://www.youtube.com/watch?v=p3lsYlod5OU

1:44:29: Lex: Whenever I think about something like unconventional cognition, I think about cellular automata. I'm just captivated by the beauty of the thing. The fact that from simple little objects you can create such beautiful complexity that very quickly you forget about the individual objects and you see the things that it creates as its own organisms. That blows my mind every time. Like honestly I could full time just eat mushrooms and watch cellular automata. I don't even have to do mushrooms. Cellular automata, it feels like from the engineering perspective, I love when a very simple system captures something really powerful. Because then you can study that system to understand something fundamental about complexity, about life on earth. Anyway how Do I communicate with the thing? If a cellular automata can do cognition, if a plant can do cognition, if a xenobot can do cognition, how do I whisper in its ear and get an answer back, too? How do I have a conversation? How do I have a xenobot on the podcast?

Then I watched Lex interview Steven Wolfram:

https://www.youtube.com/watch?v=ez773teNFYA

Stephen Wolfram: Cellular Automata, Computation, and Physics | Lex Fridman Podcast #89 - YouTube

https://www.youtube.com

I hate it when a program I wrote mocks me. In Lex Fridman's interview of Steven Wolfram, he demonstrates the machine learning functions in Mathematica by taking a photo of himself, which identifies him as a .... (I won't give it away):

https://www.youtube.com/watch?v=ez773teNFYA&t=2h20m05s

Here's a video I recently recorded of the CAM-6 simulator I implemented decades ago, and rewrote in JavaScript a few years ago.

https://www.youtube.com/watch?v=LyLMHxRNuck

I recorded that demo to show to Norman Margolus, who co-wrote the book and wrote the CAM6 PC Forth code and many rules, so it's pretty long and technical and starts out showing lots of code, but I'm sure you'll totally get and appreciate it. I linked to a pdf copy of the book in the comments, as well as the source code and playable app.

Demo of Don Hopkins' CAM6 Cellular Automata Machine simulator.

Live App: https://donhopkins.com/home/CAM6

Github Repo: https://github.com/SimHacker/CAM6/

Javacript Source Code: https://github.com/SimHacker/CAM6/blob/master/javascript/CAM...

PDF of CAM6 Book: https://donhopkins.com/home/cam-book.pdf

Comments from the code:

    // This code originally started life as a CAM6 simulator written in C
    // and Forth, based on the original CAM6 hardware and compatible with
    // the brilliant Forth software developed by Toffoli and Margolus. But
    // then it took on a life of its own (not to mention a lot of other CA
    // rules), and evolved into supporting many other cellular automata
    // rules and image processing effects. Eventually it was translated to
    // C++ and Python, and then more recently it has finally been
    // rewritten from the ground up in JavaScript.
    // The CAM6 hardware and Forth software for defining rules and
    // orchestrating simulations is thoroughly described in this wonderful
    // book by Tommaso Toffoli and Norman Margolus of MIT.
    // Cellular Automata Machines: A New Environment for Modeling
    // Published April 1987 by MIT Press. ISBN: 9780262200608.
    // http://mitpress.mit.edu/9780262526319/

https://www.loom.com/share/cc09b916134b43cea201b61e71432f32

Making a Falling Sand Simulator:

https://news.ycombinator.com/item?id=31309616

Oh My Gosh, It's Covered in Rule 30s:

https://news.ycombinator.com/item?id=14458955

Wolfram Rule 30 Prizes:

https://news.ycombinator.com/item?id=21130098

Finding Mona Lisa in the Game of Life:

https://news.ycombinator.com/item?id=22552006

Andrew Wuensche's and Mike Lesser's gorgeous coffee table book, "The Global Dynamics of Cellular Automata":

https://news.ycombinator.com/item?id=22570750

Here's some stuff about Dave Ackley's "Robust First Computing" and "Moveable Feast Machine":

https://news.ycombinator.com/item?id=22304063

DonHopkins on Feb 11, 2020 | parent | context | favorite | on: Growing Neural Cellular Automata: A Differentiable...

Also check out the "Moveable Feast Machine", Robust-first Computing, and this Distributed City Generation example: https://news.ycombinator.com/item?id=21858577

DonHopkins on Oct 26, 2017 | parent | favorite | on: Cryptography with Cellular Automata (1985) [pdf]

A "Moveable Feast Machine" is a "Robust First" asynchronous distributed fault tolerant cellular-automata-like computer architecture. It's similar to a Cellular Automata, but it different in several important ways, for the sake of "Robust First Computing". These differences give some insight into what CA really are, and what their limitations are.

Cellular Automata are synchronous and deterministic, and can only modify the current cell: all cells are evaluated at once (so the evaluation order doesn't matter), so it's necessary to double buffer the "before" and "after" cells, and the rule can only change the value of the current (center) cell. Moveable Feast Machines are like asynchronous non-deterministic cellular automata with large windows that can modify adjacent cells.

Here's a great example with an amazing demo and explanation, and some stuff I posted about it earlier:

https://news.ycombinator.com/item?id=14236973

Robust-first Computing: Distributed City Generation:

https://www.youtube.com/watch?v=XkSXERxucPc

https://news.ycombinator.com/item?id=22304110

DonHopkins on Feb 11, 2020 | parent | context | favorite | on: Growing Neural Cellular Automata: A Differentiable...

Dave Ackley, who developed the Moveable Feast Machine, had some interesting thoughts about moving from 2D to 3D grids of cells:

https://news.ycombinator.com/item?id=21131468

DonHopkins 4 months ago | parent | favorite | on: Wolfram Rule 30 Prizes

Very beautiful and artistically rendered! Those would make great fireworks and weapons in Minecraft! From a different engineering perspective, Dave Ackley had some interesting things to say about the difficulties of going from 2D to 3D, which I quoted in an earlier discussion about visual programming:

https://news.ycombinator.com/item?id=18497585

David Ackley, who developed the two-dimensional CA-like "Moveable Feast Machine" architecture for "Robust First Computing", touched on moving from 2D to 3D in his retirement talk:

https://youtu.be/YtzKgTxtVH8?t=3780

"Well 3D is the number one question. And my answer is, depending on what mood I'm in, we need to crawl before we fly."

"Or I say, I need to actually preserve one dimension to build the thing and fix it. Imagine if you had a three-dimensional computer, how you can actually fix something in the middle of it? It's going to be a bit of a challenge."

"So fundamentally, I'm just keeping the third dimension in my back pocket, to do other engineering. I think it would be relatively easy to imaging taking a 2D model like this, and having a finite number of layers of it, sort of a 2.1D model, where there would be a little local communication up and down, and then it was indefinitely scalable in two dimensions."

"And I think that might in fact be quite powerful. Beyond that you think about things like what about wrap-around torus connectivity rooowaaah, non-euclidian dwooraaah, aaah uuh, they say you can do that if you want, but you have to respect indefinite scalability. Our world is 3D, and you can make little tricks to make toruses embedded in a thing, but it has other consequences."

Here's more stuff about the Moveable Feast Machine:

https://news.ycombinator.com/item?id=15560845

https://news.ycombinator.com/item?id=14236973

The most amazing mind blowing demo is Robust-first Computing: Distributed City Generation:

https://www.youtube.com/watch?v=XkSXERxucPc

And a paper about how that works:

https://www.cs.unm.edu/~ackley/papers/paper_tsmall1_11_24.pd...

Plus there's a lot more here:

https://movablefeastmachine.org/

Now he's working on a hardware implementation of indefinitely scalable robust first computing:

https://www.youtube.com/channel/UC1M91QuLZfCzHjBMEKvIc-A

John von Neumann's 29 State Cellular Automata

https://news.ycombinator.com/item?id=22304084

John von Neumann's book "Theory of Self-Reproducing Automata":

https://news.ycombinator.com/item?id=21858465

Factorio, and Will Wright on simulation games:

https://news.ycombinator.com/item?id=24163024

[0] https://github.com/mxgmn/WaveFunctionCollapse

Similar idea without neural networks:

https://github.com/mxgmn/WaveFunctionCollapse

I'm surprised no one applied this idea to Minecraft yet. There's even a 3D version:

https://i.imgur.com/IEOsbIy.png

In the in the original WFC(wave function collapse)[0] versions it always used an example of a completed output and generated all valid tile combinations from that. For a game this means that the developer can use hand made maps as examples for it to create new plausible maps. In the common vernacular WFC doesn't use this step and just directly either generates what combinations work or the devs manually decide what tiles can be next to others. Perhaps only the combination of both steps should be considered WFC.

Just to guess I think this is probably a case of it being developed independently in different fields. It's good to have another name for it as I always hated it being called "wave function collapse" when that is a physics thing.

[0]https://github.com/mxgmn/WaveFunctionCollapse

0. This neural thing, of course, to create landscape-like 2D projections of a plausible scene.

1. Wave-function collapse models that synthesize domain data quite nicely when parametrized with artistic care - this is a "simpler" example of the concept. https://github.com/mxgmn/WaveFunctionCollapse

2. Fairly good understanding how to synthesize terrain. Terragen is a good example of this (although not public research, the images drive the point home nicely) https://planetside.co.uk/

So, we could use the source image from this as a 2D projection of an intended landscape as a seed to a wave-function collapse model that would use known terrain parametrization schemes to synthesize something usable (so basically create a Terragen equivalent model).

I think that's it plausibly more or less. But it's a "research" level problem still, I think, not something one can cook up by chaining the data flow from a few open source libraries together.

GitHub page (demo in Unity) https://github.com/marian42/wavefunctioncollapse

Original Wave Function Collapse Algorithm https://github.com/mxgmn/WaveFunctionCollapse

[1]: https://github.com/mxgmn/WaveFunctionCollapse [2]: https://www.boristhebrave.com/2018/04/28/random-paths-via-ch... [3]: https://boristhebrave.github.io/DeBroglie/articles/features....