What does HackerNews think of clownfish?

Constrained Decoding for LLMs against JSON Schema

Language: Python

Show HN: LLMs can generate valid JSON 100% of the time | Aug 2023
https://github.com/newhouseb/clownfish

Which I've been using for a while now, also restricts the sampling space to force correct generation, but does so as the result of a different process than yours.

Show HN: LLMs can generate valid JSON 100% of the time | Aug 2023
I'm not sure how this is different than:

https://github.com/1rgs/jsonformer

or

https://github.com/newhouseb/clownfish

or

https://github.com/mkuchnik/relm

or

https://github.com/ggerganov/llama.cpp/pull/1773

or

https://github.com/Shopify/torch-grammar

Overall there are a ton of these logit based guidance systems, the reason they don't get tons of traction is the SOTA models are behind REST APIs that don't enable this fine-grained approach.

Those models perform so much better that people generally settle for just re-requesting until they get the correct format (and with GPT-4 that ends up being a fairly rare occurrence in my experience)

ReLLM: Exact Structure for Large Language Model Completions | May 2023
This seems extremely similar to https://github.com/newhouseb/clownfish
BlenderGPT: Use commands in English to control Blender with OpenAI's GPT-4 | Mar 2023
Expand Context ↕
We also use GPT to perform actions in the software I build at work and we hit the same issue of inconsistency which lead me down a long rabbit hole to see if I could force an LLM to only emit grammatically correct output that follows a bespoke DSL (a DSL ideally safer and more precise than just eval'ing random AI-produced Python).

I just finished writing up a long post [1] that describes how this can work on local models. It's a bit tricky to do via API efficiently, but hopefully OpenAI will give us the primitives one day to appropriately steer these models to do the right things (tm).

[1] https://github.com/newhouseb/clownfish