Embeddings-based search is a nice improvement on search, but it's still search. Relative to ChatGPT answering on its training data, I find embeddings-based search to be severely lacking. The right comparison is to traditional search, where it becomes favorable.

It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to.

Maybe with more context I'll change my tune, but it's very much at the whim of the context retrieval finding everything you need to answer the query. That's easy for stuff that search is already good at, and so provides a better interface for search. But it's hard for stuff that search isn't good at, because, well: it's search.

Many points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the answer (I still kinda feel like hundreds of agents might be).

But these optimizations are applications of technology stacks we already know about. Sometimes, this era of AI research reminds me of all the whacky contraptions from the era before building airplanes became an engineering discipline.

I would likely have tried building a backyard ornithopter powered by mining explosives, if I had been alive during that period of experimentation.

Prediction: the best interfaces for this will be the ones we use for everything else as humans. I am trying to approach it more like that, and less like APIs and “document vs relational vs vector storage”.