What does HackerNews think of serge?

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

Language: Python

#61 in Docker

#164 in Python

Google “We have no moat, and neither does OpenAI” | May 2023

>Until I can spin up a docker image capable of the same as OpenAI in hetzner for 30 bucks a month

I do exactly this with https://github.com/nsarrazin/serge

Hetzner will install any hardware you send them for $100. So you can send them a $200 P40 24GB to run 33B parameter GPU models at ChatGPT speeds without increasing your monthly cost.

SparseGPT: Language Models Can Be Accurately Pruned in One-Shot | May 2023

Expand Context ↕

Is Serge what you are looking for? I've been using it to play around with prompting a few large language models.

https://github.com/nsarrazin/serge

Ask HN: Are there any ready-to-use Docker images for running LLMs locally? | Apr 2023

I'm using Serge[0] as an API for a local Discord bot. You probably won't find anything for StableLM this soon after release, but this will download and run the Ll*ma stuff with a decent web UI.

[0] https://github.com/nsarrazin/serge