What does HackerNews think of exllama?

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Language: Python

Ollama for Linux – Run LLMs on Linux with GPU Acceleration | Sep 2023
How does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G?

https://github.com/vllm-project/vllm

https://github.com/turboderp/exllama

https://github.com/turboderp/exllamav2

Ask HN: Cheapest hardware to run Llama 2 70B | Aug 2023
one 4090 + 3090-Ti

https://github.com/turboderp/exllama

Falcon LLM – A 40B Model | Jun 2023
For Falcon specifically, this is easy, it's embedded here: https://huggingface.co/blog/falcon#demo or you can access the demo here: https://huggingface.co/spaces/HuggingFaceH4/falcon-chat

I just tested both and it's pretty zippy (faster than AMD's recent live MI300 demo).

For llama-based models, recently I've been using https://github.com/turboderp/exllama a lot. It has a Dockerfile/docker-compose.yml so it should be pretty easy to get going. llama.cpp is the other easy one and the most recent updates put it's CUDA support only about 25% slower and generally is a simple `make` with a flag depending on which GPU you support you want and has basically no dependencies.

Also, here's a Colab notebook that should let shows you run up to 13b quantized models (12G RAM, 80G disk, Tesla T4 16G) for free: https://colab.research.google.com/drive/1QzFsWru1YLnTVK77itW... (for Falcon, replace w/ Koboldcpp or ctransformers)

AMD Expands AI Product Lineup with GPU-Only Instinct Mi300X with 192GB Memory | Jun 2023
Expand Context ↕
GPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.
Llama.cpp: Full CUDA GPU Acceleration | Jun 2023
Expand Context ↕
Not a 30 series, but on my 4090 I'm getting 32.0 tokens/s on a 13b q4_0 model (uses about 10GiB of VRAM) w/ full context (`-ngl 99 -n 2048 --ignore-eos` to force all layers on GPU memory and to do a full 2048 context). As a point of reference, currently exllama [1] runs a 4-bit GPTQ of the same 13b model at 83.5 tokens/s.

[1] https://github.com/turboderp/exllama