What does HackerNews think of serving?

This project gained popularity due to the HIGH DEMAND for running large models with 1B+ parameters, like `llama`. Python dominates the interface and training ecosystem, but prior to llama.cpp, non-ML professionals showed little interest in a fast C++ interface library. While existing solutions like tensorflow-serving [1] in C++ were sufficiently fast with GPU support, llama.cpp took the initiative to optimize for CPU and trim unnecessary code, essentially code-golfing and sacrificing some algorithm correctness for improved performance, which isn't favored by "ML research".

NOTE: In my opinion, a true pioneer was DarkNet, which implemented the YOLO model series and significantly outperformed others [2]. Same trick basically like llama.cpp

[1] https://github.com/tensorflow/serving [2] https://github.com/pjreddie/darknet

https://thenewstack.io/tutorial-deploying-tensorflow-models-...

Meanwhile, here is a list of open source ML deployment packages:

https://github.com/oracle/graphpipe

https://github.com/eliorc/denzel

https://github.com/tensorflow/serving

https://github.com/ucbrise/clipper

https://github.com/DLHub-Argonne/dlhub_sdk

https://github.com/kubeflow/pipelines