Open Source Alternatives
Open source alternatives to LMDeploy
Open source alternatives to LMDeploy, ranked by GitHub stars and freshness.
7 open-source alternatives in the index, ranked by GitHub stars and freshness.
llama.cpp
Community
LLM inference in C/C++
Alternative to Vllm, Sglang, Tensorrt Llm +2 more
Best for: Developers building privacy-first or offline-capable applications with constrained hardware
vLLM
Community
A high-throughput and memory-efficient inference and serving engine for LLMs
Alternative to Tensorrt Llm, Lmdeploy, Sglang +1 more
Best for: Teams building production LLM APIs and services that need to maximize throughput and minimize latency under concurrent load.
FastChat
Community
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Alternative to Vllm, Sglang, Lmdeploy
Best for: Researchers and ML engineers building custom LLM applications who need training, serving, and evaluation in one framework.
SGLang
Community
SGLang is a high-performance serving framework for large language models and multimodal models.
Alternative to Vllm, Tensorrt Llm, Lmdeploy
Best for: Teams building production LLM services who need performance-optimized serving infrastructure
MNN-LLM
Community
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
Alternative to Llama Cpp, Tensorrt Llm, Lmdeploy
Best for: Developers building production on-device LLM and edge AI applications where latency and resource efficiency are critical.
TensorRT-LLM
Community
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NV
Alternative to Vllm, Lmdeploy, Sglang
Best for: Teams deploying LLMs at scale on NVIDIA infrastructure who need maximum inference performance.
OpenLLM
Community
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Alternative to Vllm, Sglang, Lmdeploy
Best for: Developers who need to serve open-source LLMs with OpenAI API compatibility