Open Source Alternatives
Open source alternatives to vLLM
Open source alternatives to vLLM, ranked by GitHub stars and freshness.
20 open-source alternatives in the index, ranked by GitHub stars and freshness.
llama.cpp
Community
LLM inference in C/C++
Alternative to Vllm, Sglang, Tensorrt Llm +2 more
Best for: Developers building privacy-first or offline-capable applications with constrained hardware
FastChat
Community
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Alternative to Vllm, Sglang, Lmdeploy
Best for: Researchers and ML engineers building custom LLM applications who need training, serving, and evaluation in one framework.
SGLang
Community
SGLang is a high-performance serving framework for large language models and multimodal models.
Alternative to Vllm, Tensorrt Llm, Lmdeploy
Best for: Teams building production LLM services who need performance-optimized serving infrastructure
TensorRT-LLM
Community
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NV
Alternative to Vllm, Lmdeploy, Sglang
Best for: Teams deploying LLMs at scale on NVIDIA infrastructure who need maximum inference performance.
OpenLLM
Community
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Alternative to Vllm, Sglang, Lmdeploy
Best for: Developers who need to serve open-source LLMs with OpenAI API compatibility
text-generation-inference
Community
Large Language Model Text Generation Inference
Alternative to Vllm, Llama Cpp
Best for: Developers needing a production-grade, self-hosted LLM serving solution.
Triton Server (TRTIS)
Community
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Alternative to Vllm, Ollama
Best for: Teams deploying large-scale inference services that need high throughput and multi-framework support.
FlexGen
Community
Running large language models on a single GPU for throughput-oriented scenarios.
Alternative to Vllm
Best for: Developers who need to run large language models at high throughput on a single GPU, especially in budget-constrained or research environments
LMDeploy
Community
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Alternative to Vllm, Sglang, Tensorrt Llm +2 more
Best for: Developers who need to compress and serve LLMs efficiently in production
mistral.rs
Community
Fast, flexible LLM inference
Alternative to Llama Cpp, Vllm, Sglang +2 more
Best for: Rust developers seeking a fast, flexible LLM inference framework for performance-critical or resource-constrained environments.
FasterTransformer
Community
Transformer related optimization, including BERT, GPT
Alternative to Vllm, Tensorrt Llm
Best for: Developers seeking maximum inference performance for transformer models on NVIDIA hardware
Shimmy
Community
⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.
Alternative to Vllm, Llama Cpp
Best for: Developers seeking a free, no-fuss Rust-based inference server with OpenAI API compatibility
Text-Embeddings-Inference
Community
A blazing fast inference solution for text embeddings models
Alternative to Vllm, Sglang
Best for: Developers who need fast, scalable embedding serving for search or NLP pipelines
exllama
Community
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
Alternative to Llama Cpp, Vllm
Best for: Developers running quantized Llama models on resource-constrained hardware
Infinity
Community
Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
Alternative to Vllm, Sglang
Best for: Developers needing a fast, scalable open-source serving layer for embedding and reranking models in production.
Rapid-MLX
Community
The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Dr
Alternative to Ollama, Llama Cpp, Vllm
Best for: Developers on Apple Silicon who need a fast, local OpenAI-compatible inference engine for tool-calling and reasoning tasks.
ray-llm
Community
RayLLM - LLMs on Ray (Archived). Read README for more info.
Alternative to Vllm, Fastchat, Sglang
Best for: Developers already using Ray who need legacy code or patterns for running LLMs at scale.
MInference
Community
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to
Alternative to Vllm, Sglang, Tensorrt Llm +1 more
Best for: Developers optimizing long-context LLM inference on NVIDIA GPUs
Llama 3-8|70B
Community
[Llama 2-7 13 70B](https://llama.meta.com/llama2/)
Alternative to Vllm, Llama Cpp, Litgpt
Best for: Developers and researchers seeking a community-driven framework to deploy and customize Llama 3 models without proprietary dependencies.
TGI
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Alternative to Vllm, Sglang
Best for: Developers and teams who need to self-host or fine-tune open-source LLMs at scale