Open Source Alternatives
Open source alternatives to vLLM
Open source alternatives to vLLM, ranked by GitHub stars and freshness.
9 open-source alternatives in the index, ranked by GitHub stars and freshness.
SGLang
Community
SGLang is a high-performance serving framework for large language models and multimodal models.
Best for: Teams building production LLM services who need performance-optimized serving infrastructure
TensorRT-LLM
Community
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NV
Best for: Teams deploying LLMs at scale on NVIDIA infrastructure who need maximum inference performance.
text-generation-inference
Community
Large Language Model Text Generation Inference
Best for: Developers needing a production-grade, self-hosted LLM serving solution.
LMDeploy
Community
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Best for: Developers who need to compress and serve LLMs efficiently in production
mistral.rs
Community
Fast, flexible LLM inference
Best for: Rust developers seeking a fast, flexible LLM inference framework for performance-critical or resource-constrained environments.
FasterTransformer
Community
Transformer related optimization, including BERT, GPT
Best for: Developers seeking maximum inference performance for transformer models on NVIDIA hardware
Shimmy
Community
⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.
Best for: Developers seeking a free, no-fuss Rust-based inference server with OpenAI API compatibility
Text-Embeddings-Inference
Community
A blazing fast inference solution for text embeddings models
Best for: Developers who need fast, scalable embedding serving for search or NLP pipelines
TGI
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Best for: Developers and teams who need to self-host or fine-tune open-source LLMs at scale