Open Source Alternatives
Open source alternatives to TensorRT-LLM
Open source alternatives to TensorRT-LLM, ranked by GitHub stars and freshness.
8 open-source alternatives in the index, ranked by GitHub stars and freshness.
llama.cpp
Community
LLM inference in C/C++
Alternative to Vllm, Sglang, Tensorrt Llm +2 more
Best for: Developers building privacy-first or offline-capable applications with constrained hardware
vLLM
Community
A high-throughput and memory-efficient inference and serving engine for LLMs
Alternative to Tensorrt Llm, Lmdeploy, Sglang +1 more
Best for: Teams building production LLM APIs and services that need to maximize throughput and minimize latency under concurrent load.
SGLang
Community
SGLang is a high-performance serving framework for large language models and multimodal models.
Alternative to Vllm, Tensorrt Llm, Lmdeploy
Best for: Teams building production LLM services who need performance-optimized serving infrastructure
MNN-LLM
Community
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
Alternative to Llama Cpp, Tensorrt Llm, Lmdeploy
Best for: Developers building production on-device LLM and edge AI applications where latency and resource efficiency are critical.
LMDeploy
Community
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Alternative to Vllm, Sglang, Tensorrt Llm +2 more
Best for: Developers who need to compress and serve LLMs efficiently in production
mistral.rs
Community
Fast, flexible LLM inference
Alternative to Llama Cpp, Vllm, Sglang +2 more
Best for: Rust developers seeking a fast, flexible LLM inference framework for performance-critical or resource-constrained environments.
FasterTransformer
Community
Transformer related optimization, including BERT, GPT
Alternative to Vllm, Tensorrt Llm
Best for: Developers seeking maximum inference performance for transformer models on NVIDIA hardware
MInference
Community
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to
Alternative to Vllm, Sglang, Tensorrt Llm +1 more
Best for: Developers optimizing long-context LLM inference on NVIDIA GPUs