Open source alternatives to TensorRT-LLM

Open source alternatives to TensorRT-LLM, ranked by GitHub stars and freshness.

4 open-source alternatives in the index, ranked by GitHub stars and freshness.

Community

A high-throughput and memory-efficient inference and serving engine for LLMs

Best for: Teams building production LLM APIs and services that need to maximize throughput and minimize latency under concurrent load.

Community

SGLang is a high-performance serving framework for large language models and multimodal models.

Best for: Teams building production LLM services who need performance-optimized serving infrastructure

Community

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Best for: Developers who need to compress and serve LLMs efficiently in production

Community

Transformer related optimization, including BERT, GPT

Best for: Developers seeking maximum inference performance for transformer models on NVIDIA hardware