vLLM
by Community
A high-throughput and memory-efficient inference and serving engine for LLMs
OSS
vLLM
Added 1 June 2026
Overview
vLLM is a Python framework for serving large language models with optimized throughput and memory efficiency. It uses techniques like paged attention and continuous batching to reduce latency and increase request throughput compared to standard inference servers. Designed for production deployments that need to handle multiple concurrent requests.
Best for
Best for
Teams building production LLM APIs and services that need to maximize throughput and minimize latency under concurrent load.
Use cases
- Running inference servers that handle high request volume with low latency
- Reducing GPU memory footprint when serving large models
- Batching and scheduling inference requests efficiently
Notes
vLLM is a Python framework for serving large language models with optimized throughput and memory efficiency. It uses techniques like paged attention and continuous batching to reduce latency and increase request throughput compared to standard inference servers. Designed for production deployments that need to handle multiple concurrent requests.
81,619 stars on GitHub. Last updated 2026-06-01. Licensed Apache-2.0.
Use cases
- Running inference servers that handle high request volume with low latency
- Reducing GPU memory footprint when serving large models
- Batching and scheduling inference requests efficiently
Pros
- Significantly higher throughput than standard LLM serving approaches
- Lower memory consumption enables serving larger models on same hardware
- Active community with 81k+ GitHub stars and ongoing development
Cons
- Requires Python and GPU infrastructure, not suitable for CPU-only deployments
- Steeper learning curve than simple inference libraries for basic use cases
- Performance gains depend on workload characteristics and batch patterns
Indexed from awesome-llm and enriched against its public facts.
Pros
- Significantly higher throughput than standard LLM serving approaches
- Lower memory consumption enables serving larger models on same hardware
- Active community with 81k+ GitHub stars and ongoing development
Cons
- Requires Python and GPU infrastructure, not suitable for CPU-only deployments
- Steeper learning curve than simple inference libraries for basic use cases
- Performance gains depend on workload characteristics and batch patterns
Open-source & AI alternatives
Swap-in tools that solve the same job. Weigh the trade-offs before you commit.
TensorRT-LLM
Community
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NV
LMDeploy
Community
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
SGLang
Community
SGLang is a high-performance serving framework for large language models and multimodal models.
OpenLLM
Community
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
exllama
Community
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
FastChat
Community
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
FasterTransformer
Community
Transformer related optimization, including BERT, GPT
FlexGen
Community
Running large language models on a single GPU for throughput-oriented scenarios.
Infinity
Community
Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
Llama 3-8|70B
Community
[Llama 2-7 13 70B](https://llama.meta.com/llama2/)
llama.cpp
Community
LLM inference in C/C++
LMDeploy
Community
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
MInference
Community
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to
mistral.rs
Community
Fast, flexible LLM inference
OpenLLM
Community
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Rapid-MLX
Community
The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Dr
ray-llm
Community
RayLLM - LLMs on Ray (Archived). Read README for more info.
SGLang
Community
SGLang is a high-performance serving framework for large language models and multimodal models.
Shimmy
Community
⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.
TensorRT-LLM
Community
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NV
Text-Embeddings-Inference
Community
A blazing fast inference solution for text embeddings models
text-generation-inference
Community
Large Language Model Text Generation Inference
TGI
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Triton Server (TRTIS)
Community
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
inference.sh
ac.inference.sh
Run 150+ AI apps — image, video, audio, LLMs, 3D and more. Browse, execute, stream results.
CodeQwen1.5-7B
Community
GITHUB HUGGING FACE MODELSCOPE DEMO DISCORD Introduction The advent of advanced programming tools, which harnesses the power of large language models (LLMs), has significantly en
Command R-35B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Dify
Community
Production-ready platform for agentic workflow development.
distilabel
Community
Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
DSPy
Stanford NLP
Programming, not prompting. Declare what you want, compile prompts and weights against an objective.
Falcon 40B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
FlagAI
Community
FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.
GPUStack
Community
A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.
Intelli
Community
Build multi-model chatbots and agents from intent.
LangChain
Community
The agent engineering platform.
Llama 3-8|70B
Community
[Llama 2-7 13 70B](https://llama.meta.com/llama2/)
LLMKube
Community
Kubernetes operator for local LLM inference with llama.cpp, vLLM, TGI, and mlx-server — multi-GPU NVIDIA + Apple Silicon Metal, autoscaling, air-gapped, production-ready
lm-evaluation-harness
Community
A framework for few-shot evaluation of language models.
LMQL
Community
Language Model Query Language
MLflow
Community
MLflow - Open Source AI Platform for Agents, LLMs & Models
Modelz-LLM
Community
OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)
Nemotron-4-340B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
OpenModelZ
Community
Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
OpenRLHF
Community
An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)
Rigging
Community
Lightweight LLM Interaction Framework
ROLL
Community
An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models
Scale Spellbook
Community
Accelerate and scale Generative AI across your enterprise with the platform to transform your data into customized enterprise-ready Generative AI applications.
State of GPT
Community
Go deep on real code and real systems with the teams building and scaling AI at Microsoft Build, June 2–3, 2026, in San Francisco and online.
TensorZero
Community
TensorZero builds open-source tools for production-grade LLM applications: LLM gateway, observability, optimization, evaluations, and experimentation.
Tune Studio
Community
Playground for devs to finetune & deploy LLMs
veRL
Community
verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework
Forefront
Various
Forefront is a platform to fine-tune and inference open-source-language-models.
Harbor
Various
Stop configuring your AI stack. Start using it. One command brings a complete pre-wired LLM stack with hundreds of services to explore.
Together AI
Various
Build what's next on the AI Native Cloud. Full-stack AI platform for inference, fine-tuning, and GPU clusters — powered by cutting-edge research.
Jwrede/llmprobe
Various
Synthetic monitoring and CI smoke tests for LLM inference endpoints.
Agency
Community
🕵️♂️ Library designed for developers eager to explore the potential of Large Language Models (LLMs) and other generative AI through a clean, effective, and Go-idiomatic approach.
AgentField
Community
Build, run and scale AI agents like API and microservices - observable,auditable and identity-aware from day one.
ai-evaluation
Community
Evaluation Framework for all your AI related Workflows
AI Gateway
Community
A blazing fast AI Gateway with integrated guardrails. Route to 1,600+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
AutoGen
Microsoft
Microsoft's framework for multi-agent conversations. Agents that talk to each other to solve hard problems.
Awesome-Code-LLM
Community
👨💻 An awesome and curated list of best code-LLM for research.
awesome-japanese-llm
Community
日本語LLMまとめ - Overview of Japanese LLMs
Awesome-LLM-Inference
Community
📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism, etc. 🎉🎉
Axolotl
Community
Go ahead and axolotl questions
Baichuan-7|13B
Community
AGI Large Language Models
BELLE
Community
BELLE: Be Everyone's Large Language model Engine（开源中文对话大模型）
BentoML
Community
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Bifrost
Community
Fastest enterprise AI gateway (50x faster than LiteLLM) with adaptive load balancer, cluster mode, guardrails, 1000+ models support & <100 µs overhead at 5k RPS.
BLOOMZ&mT0
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Chainlit
Community
A Python library for making chatbot interfaces.
Codestral-7|22B
Community
The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.
CompassRank
Community
评测榜单旨在为大语言模型和多模态模型提供全面、客观且中立的得分与排名，同时提供多能力维度的评分参考，以便用户能够更全面地了解大模型的能力水平。
Continue
Community
⏩ Source-controlled AI checks, enforceable in CI. Powered by the open-source Continue CLI
Datatrove
Community
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
DeepSeek-R1
Community
First-generation reasoning models from DeepSeek.
DeepSeek-Math-7B
Community
DeepSeek Math series
DeepSeek-V2.5
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
DeepSeek-VL-1.3|7B
Community
DeepSeek-VL model series
dolly
Community
Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform
EasyEdit
Community
[ACL 2024] An Easy-to-use Knowledge Editing Framework for LLMs.
FlagAI
Community
FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.
Flyflow
Community
Open source, high performance fine tuning as a service for GPT4 quality models with 5x lower latency and 3x lower cost
Gemma
Community
Checking your browser - reCAPTCHA
Gemma2-9|27B
Community
Gemma 2, our next generation of open models, is now available globally for researchers and developers.
GLM-2|6|10|13|70B
Community
Org profile for THUDM on Hugging Face, the AI community building the future.
Google "We Have No Moat, And Neither Does OpenAI"
Community
Leaked Internal Google Document Claims Open Source AI Will Outcompete Google and OpenAI
Grok-1-314B-MoE
Community
Grok-1-314B-MoE — indexed from awesome-llm
Guidance
Community
A guidance language for controlling large language models.
Haystack
Community
Create agentic, context engineered AI systems using Haystack’s modular and customizable building blocks, built for real-world, production-ready applications.
InternLM-XComposer2-1.8|7B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
InternLM2-1.8|7|20B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Kaito
Community
Kubernetes AI Toolchain Operator
Kimi-K2
Community
Kimi K2 is the large language model series developed by Moonshot AI team
Kserve
Community
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
KubeAI
Community
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
LazyLLM
Community
Easiest and laziest way for building multi-agent LLMs applications.
lighteval
Community
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
LiteLLM 🚅
Community
Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, Vertex
Llama 3.2-1|3|11|90B
Community
[Llama 3.1-8 70 405B](https://llama.meta.com/)
Litgpt
Community
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Llama 3-8|70B
Community
[Llama 2-7 13 70B](https://llama.meta.com/llama2/)
LLaMA Cult and More
Community
Large Language Models for All, 🦙 Cult and More, Stay in touch !
LlamaIndex
LlamaIndex
The data framework for LLM apps. RAG, ingestion, structured extraction, agents over your data.
llm-ui
Community
The React library for LLMs
LMQL
Community
Language Model Query Language
Megatron-DeepSpeed
Community
Ongoing research training transformer language models at scale, including: BERT & GPT-2
MiniCPM-2B
Community
The MiniCPM family of LLMs and VLLMs.
Mixtral-8x7B
Community
The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.
MLflow
Community
MLflow - Open Source AI Platform for Agents, LLMs & Models
Moby
moby
The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
ModelFusion
Community
The TypeScript library for building AI applications.
Moonlight-A3B
Community
Moonshot's Compute-efficient MoE LLM, first Scaling Up of Muon Optimizer
MPT-7B
Community
Introducing MPT-7B, the first entry in our MosaicML Foundation Series. MPT-7B is a transformer trained from scratch on 1T tokens of text and code. It is open source, available fo
OLMo-7B
Community
Artifacts for the first set of OLMo models.
open-r1
Community
Fully open reproduction of DeepSeek-R1
Open Responses
Community
OpenAGI
Community
OpenAGI: When LLM Meets Domain Experts
OpenELM-1.1|3B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
OpenLIT
Community
Open source platform for AI Engineering: OpenTelemetry-native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground. 🚀💻 Integrates with
Outlines
Community
Structured Outputs
PAI
Community
Resource scheduling and cluster management for AI
Phi1-1.3B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Phi3-3.8|7|14B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Puzzlet AI
Community
Redirecting...
Qwen-1.8B|7B|14B|72B
Community
Qwen - a Qwen Collection
Qwen2-0.5B|1.5B|7B|57B-A14B-MoE|72B
Community
GITHUB HUGGING FACE MODELSCOPE DEMO DISCORD Introduction After months of efforts, we are pleased to announce the evolution from Qwen1.5 to Qwen2. This time, we bring to you: Pret
Qwen2.5-1M-7|14B
Community
Tech Report HuggingFace ModelScope Qwen Chat HuggingFace Demo ModelScope Demo DISCORD Introduction Two months after upgrading Qwen2.5-Turbo to support context length up to one mi
Qwen2.5-Max
Community
QWEN CHAT API DEMO DISCORD It is widely recognized that continuously scaling both data size and model size can lead to significant improvements in model intelligence. However, th
Qwen2-Audio-7B
Community
DEMO PAPER GITHUB HUGGING FACE MODELSCOPE DISCORD To achieve the objective of building an AGI system, the model should be capable of understanding information from different moda
Qwen2-Math-1.5B|7B|72B
Community
GITHUB HUGGING FACE MODELSCOPE DISCORD 🚨 This model mainly supports English. We will release bilingual (English and Chinese) math models soon. Introduction Over the past year, w
R2R
Community
SoTA production-ready AI retrieval system. Agentic Retrieval-Augmented Generation (RAG) with a RESTful API.
RecurrentGemma-2B
Community
Open weights language model from Google DeepMind, based on Griffin.
ROLL
Community
An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models
RWKV-v4|5|6
Community
Org profile for RWKV on Hugging Face, the AI community building the future.
simple-evals
Community
Eval tools by OpenAI.
SkyPilot
Community
Run, manage, and scale AI workloads on any AI infrastructure. Use one system to access & manage all AI compute (Kubernetes, Slurm, 20+ clouds, on-prem).
StableLM-v2-12B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
StableLM-3B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
StarCoder-1|3|7B
Community
All models, datasets, and demos related to StarCoder!
torchtune
Community
PyTorch native post-training library
TRL
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Tune Studio
Community
Playground for devs to finetune & deploy LLMs
unslothai
Community
Unsloth Studio is a web UI for training and running open models like Gemma 4, Qwen3.6, DeepSeek, gpt-oss locally.
VisCPM-10B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Weights & Biases
Community
W&B Weave helps developers evaluate, monitor, and iterate continuously to deliver generative AI applications with confidence.
Yi-34B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Yi-VL-6B|34B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
DeepSeek
Various
Org profile for DeepSeek on Hugging Face, the AI community building the future.
Grok
Various
An LLM by xAI with [open source](https://github.com/xai-org/grok-1) and open weights. #opensource
LibreChat
Various
LibreChat brings together all your AI conversations in one unified, customizable interface.
LLaMA
Various
Llama LLM, a foundational, 65-billion-parameter large language model by Meta. Meta, February 23rd, 2023. #opensource
LLM Stats
Various
The LLM Leaderboard — independent ranking of GPT, Claude, Gemini, Llama, DeepSeek and 300+ AI models by intelligence, speed and price. Composite LLM Stats Score updated continuou
Open LLMs
Various
📋 A list of open LLMs available for commercial use.
Qwen
Various
Qwickly forging AGI, enhancing intelligence.
NeMo
Various
A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.