llama.cpp
by Community
LLM inference in C/C++
OSS
llama.cpp
Added 1 June 2026
Overview
llama.cpp is a C++ inference framework that runs large language models locally on consumer hardware. It provides optimized tensor operations and quantization support to reduce model size and memory footprint, enabling fast inference without cloud dependencies.
Best for
Best for
Developers building privacy-first or offline-capable applications with constrained hardware
Use cases
- Running open-source LLMs on laptops or edge devices
- Building offline AI applications with minimal latency
- Quantizing and deploying models with reduced VRAM requirements
Notes
llama.cpp is a C++ inference framework that runs large language models locally on consumer hardware. It provides optimized tensor operations and quantization support to reduce model size and memory footprint, enabling fast inference without cloud dependencies.
114,160 stars on GitHub. Last updated 2026-06-01. Licensed MIT.
Use cases
- Running open-source LLMs on laptops or edge devices
- Building offline AI applications with minimal latency
- Quantizing and deploying models with reduced VRAM requirements
Pros
- Minimal dependencies and fast startup, runs on CPU and GPU
- Extensive quantization options (4-bit, 8-bit) dramatically reduce model size
- Active community with broad hardware support including Apple Silicon
Cons
- Requires manual model conversion and quantization workflows
- Performance varies significantly by hardware, CPU inference is slower than GPU alternatives
- Limited built-in abstractions for complex multi-model pipelines
Indexed from awesome-llm and enriched against its public facts.
Pros
- Minimal dependencies and fast startup, runs on CPU and GPU
- Extensive quantization options (4-bit, 8-bit) dramatically reduce model size
- Active community with broad hardware support including Apple Silicon
Cons
- Requires manual model conversion and quantization workflows
- Performance varies significantly by hardware, CPU inference is slower than GPU alternatives
- Limited built-in abstractions for complex multi-model pipelines
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
ollama
Community
Get up and running with Kimi-K2.5, GLM-5, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
Open WebUI
Various
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
gpt4all
Various
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
gpt-migrate
Community
Easily migrate your codebase from one language or framework to another.
memfree
Community
Welcome to the MemFree documentation.
openinterpreter
Community
Interpreter lets you work alongside agents that can edit your documents, fill PDF forms, and more.
bgauryy/octocode-mcp
Various
MCP server for semantic code research and context generation on real-time using LLM patterns | Search naturally across public & private repos based on your permissions | Transform
dcostenco/prism-mcp
Various
The Mind Palace for AI Agents - HIPAA-hardened Cognitive Architecture with on-device LLM (prism-coder:7b), Hebbian learning, ACT-R spreading activation, adversarial evaluation, per
ShipItAndPray/mcp-turboquant
Various
MCP server for LLM quantization. Compress any model to GGUF/GPTQ/AWQ in one tool call. First MCP server for model compression.
Anything LLM
Community
The all-in-one AI productivity accelerator. On device and privacy first with no annoying setup or configuration.
Codestral-7|22B
Community
The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.
deploy-llms-with-ansible
Community
Easily deploy LLMs with Ansible. Uses Docker with llama.cpp or ollama. Secured with whitelisted IPs.
FastChat
Community
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
fauxpilot
Community
FauxPilot - an open-source alternative to GitHub Copilot server
LiteChain
Community
Build robust LLM applications with true composability 🔗
LLama Cpp Agent
Community
The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM models, execute structured function calls a
LLMKube
Community
Kubernetes operator for local LLM inference with llama.cpp, vLLM, TGI, and mlx-server — multi-GPU NVIDIA + Apple Silicon Metal, autoscaling, air-gapped, production-ready
Local GPT
Community
Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.
Off Grid
Community
The Swiss Army Knife of Offline AI. Chat, Speak, and Generate Images - Privacy First, Zero Internet. Download an LLM and use it on your mobile device. No data ever leaves your phon
OpenLLM
Community
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Pipecat
Community
Open Source framework for voice and multimodal conversational AI
Private GPT
Community
Interact with your documents using the power of GPT, 100% privately, no data leaks
QA-Pilot
Community
QA-Pilot is an interactive chat project that leverages online/local LLM for rapid understanding and navigation of GitHub code repository.
Serge
Community
A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.
Build a Reasoning Model (From Scratch)
Various
Understand LLM reasoning by creating your own reasoning model–from scratch! LLM reasoning models have the power to tackle truly challenging problems that require finding the righ
gpt4all
Various
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Jan
Various
Jan is an open-source alternative to ChatGPT. Run open-source AI models locally or connect to cloud models like GPT, Claude and others.
Jenni
Various
Jenni is an AI research and academic writing assistant that helps students, researchers, and academics create essays, papers, and citations with ease.
LangChain
Various
LangChain provides the engineering platform and open source frameworks developers use to build, test, and deploy reliable AI agents.
LM Studio
Various
Run local AI models like gpt-oss, Llama, Gemma, Qwen, and DeepSeek privately on your computer.
Local Deep Research
Various
~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Every
privateGPT
Various
Interact with your documents using the power of GPT, 100% privately, no data leaks
PyGPT
Various
PyGPT is an open‑source desktop AI assistant for Windows, macOS and Linux. Chat, agents, web search, run Python, TTS/STT, plugins, long‑term memory.
RunThisLLM
Various
Find out exactly what hardware you need to run any local LLM, image, video, or audio AI model. 275+ models with full build specs and performance estimates.
Unsloth
Various
Unsloth is an open-source, no-code web UI for training, running and exporting open models in one unified local interface.
MikkoParkkola/nab
Various
Ultra-minimal browser engine with HTTP/3, JS support, cookie auth, passkeys, and anti-fingerprinting
srclight/srclight
Various
Deep code indexing MCP server for AI agents. 25 tools: hybrid FTS5 + embedding search, call graphs, git blame/hotspots, build system analysis. Multi-repo workspaces, GPU-accelerate
ollama
Community
Get up and running with Kimi-K2.5, GLM-5, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
prima.cpp
Community
A distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.
TreeScale
Community
TreeScale Platform is an All in One solution to use LLM Prompts to build powerful APIs and integrate them with your favorite tools and services.
Wllama
Community
WebAssembly binding for llama.cpp - Enabling on-browser LLM inference
gpt4all
Various
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
AilingBot
Community
One-stop solution to empower your IM bot with AI.
AutoGen
Microsoft
Microsoft's framework for multi-agent conversations. Agents that talk to each other to solve hard problems.
Awesome GPT
Community
A curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more.
awesome-japanese-llm
Community
日本語LLMまとめ - Overview of Japanese LLMs
Awesome-LLM-Inference
Community
📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism, etc. 🎉🎉
Baichuan-7|13B
Community
AGI Large Language Models
Build a Large Language Model (From Scratch)
Community
How to implement LLM attention mechanisms and GPT-style transformers.
ChatAbstractions
Community
LangChain chat model abstractions for dynamic failover, load balancing, chaos engineering, and more!
DeepSeek-VL-1.3|7B
Community
DeepSeek-VL model series
Future AGI
Community
Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript.
Gemma
Community
Checking your browser - reCAPTCHA
Gemma2-9|27B
Community
Gemma 2, our next generation of open models, is now available globally for researchers and developers.
Google "We Have No Moat, And Neither Does OpenAI"
Community
Leaked Internal Google Document Claims Open Source AI Will Outcompete Google and OpenAI
Guidance
Community
A guidance language for controlling large language models.
InternLM2-1.8|7|20B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Lancedb
Community
Developer-friendly OSS embedded retrieval library for multimodal AI. Search More; Manage Less.
Langchain-Chatchat
Community
Langchain-Chatchat（原Langchain-ChatGLM）基于 Langchain 与 ChatGLM, Qwen 与 Llama 等语言模型的 RAG 与 Agent 应用 | Langchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like
LiteLLM 🚅
Community
Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, Vertex
Llama 3.2-1|3|11|90B
Community
[Llama 3.1-8 70 405B](https://llama.meta.com/)
Llama 3-8|70B
Community
[Llama 2-7 13 70B](https://llama.meta.com/llama2/)
LLaMA Cult and More
Community
Large Language Models for All, 🦙 Cult and More, Stay in touch !
LlamaIndex
LlamaIndex
The data framework for LLM apps. RAG, ingestion, structured extraction, agents over your data.
llm-ui
Community
The React library for LLMs
MiniCPM-2B
Community
The MiniCPM family of LLMs and VLLMs.
Mixtral-8x7B
Community
The most powerful AI platform for enterprises. Customize, fine-tune, and deploy AI assistants, autonomous agents, and multimodal AI with open models.
Moonlight-A3B
Community
Moonshot's Compute-efficient MoE LLM, first Scaling Up of Muon Optimizer
MPT-7B
Community
Introducing MPT-7B, the first entry in our MosaicML Foundation Series. MPT-7B is a transformer trained from scratch on 1T tokens of text and code. It is open source, available fo
OLMo-7B
Community
Artifacts for the first set of OLMo models.
OneComp
Community
Python package for LLM compression
Outlines
Community
Structured Outputs
Phi1-1.3B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Phi3-3.8|7|14B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
prima.cpp
Community
A distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.
Puzzlet AI
Community
Redirecting...
Qwen2-0.5B|1.5B|7B|57B-A14B-MoE|72B
Community
GITHUB HUGGING FACE MODELSCOPE DEMO DISCORD Introduction After months of efforts, we are pleased to announce the evolution from Qwen1.5 to Qwen2. This time, we bring to you: Pret
Qwen-1.8B|7B|14B|72B
Community
Qwen - a Qwen Collection
Shimmy
Community
⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.
simulate-sdk
Community
Enterprise Grade, Voice AI simulation SDK for testing your AI Agents
StableLM-3B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
StableLM-v2-12B
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
StarCoder-1|3|7B
Community
All models, datasets, and demos related to StarCoder!
torchtune
Community
PyTorch native post-training library
DeepSeek
Various
Org profile for DeepSeek on Hugging Face, the AI community building the future.
Grok
Various
An LLM by xAI with [open source](https://github.com/xai-org/grok-1) and open weights. #opensource
Groq
Various
The Groq LPU delivers inference with the speed and cost developers need.
LLaMA
Various
Llama LLM, a foundational, 65-billion-parameter large language model by Meta. Meta, February 23rd, 2023. #opensource
LLM Stats
Various
The LLM Leaderboard — independent ranking of GPT, Claude, Gemini, Llama, DeepSeek and 300+ AI models by intelligence, speed and price. Composite LLM Stats Score updated continuou
LLM
Various
LLM: A CLI utility and Python library for interacting with Large Language Models
Open Interpreter
Various
A natural language interface for computers
Open Notebook
Various
Take Control of Your Research, Privately
Open LLMs
Various
📋 A list of open LLMs available for commercial use.
Qwen
Various
Qwickly forging AGI, enhancing intelligence.
SEAL LLM Leaderboard
Various
Explore leaderboards with expert-driven LLM benchmarks and updated AI model rankings across coding, reasoning and more.
Vicuna-13B
Various
We introduce Vicuna-13B, an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge s
CTranslate2
Community
Fast inference engine for Transformer models
exllama
Community
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
mistral.rs
Community
Fast, flexible LLM inference
MNN-LLM
Community
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
vLLM
Community
A high-throughput and memory-efficient inference and serving engine for LLMs
bitnet.cpp
Various
Official inference framework for 1-bit LLMs
gpt4all
Various
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
OpenAI API
Various
Announcement of the OpenAI API for text-to-text general-purpose AI models based on GPT-3. OpenAI blog, June 11, 2020.
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.