What LiteLLM actually is

LiteLLM is a Python library and proxy server that gives you a single, OpenAI-compatible interface to roughly one hundred language model providers. Under the hood it translates requests and responses between OpenAI’s API format and the native formats used by Anthropic, Google, Azure, AWS Bedrock, Cohere, Mistral, Together, Groq, Ollama, and many others. You write code against one schema and LiteLLM handles the protocol differences in the background.

The package itself has two distinct modes. The first is the Python SDK, where you import litellm and call completion() the same way you would call OpenAI’s client. The second is a proxy server you run locally or on a VM, which exposes an OpenAI-shaped HTTP endpoint. Anything that already speaks the OpenAI protocol (LangChain, LlamaIndex, custom apps, the OpenAI Python client itself) can point at the proxy and start hitting other providers without code changes.

It is worth distinguishing this from a routing framework. LiteLLM is not an agent orchestrator or a workflow engine. It does not plan multi-step reasoning, manage tool schemas, or build retrieval pipelines. It is a translation and routing layer, and once you understand that boundary, most of its design choices make sense.

The project is open source under an MIT license with optional enterprise features. The Python SDK works as a drop-in dependency. The proxy server is a FastAPI app you run via litellm command line or container image.

Setup and authentication

The SDK installation is a single pip line. The proxy server is a separate but related setup, and you will likely want both eventually.

For the SDK:

pip install litellm

That gives you the completion function, response objects shaped like OpenAI’s, and the routing primitives. Most providers are supported out of the box. For a few, you may need an extra extra package, for example litellm[anthropic] or litellm[bedrock], though in practice the core install pulls in most of these.

Authentication follows the convention each provider already uses. LiteLLM reads the same environment variables the native SDKs read. OpenAI uses OPENAI_API_KEY, Anthropic uses ANTHROPIC_API_KEY, Google uses GEMINI_API_KEY or a service account file, AWS Bedrock uses the standard AWS_* credential chain. If you have these set in your shell or .env, LiteLLM picks them up without extra configuration.

You can also pass the key directly in the model string using a api_key=... prefix, but for anything beyond quick testing, environment variables are the right pattern. Keys in code are an anti-pattern that LiteLLM cannot protect you from.

For the proxy server, the typical install is:

pip install 'litellm[proxy]'

Then create a config.yaml file describing your models, and start the server with:

litellm --config config.yaml --port 4000

The proxy now listens on http://localhost:4000 and accepts the same request shape as https://api.openai.com/v1. Point any OpenAI-compatible client at it and it will route to the correct upstream provider based on the model name in the request.

First working example

Here is a minimal SDK call that hits three different providers in a single script. The whole point of LiteLLM is that you do not need three clients to do this.

import os
from litellm import completion

providers = [
    ("gpt-4o-mini", "openai"),
    ("claude-3-5-sonnet-20240620", "anthropic"),
    ("gemini/gemini-1.5-flash", "google"),
]

for model, label in providers:
    response = completion(
        model=model,
        messages=[{"role": "user", "content": "Reply with one sentence about LiteLLM."}],
    )
    print(f"{label}: {response.choices[0].message.content}")

Run that with the three API keys in your environment and you get three answers, one per provider, through one code path. The gemini/ prefix is LiteLLM’s way of disambiguating when multiple providers host a similarly named model.

If you prefer the proxy pattern, the equivalent is to set up a config.yaml like this:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: claude-3-5-sonnet-20240620
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gemini-flash
    litellm_params:
      model: gemini/gemini-1.5-flash
      api_key: os.environ/GEMINI_API_KEY

Start the server, then in any client code:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="anything")
resp = client.chat.completions.create(
    model="claude-sonnet",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

The api_key="anything" is intentional. The proxy ignores it for upstream routing but the OpenAI client requires a non-empty value.

Key settings that matter

Most people stop at the basics and miss the controls that actually save cost and reduce incidents. These are the dials worth learning early.

Model aliasing is the proxy feature you will use most. The model_name field in the config is a free-form string your clients send. The litellm_params.model field is the real upstream model. This decoupling lets you swap gpt-4o for gpt-4o-mini or rotate to a different provider without touching client code.

Routing strategies go beyond simple aliasing. You can set weight on each entry to load-balance across providers probabilistically. You can use rpm/tpm rate limits to cap how much each model receives per minute. The proxy also supports timeout, stream_timeout, and per-model max_retries, which become important when one provider is flaky.

Fallbacks are the most valuable single feature. In the config, a fallbacks block says “if gpt-4o fails, try claude-sonnet, then gemini-flash”. This is not just convenience. It is how you build systems that survive provider outages. A real production setup usually defines a primary model, a same-tier fallback, and a cheap fallback that always answers.

litellm_settings:
  fallbacks:
    - gpt-4o: [claude-sonnet, gemini-flash]
    - claude-sonnet: [gpt-4o, gemini-flash]

Spend tracking and budgets come built in. The proxy can enforce a max_budget per virtual key, per team, or per model. When the budget is hit, requests return a 429. For anyone running shared infrastructure, this is the difference between a controllable cost line and an open bar.

Virtual keys and the admin UI are part of the proxy. You mint keys for users, set per-key rate limits, and watch usage in a built-in dashboard. The UI is functional rather than pretty, but it is genuinely useful for small teams that do not want to build their own metering layer.

Streaming and async are first-class. acompletion works the same way as completion but returns an async iterator. Function calling, vision inputs, JSON mode, and tool use all work consistently across providers, which is where LiteLLM earns its keep compared to writing three custom adapters.

Logging and callbacks plug into the proxy. You can register callbacks to send every request to a database, a log aggregator, or an observability tool. The callback API is the natural integration point for Langfuse, Helicone, or your own analytics layer.

Where it shines

LiteLLM is genuinely the right tool for a few specific situations, and mediocre for others. It is worth being precise.

Multi-provider production systems are the sweet spot. If your product hits OpenAI for some features, Anthropic for others, and Google for cheap batch work, the maintenance cost of three separate clients is real. LiteLLM collapses that to one. The translation layer also smooths over provider-specific quirks, like Anthropic’s system prompt handling or Gemini’s safety blocks, though it cannot make them identical.

Cost-driven model selection is a strong use case. You can route cheap prompts to a small model, complex prompts to a flagship model, and use fallbacks so the system never blocks. Teams that want to A/B test providers without rewriting application code use the aliasing pattern heavily.

Local development and prototyping is a place where the proxy shines. You can point the same client at ollama/llama3 locally, gpt-4o in staging, and claude-sonnet in production, all by changing the URL or a single model name.

OpenAI-compatible drop-in replacement is a structural win. If you have an existing app using the OpenAI SDK and you want to add a second provider, the proxy is a five-minute change. No code modifications, no retraining, no schema work.

Migrating off OpenAI is a real scenario. The proxy lets you run a split test where a percentage of traffic goes to a non-OpenAI provider. The migration is reversible at the proxy config level, which is a much safer posture than rewriting your application.

Where it fails

Honest list of where LiteLLM does not help, or actively gets in the way.

It does not make models equivalent. A 400 error from one provider is not the same shape as a 400 error from another, even after translation. Reasoning quality, tool-calling reliability, and latency profiles remain wildly different. LiteLLM is a transport, not a quality layer.

New model features lag. When OpenAI ships a new output format or Anthropic ships a new tool-use mode, there is usually a delay before LiteLLM exposes it cleanly. The library tracks upstream providers closely but is not faster than they are.

The proxy adds latency. You are adding a network hop and a translation step. For high-throughput production systems, this matters in the range of tens of milliseconds per request. For most use cases, this is invisible. For latency-critical paths, it can compound.

The proxy is a single point of failure unless you run it redundantly. If you centralize all your LLM traffic through one LiteLLM process, that process becomes a critical dependency. You will want to run it behind a load balancer with health checks, which is a small DevOps project in itself.

The error messages can be opaque. When a translated request fails, the error often comes back wrapped in a generic shape that does not immediately reveal which provider or which field caused the problem. The library has improved here over time, but debugging multi-provider issues through LiteLLM is harder than debugging them against the native SDK.

It does not handle embeddings uniformly across all providers. Embedding endpoints are supported, but the dimension and similarity characteristics differ. A pipeline that assumes OpenAI embedding dimensions will break when you swap in Voyage or Cohere.

Large enterprises may want more than it offers. If you need SSO, audit logs, fine-grained RBAC, and SOC2 documentation, the open-source version is the floor of what is available. The commercial product fills some of this, but you should evaluate it against your requirements rather than assume it matches a purpose-built gateway.

Practical workflow pattern

The pattern that works in real work, in the order you should adopt it.

Start with the SDK for a single provider to confirm your application code is correct. Then add a second provider through the same litellm.completion call. Get the multi-provider call working before introducing the proxy. The proxy adds operational complexity and you want to know whether a failure is in your code, in the translation layer, or in the upstream provider.

Once multi-provider calls work via the SDK, move to the proxy. Start it locally, point one client at it, and confirm parity. Then start routing real traffic. Add virtual keys at this stage so you can attribute cost to specific applications or teams.

Define fallbacks early. Even a naive “primary plus cheap fallback” setup will save you the first time a provider has a bad afternoon. Add budget caps next, because cost surprises on metered APIs are typically larger than latency surprises.

Wire the proxy to your logging stack. The callback API is the right place to send every request and response to a database or an observability tool. If you cannot see your traffic, you cannot optimize it.

For local development, run the proxy with an Ollama model configured. The cost is zero, the latency is local, and you can develop against the same OpenAI client your production code uses. This is one of the highest-leverage habits to build.

For production, run the proxy as a container behind a load balancer. Set health checks. Set a sensible max_retries and timeout on each model. Rotate virtual keys periodically. Treat the proxy the way you would treat any other critical service in your stack.

One final point on team ergonomics. When everyone uses the same OpenAI-shaped client and routes through one proxy, the conversation about “which model should we use” moves from application code into infrastructure config. That is the right place for that conversation, and LiteLLM is a clean way to make it happen.

If you want the playbook other teams are using with Claude and Codex right now, grab the free Working With Claude field guide. Download it here.