What Ollama API actually is

Ollama is an open-source runtime that packages open-weight large language models into a local server with a clean REST API. Under the hood it wraps llama.cpp and similar inference engines, manages model downloads, handles GPU offloading, and exposes a simple HTTP interface on localhost:11434 by default.

The API is the part most people care about. It is a thin JSON-over-HTTP layer that lets any language or framework send prompts and receive completions, chat turns, or embeddings without depending on a cloud provider. There is also an OpenAI-compatible endpoint at /v1/chat/completions, which means existing OpenAI client libraries work with minimal changes.

What it is not: a hosted service, a model marketplace in the SaaS sense, or a fine-tuning platform. You bring your own hardware, you choose which weights to pull, and you pay nothing per token. The trade is that throughput, model size, and concurrency are bounded by your machine.

Setup and authentication

Install Ollama on macOS, Linux, or Windows from ollama.com. On macOS it ships as a standard app. On Linux the install is a single curl pipe to the official script. On Windows you get an installer. After install the daemon runs in the background and listens on port 11434.

Authentication is intentionally minimal. By default the server binds to 127.0.0.1, so any process on the same machine can hit it without credentials. If you expose it to a network, set OLLAMA_HOST to bind to a specific interface and use a reverse proxy with auth in front of it. Treat it like any other local service.

Pull a model. The default tag for many families is the smallest reasonable size, but you can be explicit:

ollama pull llama3.2:3b ollama pull mistral:7b ollama pull qwen2.5:7b ollama pull phi3:mini

Verify the server is up:

curl http://localhost:11434/api/tags

That returns a JSON list of every model currently on disk. If you see your pulled model, you are ready to make calls.

First working example

The two endpoints you will use most are /api/generate for single-turn prompts and /api/chat for multi-turn conversations. Both accept JSON, both support streaming, and both return the same shape whether you stream or not.

A simple generate call from curl:

curl http://localhost:11434/api/generate -d ’{ “model”: “llama3.2:3b”, “prompt”: “Explain what a vector database does in two sentences.”, “stream”: false }’

The response includes the model output, total duration, token counts, and a done flag. Set stream to true and you get newline-delimited JSON chunks instead, which is what you want for chat UIs.

For a chat turn:

curl http://localhost:11434/api/chat -d ’{ “model”: “llama3.2:3b”, “messages”: [ {“role”: “system”, “content”: “You are a concise technical writer.”}, {“role”: “user”, “content”: “What is the difference between TCP and UDP?”} ], “stream”: false }’

If you prefer to keep your existing OpenAI client code, point it at the OpenAI-compatible endpoint:

import os from openai import OpenAI

client = OpenAI( base_url=“http://localhost:11434/v1”, api_key=“ollama” )

response = client.chat.completions.create( model=“llama3.2:3b”, messages=[{“role”: “user”, “content”: “Summarize the last quarter’s revenue drivers.”}] ) print(response.choices[0].message.content)

That single change is often the fastest way to drop a local model into an existing app. The api_key value is not validated, so any non-empty string works.

Key settings that matter

The default parameters are tuned for chat, but production use usually needs adjustment. The most important dials live in the request body or in a Modelfile.

num_ctx controls the context window size in tokens. The default is often 2048 or 4096 depending on the model. Bigger windows consume more memory and slow inference. Set this to the smallest value that fits your longest prompt plus expected output.

temperature controls randomness. 0 is near-deterministic, 1 is the model’s default sampling behavior, higher values get creative. For extraction, classification, and structured output tasks, 0 to 0.3 usually works best.

top_p and top_k control nucleus and top-k sampling. Most users can leave these alone, but if you are getting repetitive outputs, lowering top_k to around 40 and top_p to 0.9 often helps.

repeat_penalty discourages the model from looping. The default of 1.1 is fine for chat but can be raised to 1.2 or 1.3 for code generation where loops are common.

num_gpu sets how many layers to offload to the GPU. On Apple Silicon this is automatic. On NVIDIA you can tune it manually. If you run out of VRAM the server will fall back to CPU, which is much slower but still works.

num_predict caps the number of tokens generated. Use this to prevent runaway outputs and to bound latency.

For persistent configuration, create a Modelfile:

FROM llama3.2:3b PARAMETER temperature 0.2 PARAMETER num_ctx 8192 PARAMETER repeat_penalty 1.15 SYSTEM “You are a careful analyst. Always cite the source line you used.”

Build it with ollama create analyst -f Modelfile, then reference it by name in API calls. Modelfiles are the right place to encode system prompts, sampling defaults, and any custom templates your app relies on.

Where it shines

Local development and prototyping. You can iterate on prompts, chains, and tool-calling logic without burning API budget or leaking data to a third party. Latency is often lower than a cloud round trip for small models.

Privacy-sensitive workloads. Healthcare, legal, financial, and internal corporate data often cannot leave the perimeter. Ollama runs entirely on your hardware, which makes compliance reviews much simpler.

Offline and edge deployments. Field laptops, factory floors, ships, and air-gapped environments can run real LLMs without internet. The same API works in all of them.

Cost-free inference at scale of one. For a single user or a small team, the marginal cost of running a 7B model on a workstation is zero. That changes the economics of building internal tools.

OpenAI drop-in replacement. If you are migrating off OpenAI for cost or privacy reasons, the OpenAI-compatible endpoint means most codebases need only a base URL change.

Embedding generation. The /api/embed endpoint produces vectors for any embedding-capable model in your library. Local embedding generation is one of the most common production uses.

Where it fails

Hardware ceiling. A 70B model needs serious RAM and VRAM. Most laptops cap out around 13B comfortably. If you need frontier-class reasoning, the cloud still wins.

Concurrency. The default Ollama server handles one request at a time per model. There is experimental parallel support, but if you need to serve many users you will need to run multiple instances behind a load balancer.

Model freshness. The catalog moves quickly but lags behind the absolute latest releases. If you need a model the day it drops, you may have to wait or convert weights yourself.

No built-in fine-tuning. Ollama can load adapters and custom Modelfiles, but it is not a training platform. For fine-tuning you go elsewhere and then import the result.

Operational overhead. You own the uptime, the backups, the security patches, and the GPU driver compatibility. For a small team that is fine. For a 200-person company it is a real job.

Quality variance. A 3B local model is not a GPT-4 class model. Pick tasks where the bar is achievable. Routing hard tasks to a cloud model and easy tasks to Ollama is a common pattern.

Practical workflow pattern

The setup that tends to work in production looks like this. Keep a small model warm on every developer machine for fast iteration. Use a larger model on a dedicated GPU box or small cluster for shared workloads. Route requests through a thin abstraction layer so the application does not care which one answers.

A typical request flow:

The app sends a request to a router with a task type and a difficulty hint. The router picks between local Ollama and a cloud provider based on cost, latency, and capability rules. Local wins for embeddings, classification, extraction, and short completions. Cloud wins for long-context reasoning, complex code generation, and anything that needs the absolute best model.

For the local side, run Ollama as a systemd service on Linux or a launchd service on macOS. Pin the model versions in your Modelfile so behavior is reproducible. Stream responses back to the UI to keep perceived latency low. Log token counts and durations so you can spot regressions.

For the cloud side, keep the same OpenAI-compatible client code. The router swaps the base URL and API key. Your application code stays identical.

A few habits that pay off:

Pin model tags explicitly. llama3.2:latest can change underneath you. Use llama3.2:3b-instruct-q4_K_M or whatever the exact quantization you tested is.

Benchmark before you commit. Run your real prompt set against two or three candidate models and measure quality and tokens per second. The numbers often surprise people.

Watch VRAM. If Ollama starts swapping or your inference slows to a crawl, you have hit the GPU memory wall. Drop num_gpu, switch to a smaller quantization, or move to a bigger box.

Use the OpenAI-compatible endpoint unless you need a feature only the native API exposes. Streaming, function calling, and JSON mode are all supported there. The native /api/chat gives you extra control over Modelfile parameters and image inputs.

Version your Modelfiles in git. They are small text files and they define the behavior of every model in your stack. Treat them like schema