What OpenAI Batch API Actually Is

The OpenAI Batch API is a job-queue endpoint that processes a collection of chat completion requests as a single batch rather than one HTTP call at a time. Instead of firing requests synchronously and waiting on each response, you upload a JSON Lines file containing many requests, submit it as a batch job, and retrieve results once processing finishes.

The critical tradeoff is that you trade latency for cost. OpenAI charges roughly half the per-token price for batch requests compared to the standard Chat Completions endpoint, in exchange for results that can take up to 24 hours to complete. Some batches finish in minutes, but you cannot depend on that. Anything that needs a response inside a user-facing session is the wrong use case. Anything that can wait overnight and where the math on tokens is the dominant cost is exactly right.

Each batch job has four lifecycle states: validating, in_progress, finalizing, and completed. You submit the file, OpenAI validates the schema, processes the requests in parallel on shared infrastructure, then makes an output file available with the responses. Failed requests come back with error details rather than crashing the whole batch.

The API supports the same model lineup as the synchronous endpoint, including GPT-4o, GPT-4o mini, and the o-series reasoning models. Each request in a batch is independent, so there is no built-in mechanism for chaining or sharing state across requests inside a single batch.

Setup and Authentication

You need three things: an OpenAI account with a paid API key, the openai Python SDK at version 1.40 or later, and a way to write JSON Lines files.

Install the SDK with pip install --upgrade openai.

Set your API key as an environment variable. On Linux or macOS, that looks like export OPENAI_API_KEY="sk-...". On Windows in PowerShell, use $env:OPENAI_API_KEY="sk-...".

The Batch API uses the same authentication as the rest of the OpenAI API. There is no separate batch-specific key, no separate billing line, and no separate rate limit tier. Your existing limits and credits apply.

The endpoint surface is small. You create a batch by uploading a file, then call client.batches.create() with the file ID. You poll client.batches.retrieve() until status hits completed. You download the output file. That is the full lifecycle.

First Working Example

A working example end to end looks like this. You build a JSONL file where each line is a request object with a custom_id, the HTTP method, the URL pointing to the chat completions endpoint, and the body of the request.

If you want to classify 500 customer support tickets, each line follows a shape like this:

{"custom_id": "ticket-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "system", "content": "Classify the following support ticket into one of: billing, bug, feature_request, other. Respond with only the label."}, {"role": "user", "content": "My invoice for May is wrong, I was charged twice."}]}}

You repeat that pattern for every ticket, with a unique custom_id per line. Then you upload the file with client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch"). That returns a file object with an id. You pass that id into the batch creation call:

batch = client.batches.create(input_file_id=file.id, endpoint="/v1/chat/completions", completion_window="24h")

You then poll for completion. A simple loop every 30 seconds works:

import time while batch.status not in ["completed", "failed", "expired"]: time.sleep(30) batch = client.batches.retrieve(batch.id)

When the status is completed, the batch object exposes an output_file_id. Download it with result = client.files.content(batch.output_file_id) and write the bytes to a local file. Each line of the output is a JSON object keyed by the custom_id you provided, with the response body or an error. You join it back to your source data on custom_id and you have your labels.

The whole flow takes ten to fifteen minutes of coding the first time and runs without further intervention. On a 500-ticket job using gpt-4o-mini, the total cost lands in the low single-digit dollar range rather than the higher single digits you would pay with synchronous calls.

Key Settings That Matter

The first dial is completion_window. The only supported value is “24h”. Mentioning it in the request signals that you understand the batch is asynchronous, but it does not change the timing. The window is the upper bound, not a target.

The second dial is the model choice. gpt-4o-mini is the right default for classification, extraction, summarisation, and any task where the prompt structure is doing most of the work. gpt-4o earns its place when reasoning quality is the bottleneck. o-series reasoning models work in the batch endpoint but the cost premium is large and the latency savings matter less in a context where you are already waiting hours, so reach for them only when the task genuinely needs them.

Third, request metadata. Every line carries a custom_id that comes back attached to the response. Use it. Do not rely on array order or line order, because the Batch API does not guarantee output preserves input ordering. Anything you need to rejoin downstream needs a custom_id that points back to a stable record key in your source data.

Fourth, endpoint versioning. The url field on each line must match the model family. Pointing a gpt-4o request at /v1/completions breaks at validation time. Keep all lines pointed at /v1/chat/completions for chat models, or /v1/embeddings if you are running embedding batches.

Fifth, error handling per request. The output file contains both successful responses and error objects. A request can fail because the input exceeded the context window, because the model rejected a parameter, or because of a content filter. Code your consumer to handle both shapes, not just the happy path.

Sixth, idempotency. The Batch API does not have a built-in dedup key across batches. If you submit the same batch twice you get billed twice. Track the custom_id prefixes and submitted batch ids in your own store, and treat a batch as a write operation against your job ledger.

Seventh, the metadata field on the batch creation call. It accepts up to 16 key-value pairs of strings. It does nothing functional. It does everything operational, because it is the only field you can query or filter on when you list batches a week later. Tag every batch with at least a project name and a job type.

Where It Shines

Bulk classification is the canonical win. Routing support tickets, tagging product reviews, sorting inbound sales emails, labelling CRM notes. All of these have three properties that line up with the Batch API’s strengths: high volume, no real-time requirement, and short prompts and short expected outputs.

Document extraction at scale is the second strong fit. Parsing thousands of PDFs, contracts, or invoices into structured fields. You write a prompt once, run it against a folder, and let the batch churn overnight. By morning you have a structured dataset that would have taken a person weeks to type out by hand.

Evaluation runs fit naturally. Sweeping a prompt template or a fine-tuned model against a fixed eval set is a batch job by definition. You prepare the eval cases once, submit them as a batch, compare completions against your scoring code, and you have a clear pass/fail picture.

Synthetic data generation is the fourth fit. Building a labelled training set, generating edge-case examples for red-teaming, or producing paraphrase pairs for a search index. All of these are throughput problems with no latency constraint, and the cost discount compounds sharply when you are generating tens of thousands of examples.

In all of these cases the cost shape is the same. The batch endpoint lands at roughly half the per-token cost of the synchronous endpoint, and on large jobs the absolute savings are large in dollar terms even when the percentage is unchanged.

Where It Fails

Real-time workflows are out. Anything in a request-response loop with a user is wrong. If the user is waiting, the synchronous endpoint or the streaming endpoint is the right tool.

Small jobs are a poor fit in dollar terms. A five-request batch still pays minimum processing overhead and the savings versus five synchronous calls are trivial. The break-even on the operational complexity of a batch job is somewhere in the range of 50 to 100 requests in a single submission.

Long-context workloads are awkward. A 128k context request is possible inside a batch, but the per-request cost scales with input tokens just like the synchronous endpoint, and the latency is bounded only by total queue depth. For very large context jobs, the cost saving is real but the 24-hour ceiling starts to bite if you need to process tens of thousands of long documents.

Stateful chains are not supported. If your task is “look up the previous result, then use it in the next call”, you are running a workflow, not a batch. The Batch API has no internal state, no shared memory, and no per-job scratch storage. Build that on top with your own pipeline.

Finally, the SLA is a soft promise. OpenAI does not commit to a fixed completion time. Most batches complete in well under an hour for small jobs, but a stuck validation step or a backlog spike can push completion to the full 24-hour window. Build your downstream automation to tolerate that, or you will end up with stale data and missed deadlines.

Practical Workflow Pattern

The pattern that works in production is a small job runner, not a one-off script. The job runner takes three