What Together AI actually is

Together AI is a cloud inference platform built specifically for open source language and image models. Under the hood it is a managed GPU layer that hosts hundreds of open weights models and exposes them through an HTTP API. You do not run the models yourself, you do not provision hardware, and you do not deal with vLLM or TGI configuration. You send a request, you get a response, and you pay per token.

The interesting part is the API surface. Together AI ships an OpenAI-compatible endpoint, which means the same client code you would write for OpenAI works against Together with only a base URL swap. It also offers its own native Python SDK and a few endpoints that go beyond chat, including embeddings, image generation, and fine-tuning jobs.

Models you can call include the Llama family from Meta, Mistral and Mixtral models, Qwen models, DeepSeek variants, and a long tail of community checkpoints. The catalogue changes frequently as new open releases land, so treat any specific model name as a snapshot rather than a permanent fixture. You can list the current set through the models endpoint and pin the version you want in your own config.

The platform is run by Together Computer, a company that has been in the open source ML infrastructure space for several years. Their positioning is straightforward. Be the cheapest reliable place to run open models at scale, with an API that does not lock you into a single provider. If you care about not being tied to one vendor’s roadmap, that flexibility is the actual product.

Setup and authentication

Start by creating an account at together.ai. Once you are in, the dashboard gives you an API key under Settings or Account. Treat this key like any other secret. Do not commit it to a repo and do not paste it into a notebook that gets shared.

Set it as an environment variable in your shell. On macOS or Linux that looks like this in your shell profile:

export TOGETHER_API_KEY=“your-key-here”

On Windows in PowerShell:

$env:TOGETHER_API_KEY=“your-key-here”

You can confirm it is loaded with echo $TOGETHER_API_KEY on Unix or echo $env:TOGETHER_API_KEY on PowerShell.

For the Python SDK, install the official package:

pip install together

If you prefer to use the OpenAI client for portability, that works too:

pip install openai

Both options are valid. The native SDK exposes a few Together-specific parameters that the OpenAI client does not, while the OpenAI client gives you a portable code path if you intend to swap providers later.

There is no separate CLI to install for basic usage. The dashboard includes a Playground where you can test prompts in the browser, which is useful for verifying that your key works before you start writing code. You can also confirm billing is set up, since free tier accounts have hard caps that will surface as 429 errors once exceeded.

First working example

Here is a minimal working Python script using the native Together SDK. It sends a single chat completion request to a Llama model and prints the response.

import os from together import Together

client = Together(api_key=os.environ.get(“TOGETHER_API_KEY”))

response = client.chat.completions.create( model=“meta-llama/Llama-3.3-70B-Instruct-Turbo”, messages=[ {“role”: “system”, “content”: “You are a concise technical assistant.”}, {“role”: “user”, “content”: “Explain what a vector database is in two sentences.”} ], max_tokens=200, temperature=0.7 )

print(response.choices[0].message.content)

If you would rather use the OpenAI client, the same call looks like this.

import os from openai import OpenAI

client = OpenAI( api_key=os.environ.get(“TOGETHER_API_KEY”), base_url=“https://api.together.xyz/v1” )

response = client.chat.completions.create( model=“meta-llama/Llama-3.3-70B-Instruct-Turbo”, messages=[ {“role”: “system”, “content”: “You are a concise technical assistant.”}, {“role”: “user”, “content”: “Explain what a vector database is in two sentences.”} ], max_tokens=200, temperature=0.7 )

print(response.choices[0].message.content)

Run either script with python scriptname.py. If the key is set correctly you should see a short answer returned in one shot, or streamed token by token if you set stream=True.

To verify the key from the command line without writing a script, you can curl the models endpoint.

curl -H “Authorization: Bearer $TOGETHER_API_KEY” https://api.together.xyz/v1/models

A JSON list of available model IDs comes back if everything is wired up. That single command is often the fastest way to debug auth issues since the error messages are specific.

Key settings that matter

Most users only touch temperature and max_tokens. There are several other dials that change output quality in noticeable ways, and Together exposes a few that the OpenAI client does not.

temperature controls randomness on a scale from 0 to 2. Values around 0.7 are typical for general chat. For deterministic extraction or classification work, drop it to 0 or close to it. Pushing it above 1 produces more creative but less reliable output, which is rarely what you want in production.

top_p is the nucleus sampling cutoff. The default works for most cases. If you want to constrain the model to high-probability tokens only, set it lower, around 0.1 to 0.3. Pairing low top_p with low temperature is a common pattern for structured outputs.

top_k limits the candidate token pool to the k most likely next tokens. Useful when you want to suppress very low probability continuations without going full greedy.

repetition_penalty is a Together-specific parameter that discourages the model from repeating itself. Values between 1.0 and 1.2 are common. Higher values push harder against loops, which matters for long generations where the model tends to spiral.

stop lets you pass a list of strings that halt generation when emitted. Useful for forcing structured output, cutting off at a delimiter, or stopping when the model signals completion with a sentinel token.

stream returns chunks as they are produced instead of waiting for the full response. For chat interfaces this is the difference between a snappy UI and a long pause. The chunk format matches OpenAI’s, so any streaming parser you already have will work.

max_tokens caps the response length. Set this explicitly rather than relying on defaults, especially in production where runaway responses can blow up your bill.

safety_model is a Together option that runs output through a moderation model. Enable it for user-facing applications where you want a second pass filter before content reaches the end user.

For embeddings, the relevant endpoint is client.embeddings.create with model names like togethercomputer/m2-bert-80M-8k-retrieval or BAAI/bge-large-en-v1.5. The response is a list of float vectors you can store in any vector database. Embedding dimensions vary by model, so record the size when you pick one and stick with it.

Where it shines

The strongest case for Together AI is open model access without infrastructure work. If you want to run Llama 3, Mixtral, or Qwen at scale, you can do it in an afternoon instead of standing up a GPU cluster, configuring CUDA drivers, and tuning inference servers.

Cost is the second strong case. Pricing on open models is typically lower than equivalent closed model APIs at this tier, and you can route traffic to whichever model gives you the best price to quality ratio for a given task. For high volume workloads the savings add up quickly.

Model variety is the third. Because Together is model-agnostic, you can A/B test across providers and architectures with the same client code. That makes it a good fit for evaluation pipelines where you want to score the same prompt against several backends and pick a winner.

The OpenAI-compatible surface makes migration painless. If you have an existing application written against the OpenAI SDK, swapping base_url is often the only change required to start using open models. That is also your exit strategy if pricing or quality ever shifts elsewhere.

Fine-tuning is available for a subset of models, which is useful when you need a domain-specific variant and do not want to manage the training infrastructure yourself. Upload a JSONL file, kick off a job, and receive a fine-tuned model ID you can call like any other.

Dedicated endpoints let you reserve capacity for a specific model, which removes cold start latency and gives you predictable throughput for production traffic at higher spend tiers.

Where it fails

Latency on cold starts can be higher than what you would see from a fully warm closed provider. For real-time interactive applications this matters and you may need to keep a low-latency model as a fallback or pay for dedicated capacity.

Rate limits vary by model and account tier. Some of the larger open models have tighter limits than you would expect, and you can hit them during bursty workloads. Plan for retries with exponential backoff and consider request queuing for non-interactive jobs.

The dashboard and observability tooling are functional but not as polished as the major closed providers. If you need detailed tracing, custom dashboards, or SOC2 reports, plan for additional work on your side or accept the gap.

Context windows on some open models are smaller than the latest closed models. If your prompts routinely exceed 32k tokens, check the specific model card before committing. Long context summarization is a common place where this bites people.

Some models in the catalogue are community uploads and quality varies. Always evaluate a model on your own data rather than trusting the listing. A model that performs well on public benchmarks may still fail on your specific distribution.

Support response times are not at the level of hyperscale providers. If your business depends on a same-day SLA for vendor issues, factor that into your risk model.

Practical workflow pattern

A reasonable production pattern is to write your client code against the OpenAI SDK with a configurable base URL. That way you can route different tasks to different providers without changing application code.

Store the API key and base URL in environment variables, and load them through a config