What Groq actually is
Groq is an inference provider, not a model lab. The company designs custom silicon called the LPU (Language Processing Unit) and exposes that hardware through an API that serves open-source large language models. When you call Groq, you’re getting a hosted open-weight model running on hardware optimized specifically for the token-by-token work that inference requires.
The practical consequence is speed. Groq’s published numbers and the experience of using the API consistently show token generation rates in the hundreds of tokens per second for the models they host, which is substantially faster than what you’d see from general-purpose GPU-based providers for equivalent model sizes. The latency on the first token is also low, which matters for streaming applications.
The model catalog includes Llama variants, Mixtral, Gemma, and others. Groq does not train these models. They license or download the open weights and run them on their hardware. This matters because it means the API behaves like any other inference endpoint, you send prompts in, you get completions out, and the underlying model is one you could in principle run yourself on your own hardware if you had enough of it.
The API itself is OpenAI-compatible. Requests look almost identical to what you’d send to OpenAI’s chat completions endpoint, which means existing code, SDKs, and tooling work with minimal changes. You swap the base URL and the API key, and most things just work.
Setup and authentication
The setup is straightforward. Create an account at console.groq.com, generate an API key from the dashboard, and store it as an environment variable. The key is a standard bearer token used in the Authorization header.
In your shell:
export GROQ_API_KEY=“your_key_here”
For Python projects, the cleanest approach is to use the official OpenAI SDK pointed at Groq’s base URL. Install the SDK if you haven’t already:
pip install openai
Then in your code:
from openai import OpenAI
client = OpenAI( api_key=os.environ.get(“GROQ_API_KEY”), base_url=“https://api.groq.com/openai/v1” )
That base URL is the only structural difference between calling Groq and calling OpenAI. Everything else, model names, parameter names, response shapes, follows the same conventions.
If you prefer not to use the OpenAI SDK, you can hit the endpoint directly with requests or any HTTP client. The endpoint is https://api.groq.com/openai/v1/chat/completions, the body shape matches OpenAI’s chat completions spec, and the response shape matches too.
For Node.js, the same pattern works with the openai npm package. For other languages, any OpenAI-compatible client library will work with just the base URL swap.
First working example
Here’s a minimal Python script that sends a prompt and prints the response:
import os from openai import OpenAI
client = OpenAI( api_key=os.environ.get(“GROQ_API_KEY”), base_url=“https://api.groq.com/openai/v1” )
response = client.chat.completions.create( model=“llama-3.1-70b-versatile”, messages=[ {“role”: “user”, “content”: “Explain inference latency in two sentences.”} ], temperature=0.7, max_tokens=200 )
print(response.choices[0].message.content) print(f”Tokens used: {response.usage.total_tokens}”)
When you run this, you’ll see the response print almost immediately, typically well under a second for a short prompt on the larger models. The usage block at the end gives you token counts, which is useful for cost tracking.
To check available models and their capabilities, hit the models endpoint:
models = client.models.list() for m in models.data: print(m.id)
This returns the current catalog. Model availability changes as Groq adds and deprecates models, so check this list rather than hardcoding names.
For streaming, add stream=True and iterate over the chunks:
stream = client.chat.completions.create( model=“llama-3.1-8b-instant”, messages=[{“role”: “user”, “content”: “Write a haiku about databases.”}], stream=True )
for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="")
Streaming is where the speed advantage becomes most visible. Tokens start arriving faster than your terminal can render them, which is unusual compared to most other providers.
Key settings that matter
The parameters that meaningfully change behavior are the same as OpenAI’s, but a few deserve specific attention when you’re targeting speed.
Model selection is the biggest lever. Smaller models like the 8B parameter variants return tokens faster than larger ones. For tasks where the smaller model is good enough, switching from a 70B to an 8B can roughly double your tokens-per-second. Test before assuming you need the bigger model.
Temperature controls randomness. For deterministic outputs like JSON extraction or classification, set it to 0. For creative work, 0.7 to 1.0 is typical. Note that setting temperature to 0 doesn’t guarantee identical outputs across runs on all providers, so if you need strict determinism, build your own caching layer.
max_tokens caps the response length. Set this explicitly rather than relying on defaults, especially in production. A runaway response can blow through your token budget and add latency you didn’t plan for.
stream=True is worth using whenever the user is waiting on the response. The time-to-first-token matters more than total response time for perceived speed in interactive applications. For batch jobs where you’re processing thousands of items, non-streaming is usually fine because you’re optimizing for throughput, not latency.
response_format with type “json_object” enables JSON mode, which constrains the model to output valid JSON. This is useful for structured extraction tasks. The model still needs to be told in the prompt what schema to produce, the parameter just enforces the format.
parallel_tool_calls controls whether the model can emit multiple function calls in a single response. Default is true. If you’re building agent systems and want strict sequential tool use, set this to false.
stop sequences let you cut off generation at specific strings. Useful for preventing the model from continuing past a delimiter in structured output tasks.
seed parameter, when supported by the model, gives you reproducible outputs for the same input. Not all models support it, check the model card.
For rate limits, the free tier has tight limits, typically in the range of a few requests per minute and a few thousand tokens per minute. Paid tiers raise these substantially. If you’re hitting limits, the response will include a 429 status with a Retry-After header.
Where it shines
Real-time chat interfaces benefit most from Groq’s speed. When a user types a message and waits for a response, the difference between a 200ms first-token latency and a 2-second first-token latency is the difference between feeling like a conversation and feeling like a form submission. If you’re building any kind of chat UI, streaming through Groq produces a noticeably better experience than streaming through slower providers.
Voice and speech-adjacent applications are another strong fit. Voice agents need low latency to feel natural, and the round-trip from speech recognition to LLM to text-to-speech has to fit in a budget of a few hundred milliseconds to keep the conversation flowing. Groq’s inference speed makes it easier to stay within that budget.
High-throughput batch processing where each item is small is a third area. If you’re classifying thousands of short documents, summarizing a backlog of tickets, or extracting structured data from a queue of inputs, the per-request latency compounds. Faster inference means more items processed per unit of wall-clock time, which directly translates to lower compute costs even at similar per-token pricing.
Function calling and tool use work well because the latency between tool execution and the model’s next response is reduced. In an agent loop where the model calls a tool, gets a result, and decides what to do next, every second shaved off the inference step is a second saved in the overall loop.
Code generation for interactive IDEs or REPL-style tools is another fit. When developers are waiting on completions, speed is the feature.
Where it fails
The model catalog is smaller than what you’d find at OpenAI or Anthropic. You’re working with open-weight models, which means you don’t get access to the latest proprietary frontier models. For tasks that genuinely require the strongest available reasoning, Groq may not be the right primary provider.
Context windows vary by model and are generally in the range of 8K to 128K tokens depending on the model. If you need to process very long documents in a single request, check the specific model’s context limit before assuming it’ll fit.
There’s no fine-tuning API. You can prompt and you can do retrieval-augmented generation, but you cannot upload training data and get a custom model back. If your use case requires fine-tuning, you’ll need to use a different provider or fine-tune yourself and host elsewhere.
Rate limits on the free tier are restrictive enough that serious production use requires a paid plan. The pricing is competitive but not free, and the cost calculus depends on your throughput.
Some models have output token limits that are lower than their context windows. A model with a 128K context might cap output at 8K tokens. Check the model documentation