What Azure OpenAI actually is

Azure OpenAI is a managed service from Microsoft that gives you API access to OpenAI’s foundation models, but routed through your own Azure subscription rather than directly through OpenAI. Same model families you’d see on platform.openai.com, including GPT-4 class chat models, embedding models, and older completions models, all served from Microsoft’s data centers under your Azure tenant.

The technical difference matters. When you call OpenAI directly, your requests hit OpenAI’s infrastructure, billing is in dollars, and the data handling terms are governed by OpenAI’s enterprise agreement. When you call Azure OpenAI, the same models are deployed as resources inside your Azure subscription, with the endpoint, keys, networking, logging, and billing all living in your Azure account. The model weights are the same. The runtime experience is deliberately similar to OpenAI’s API so existing code ports across with minimal changes.

A few things this means in practice. Your data stays inside the Azure region you deploy to, which matters for European or Australian data residency requirements. You can put the endpoint behind a private VNet, restrict it to your corporate identity through Entra ID, and stream the request and response logs into your own Log Analytics workspace. Content filtering is layered on top of model outputs by default, with configurable severity thresholds for hate, sexual, self-harm, and violence categories. You don’t get any of that on the direct OpenAI API without bolting on extra services.

The trade is friction. Model availability lags behind OpenAI direct by weeks or months in some cases. Quotas are tighter, especially when you’re starting out. The portal is heavier than the direct experience. For an individual developer who just wants to call GPT, the direct API is faster to get going. For a team building something inside a company that already runs on Azure, the value proposition is strong and obvious.

Setup and authentication

You’ll need three things before you can make a real call: an Azure subscription, approved access to the Azure OpenAI service, and a deployed model inside an Azure OpenAI resource.

Subscription is the easy part. Any pay-as-you-go Azure account works, including free trial credits. Sign in at portal.azure.com, create a subscription if you don’t have one, and note the subscription ID because quotas and billing attach to it.

Access to Azure OpenAI is still gated by an application form for new accounts, though the policy has loosened over time. You submit basic information about your use case through the Azure OpenAI access request page. Approval is typically quick, often within hours for standard use cases, though some industries or workloads take longer. Until you get the approval email, the Azure OpenAI resource type won’t appear in the portal.

Once approved, the actual setup runs through these steps:

1. In the Azure portal, click Create a resource and search for Azure OpenAI. Pick the resource, choose your subscription and resource group, pick a region, give it a name, and select a pricing tier. Standard is fine for development.
2. After deployment finishes, open the resource and go to Resource Management, then Keys and Endpoint. Copy key one and the endpoint URL, you’ll need both.
3. Go to Model deployments or the Azure AI Foundry portal depending on which interface the current version of the portal routes you through, and create a new deployment. Pick a model like gpt-4o or gpt-4o-mini, give the deployment a name (this is what you’ll reference in API calls, not the model name), set tokens-per-minute quota, and deploy.

Authentication uses API keys by default, which is fine for development. For production, swap to Microsoft Entra ID (formerly Azure AD) with managed identity so the credential is rotated automatically and never appears in code. The Azure OpenAI SDK supports both, and the request structure is identical once you’re past the auth header.

First working example

Here’s a real, runnable Python example that assumes you’ve completed the setup above. Install the SDK first, then point it at your endpoint.

The relevant package is the Azure OpenAI Python client, which has been unified with the OpenAI SDK in recent versions. If you’re starting fresh, install openai and set two environment variables: AZURE_OPENAI_ENDPOINT pointing at the endpoint you copied, and AZURE_OPENAI_API_KEY with key one. If you’re on Entra ID auth, use DefaultAzureCredential from azure-identity instead of the key variable.

The call itself looks like this. Create a client, specify api_version to one of the supported versions such as 2024-08-01-preview, and pass your deployment name into the model field. Send a chat completion request with a system message and a user message, then print the response. The deployment name is what you set in step 3, not the underlying model name, which is a common source of confusion.

If you’d rather skip the SDK, the same thing works as a curl call against the endpoint. The URL pattern is your endpoint followed by openai/deployments/your-deployment-name/chat/completions, with the api-version as a query parameter. Headers include api-key with your key and Content-Type application/json. The body is a JSON object with a messages array following the standard OpenAI chat format. The response structure matches what you’d get from OpenAI direct.

A typical first response is a short assistant message that confirms the round trip works, often something like “Hello, how can I help you today?” Once you see that, you have a working pipeline and can start experimenting with the actual settings.

Key settings that matter

Most beginners treat the API as a single text-in, text-out box and miss the dials that change behavior substantially.

Temperature controls randomness on a scale from 0 to 2. Zero gives the most deterministic response, useful for structured extraction or classification. One is the default and a good baseline for general conversation. Above one produces more creative but less reliable output, which is rarely what you want in production.

Max tokens caps the response length. Set it explicitly when you need predictable latency or cost. If you leave it at the default, the model can chew through a long response when a short one would do.

Top_p is an alternative to temperature for controlling diversity. Most production systems should pick one or the other, not both at non-default values, because they interact in ways that produce surprising results.

System message is the highest-leverage setting people underuse. The system message sets the model’s role, tone, and constraints, and it deserves the same attention you’d give a prompt template. A weak system message like “You are a helpful assistant” leaves a lot of performance on the table. A specific one like “You are a senior accountant reviewing expense reports. Respond in formal English. Never invent policy details. If a question requires a specific policy document, say you don’t have access to it” produces dramatically better outputs on real tasks.

Frequency and presence penalties nudge the model away from repeating itself, useful for creative writing but mostly noise for analytical work.

On the Azure side specifically, two settings matter beyond the API call. Content filters can be configured per-deployment with severity thresholds for each category. For internal tools you control tightly, you can set them lower. For customer-facing products, leave them at the default. The TPM quota on the deployment controls throughput, and you’ll hit it before you hit the model’s actual capacity on any real workload. Plan for it.

The model version pinned to a deployment matters for reproducibility. When you deploy gpt-4o, you can pin to a specific snapshot like 2024-08-06. Lock that down in production or your outputs drift when Microsoft rolls a new version.

Where it shines

Enterprise integration is the headline use case and it’s genuinely strong. If your company already lives in Azure, the operational story is clean. Endpoints can sit behind private endpoints with no public internet exposure. Logs flow into Log Analytics and can be wired to Application Insights. Identity is Entra ID, the same identity that gates your other internal tools. Billing rolls into the existing enterprise agreement.

Compliance and data residency are the other big win. Data sent to Azure OpenAI stays in the chosen region, is not used to train the underlying model, and is deleted according to Azure’s standard data handling policies. For healthcare, finance, and government workloads where this is a hard requirement, the direct OpenAI API is often simply not on the table.

Predictable provisioning is the underrated benefit. You can deploy specific models in specific regions, lock in TPM quotas, and run multi-region failover in a way that the direct API doesn’t really support. For a production system serving real users, that operational control is worth a lot.

The content filtering layer is genuinely useful for customer-facing deployments. The four default categories catch a lot of bad output before it reaches a user, and the configurability is enough to tune for most domains without writing your own classifier.

Where it fails

The model freshness gap is real. New OpenAI models show up on the direct API first, sometimes by months. If your competitive advantage depends on the absolute latest model, Azure OpenAI is a poor choice.

The onboarding friction is heavier than the direct API. The application form, the resource deployment, the model deployment, the portal navigation, the quota requests when you hit the limit. For a solo developer prototyping, this is annoying. The trade is worth it for a team, less so for an individual.

Regional coverage is uneven. The most capable models are available in fewer regions than the older or smaller ones. If your data residency requirement points to a specific region, the model you want may not be there.

Cost can be higher or lower depending on the tier and commitment. Reserved capacity gives you predictable pricing but ties up budget. Pay-as-you-go is comparable to direct OpenAI in most cases but adds a small margin in some regions. Do the math for your actual volume before committing to either.

Finally, the Azure portal experience is a lot. There are at least three different interfaces that touch Azure OpenAI depending on what you’re doing. The classic resource blade, the Azure AI Studio, and the new Foundry portal. The naming and capabilities shift between them. Plan for a few hours of confused clicking the first time you touch the service.

Practical workflow pattern

A reasonable workflow for a team getting serious with Azure OpenAI looks like this. Start in a development subscription with low quotas and a single deployment of gpt-4o-mini, which is cheap and fast for iteration. Use the playground in the portal to validate prompts before writing any code. The playground exposes the same parameters as the API and lets you compare system messages side by side, which is much faster than re-running scripts.

Once prompts are stable, move them into code using the SDK, with environment variables for the endpoint, key, and deployment name. Keep model parameters in config, not hardcoded, so you can tune without redeploying. Add structured logging that captures the prompt, the response, the latency, and the token counts, ideally wired into Application Insights from day one.

Test against a staging deployment with TPM quotas that reflect production, because latency and throughput under real load are very different from single-call playground behavior. Lock model versions before going to production and document the snapshot you’re pinned to. Add a content filter audit step if your domain is sensitive.

For ongoing operation, set up cost alerts in Azure Cost Management tied to the resource. Monitor token consumption by deployment in the metrics blade. Build a small internal wrapper around the SDK that enforces your prompt templates, parameter limits, and logging, so every team in the company calls Azure OpenAI the same way. This is the kind of operational layer that turns “we use Azure OpenAI” into something a real business can run.

The pattern that fails is the opposite: every developer spinning up their own resource, their own keys in config files, no logging, no version pinning, no cost visibility. Six months in, nobody knows what model is running in production or what the monthly bill is.

Enterprise DNA put together a free field guide on exactly this: the full Claude ecosystem, Claude Code, and how to roll agents out without breaking things. Get the guide.