What Voice AI Actually Is

Voice AI is a category of models and APIs that handle spoken language. Strip the marketing and it splits into three jobs. For a business-focused overview of voice AI and what it can do for your operations, the hub is a better starting point if you are evaluating deployment rather than building an API integration.

First, speech to text, also called STT or automatic speech recognition. You feed it audio and it returns written words. OpenAI’s Whisper and Deepgram are the workhorses in this space, with Google’s and Amazon’s offerings close behind.

Second, text to speech, or TTS. You feed it text and it returns spoken audio. OpenAI’s TTS, ElevenLabs, and a growing set of open source models handle this.

Third, voice agents. These are real time systems that listen, interpret, and respond with voice. Platforms like Vapi, Retell, and Bland layer turn detection, interruption handling, and telephony on top of an STT to LLM to TTS pipeline.

At the core, this is audio in, text out (or the reverse), powered by neural networks trained on huge amounts of speech data. The “intelligence” part usually means a large language model bolted onto the audio pipeline to decide what was said and what to say back.

For most builders, the practical question is which API to call and how to wire it into your app. That is what this guide covers.

Setup and Authentication

Pick a provider. For this walkthrough I’ll use OpenAI because the API is straightforward and a single account gets you STT, TTS, and LLM routing without juggling vendors.

Step one, get an account and an API key. Sign up at platform.openai.com, navigate to API keys, create a new secret key, copy it once, and store it somewhere safe. The key is only shown at creation.

Step two, install the SDK. For Python run pip install openai. For Node run npm install openai. The Python version requires Python 3.7.1 or later.

Step three, set the environment variable. In your shell run export OPENAI_API_KEY="sk-...". In production you set this in your hosting provider’s secret manager, not in code.

Step four, pick a model. For STT, the current version of Whisper is the standard choice. For TTS, the tts-1 and tts-1-hd families are the entry points. For voice agents you can chain STT to an LLM to TTS in sequence, or use the Realtime API which collapses the loop into a single WebSocket.

Authentication is a bearer token in the Authorization header. The SDK handles this automatically when the environment variable is set. If you hit rate limits you’ll get a 429 response. Back off, retry with exponential delay, and consider queueing requests on your end.

First Working Example

Here is a working STT call. Given an audio file at path.mp3, the entire transcription is this.

from openai import OpenAI
client = OpenAI()

audio_file = open("path.mp3", "rb")
transcript = client.audio.transcriptions.create(
    model="whisper-1",
    file=audio_file
)
print(transcript.text)

Audio in, text out. The same library supports TTS.

response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Hello, this is a test of the text to speech system."
)
response.stream_to_file("output.mp3")

For a real time voice loop without the Realtime API, you chain STT to an LLM to TTS.

First, capture audio from the user’s microphone. In the browser use MediaRecorder. In Python use PyAudio or sounddevice. In telephony use your carrier’s media stream.

Second, send the audio chunk to Whisper and receive a transcript.

Third, send the transcript to your language model along with a system prompt and any tool definitions.

Fourth, stream the LLM response to TTS as it generates.

Fifth, play the audio back to the user.

Each step is a separate API call. Latency adds up, typically a few hundred milliseconds per hop. For phone call quality you want under one second end to end. For a casual voice assistant you can tolerate up to two seconds.

If you want a single WebSocket that handles all three steps, look at the Realtime API. It cuts round trip latency significantly and supports interruption natively, but it has provider specific quirks around function calling and audio formats that you need to read into.

Key Settings That Matter

The defaults work for a demo but production needs tuning. These are the dials most people ignore.

Audio format and sample rate. Whisper accepts common formats like mp3, wav, m4a, and flac. Internally it works with 16kHz mono PCM. If you record at 44.1kHz stereo you are paying for bandwidth you do not need. Resample to 16kHz mono at capture time.

Chunk size for streaming. Do not send hour long files in one request. Break them into segments, typically 30 seconds to a few minutes depending on the use case. Smaller chunks give faster time to first token, larger chunks preserve more context.

Language hint. Whisper auto detects language but you can pass a language code to force it. Forced detection is faster and avoids occasional misclassification on short clips with mixed audio.

Prompt engineering for STT. Whisper accepts an optional prompt parameter. Use it to bias toward domain specific vocabulary, names, or jargon. A prompt like “This is a call about Q3 financial results for Acme Corp” primes the model better than leaving it empty. This is one of the highest leverage knobs and almost nobody uses it.

Voice selection for TTS. The standard options are alloy, echo, fable, onyx, nova, and shimmer. Each has a different tonal quality. Test them in your actual context. A voice that sounds fine for a chatbot can feel wrong for a professional IVR.

Speed control. The TTS speed parameter accepts values from 0.25 to 4.0 with a default of 1.0. For phone agents, speeds between 1.1 and 1.2 often feel more natural than the default.

Temperature. For STT, temperature controls sampling. Lower values are more deterministic, higher values more varied. For transcription you want values close to 0. You want the most likely transcript, not a creative one.

Response format. The STT endpoint supports json, text, srt, and verbose_json. Use verbose_json when you need word level timestamps for highlighting or alignment with audio.

Where It Shines

Voice AI genuinely excels in a handful of specific scenarios.

Call center augmentation. Transcribing calls in real time, surfacing relevant knowledge base articles to agents, and generating post call summaries. This is the highest ROI deployment. Call data already exists, the workflow already exists, and the value is immediate.

Meeting transcription and summarization. Recording meetings, producing searchable transcripts, and generating action items. Tools like Otter, Fireflies, and Gong exist for a reason. Accuracy on clean English audio is in the high 90s.

Accessibility. Real time captions for video calls, audio descriptions for content, voice control for users who cannot type. This is where the technology genuinely changes lives for the people using it.

Voice agents for high volume, low complexity calls. Appointment scheduling, order status, FAQ handling. When the conversation stays within a narrow domain, voice agents handle a large fraction of calls without escalation. The limiting factor is latency, not capability.

Voice interfaces for hands free work. Field technicians, warehouse staff, drivers. Anywhere typing is impractical, voice in and voice out replaces a screen.

Where It Fails

Honest list of where this still breaks.

Heavy accents, code switching, and noisy environments. Accuracy drops noticeably with strong regional accents, mixed languages, or background noise. Microphone quality matters more than people think. A cheap headset in a quiet room beats an expensive microphone in a coffee shop.

Multiple speakers without diarization. Out of the box, Whisper does not tell you who said what. Diarization is a separate problem, often handled by a different model or provider. If you need speaker labels you either pay more or build more.

Long context retention. Voice agents forget what was said a few minutes ago unless you explicitly pass the transcript back to the LLM each turn. Token costs add up. Memory management is your problem to solve, not