Voice assistants have evolved from clunky command-executors into near-human conversational agents. What used to require custom DSP pipelines and expensive telephony infrastructure can now be built with open-source tools and AI APIs. In this post, we’ll walk through how to build a real-time, scalable, and intelligent voice assistant using LiveKit for audio infrastructure, Whisper for speech-to-text, and GPT (OpenAI) for natural language understanding and generation.
Why build one yourself when Alexa, Siri, and Google Assistant exist? Simple: control, customization, and innovation. Prepackaged solutions rarely give you full control over data privacy, model behavior, or edge processing. Building your own assistant lets you own the stack, from raw microphone input to intelligent response.
This guide assumes you have experience with modern TypeScript or Go backends, some familiarity with WebRTC, and an interest in combining real-time audio processing with cutting-edge AI.
System Architecture Overview
At a high level, the system is composed of four core components:
- LiveKit Client (browser or mobile): Captures and sends audio in real time via WebRTC.
- LiveKit Server (SFU): Routes audio streams to subscribers and backend workers.
- Transcription Pipeline (Whisper): Converts audio streams to text, possibly using chunked streaming.
- Conversational Intelligence (GPT): Processes text and generates natural language responses.
The flow is straightforward in principle but requires careful orchestration in practice:
- A user speaks into their mic.
- Audio is routed to a backend worker via LiveKit’s SFU.
- The audio buffer is streamed through Whisper, either running locally on GPU or using OpenAI’s API.
- Transcribed text is fed into GPT (chat-completion model), which returns a contextually aware response.
- (Optionally) The response is synthesized back into audio via TTS and streamed to the client.
Latency is the invisible enemy here. Without careful buffering, VAD (voice activity detection), and stream chunking, your assistant will feel sluggish or unresponsive. We’ll address each of those bottlenecks as we go.
Setting Up the Real-Time Audio Pipeline
Step 1: Install LiveKit Server
This spins up a development-mode LiveKit instance with hardcoded API keys. You’ll use this to accept WebRTC connections from the browser.
Step 2: Integrate with Client SDK
In your frontend, you’ll use @livekit/client to capture audio and join a LiveKit room:
LiveKit handles all WebRTC negotiation behind the scenes, ICE, SDP, and NAT traversal. Once connected, the client publishes a MediaStreamTrack that the server receives and routes.
Real-Time Audio Capture in the Browser
Capturing microphone input reliably in the browser is a non-trivial task when latency and fidelity matter. WebRTC provides the backbone for this, but LiveKit simplifies a lot of the boilerplate around connection management, bandwidth adaptation, and track pulishing.
Mic Permissions and Capture
You’ll need to request audio permissions from the browser and handle potential denials gracefully:
This returns a MediaStream with one or more MediaStreamTrack objects, typically just the microphone in this case.
You can then feed this directly into the LiveKit LocalAudioTrack object:
LiveKit automatically encodes this into Opus and transmits it over WebRTC with congestion control and jitter handling.
Audio Constraints
For voice assistants, consider setting specific mediaConstraints:
This aligns the captured audio with the input profile Whisper expects, reducing preprocessing needs downstream.
Server-Side Audio Handling
Once audio tracks are flowing through LiveKit, you need a mechanism on the server to extract audio frames, decode them, and pipe them into a buffer for transcription.
LiveKit doesn’t natively decode audio on the server, instead, you need to subscribe to audio tracks, decode Opus frames, and aggregate them into PCM chunks.
Approaches:
- LiveKit Webhook + Media Relay – Attach a bot participant to the room that subscribes to audio tracks. This bot can run in Node.js, Go, or Rust, and receive audio via the LiveKit SDK.
- Using Egress API – For full sessions, LiveKit supports routing audio/video to an RTMP/S3/Websocket endpoint. This is overkill for live transcription but useful for post-call processing.
For real-time transcription, you’ll likely build a custom LiveKit client that does the following:
- Joins the same room as a silent subscriber
- Subscribes to remote participants’ audio tracks
- Uses AudioContext or FFMPEG bindings to decode Opus to PCM/WAV
- Buffers 1–2 seconds of audio at a time for streaming into Whisper
You may want to apply Voice Activity Detection (VAD) to avoid redundant silence transcriptions, we’ll cover that later.
Integrating Whisper for Transcription
Now the interesting part: converting audio buffers into usable text.
Option 1: OpenAI Whisper API
If you’re optimizing for speed of implementation and can tolerate cloud latency, the OpenAI Whisper API is extremely easy to use.
You POST an audio file (or stream it in chunks with multipart/form-data) and get back a JSON transcript.
Option 2: Local Whisper (via openai-whisper or faster Whisper.cpp)
For lower latency and on-prem security, use the Whisper model locally:
Or use the blazing-fast whisper.cpp project to transcribe PCM buffers in real-time with C/C++ performance.
Chunking Audio in Real-Time
Whisper isn’t designed to run on infinite audio streams, it works best with sliding windows (5–10s). So you’ll need to buffer the server audio into fixed intervals and push them into Whisper as overlapping windows to maintain context:
- 5s chunk → transcribe
- 2.5s overlap with next chunk → re-transcribe
- Deduplicate transcriptions via confidence scoring
This prevents the assistant from losing mid-sentence context or dropping words at boundaries.
Optimizing Whisper Performance
While Whisper is impressive in transcription quality, it’s computationally intensive and not tuned for real-time processing out of the box. If you’re deploying in latency-sensitive contexts (e.g., real-time assistants), you’ll need to tune the transcription pipeline for both performance and accuracy.
Voice Activity Detection (VAD)
Feeding silence or noise into Whisper is a waste of GPU cycles. You can gate audio chunks using VAD to ensure only speech-active segments are transcribed.
Use tools like webrtcvad, py-webrtcvad, or more sophisticated neural VADs like silero-vad:
Apply VAD before each chunk enters the transcription pipeline. If no speech is detected, drop the segment.
Sliding Windows for Context Preservation
Whisper lacks persistent memory between calls. To avoid truncating or misinterpreting utterances near segment boundaries, implement a sliding window:
- Transcribe 5s chunks
- Use 2s overlap between windows
- Post-process outputs to remove duplicate or incomplete phrases
This technique increases perceived continuity, especially when users pause mid-sentence or self-correct.
Stream-Friendly Whisper Implementations
- whisper-live: A project that wires real-time audio ingestion, buffering, and streaming into local Whisper using websockets.
- whisper.cpp + VAD + ring buffer: The ultimate DIY stack for running low-latency inference at the edge with minimal dependencies.
Feeding Transcribed Text into GPT
Once you have reliable transcripts, it’s time to bring the assistant to life using GPT. The raw text from Whisper becomes the input prompt to GPT, along with optional context and memory.
Choosing the Right GPT Variant
- GPT-3.5-turbo: Fast and cheap; ideal for rapid iteration or hobby use.
- GPT-4: Slower, more accurate, and far more context-aware, recommended for assistants expected to maintain multi-turn context.
- Local Models: Projects like LM Studio, Mistral, and OpenHermes support running LLMs locally, but with limited context depth and capabilities.
Basic Prompt Strategy
Every transcription becomes a user message in a structured prompt chain:
Send this prompt to the /chat/completions endpoint via OpenAI or a compatible API, and receive a response as a string. This can then be sent back to the client or turned into TTS output.
Managing Temporal Context
To keep the assistant aware of prior exchanges:
- Maintain a sliding chat history buffer, trimmed to fit within the model’s token limit.
- Optionally summarize older exchanges and store them in a semantic memory store (e.g., via vector DB like Pinecone or Chroma).
- Rehydrate past memory during each GPT invocation to emulate persistent cognition.
Building a Conversation Manager
Now that GPT is producing replies, you’ll need an intermediate conversation manager to coordinate everything: intent handling, GPT invocation, TTS control, and turn-taking logic.
Intent Parsing vs Freeform Dialogue
Depending on your use case, your assistant may need:
- Freeform responses (chit-chat, open Q&A)
- Intent detection (structured commands like “turn off the lights”)
You can blend both approaches by embedding a lightweight intent classification step before GPT, using a small model or regex-based classifier.
Example intents:
- GET_WEATHER – “What’s the weather?”
- CONTROL_DEVICE – “Turn on the lamp”
- SMALL_TALK – “Tell me a joke”
Based on the intent, the conversation manager can:
- Route the query to GPT
- Trigger custom code (e.g., smart home integration)
- Decide whether to reply via text or voice
Handling Interruptions and Overlap
In real-world use, users interrupt themselves, revise commands, or talk over the assistant.
Use:
- Silence detection (500ms idle = safe to speak)
- Streamed GPT tokens – stream responses incrementally to allow interruption mid-output
- Abortable pipelines – let users say “Stop” or “Wait” to cancel current action
Text-to-Speech (Optional Enhancement)
While a voice assistant can function fine with just visual feedback, nothing beats a spoken response for hands-free, fluid interactions. Implementing TTS bridges the final step in your voice loop.
TTS Options
Cloud APIs
- Google Cloud TTS, Amazon Polly, Azure Speech, ElevenLabs
- Pros: Natural-sounding, multilingual, high-quality voices
- Cons: Requires internet, cost per character/second, privacy concerns
Open Source TTS
- Coqui TTS, Mozilla TTS, ESPnet, VITS-based models
- Pros: Local inference, customization, offline support
- Cons: Requires GPU for real-time use, limited speaker quality out-of-box
Example with Coqui:
Streaming Voice Back via LiveKit
Once synthesized, you can push TTS audio back into LiveKit:
- Convert WAV/PCM to Opus
- Wrap it into a MediaStreamTrack or RTP stream
- Publish it to the room as a bot participant or audio track
This enables bi-directional, human-like conversation where GPT replies are heard, not read.
Latency Optimization Techniques
For a smooth user experience, low latency is critical. Ideally, users should speak and hear a response in under 1–1.5 seconds total roundtrip.
Async Pipeline Execution
Run the following concurrently:
- Audio buffer ingestion
- VAD detection
- Whisper inference
- GPT call
- TTS generation
Use event queues, streaming APIs, and micro-batching to reduce idle time between phases. Each processing stage should push data forward without waiting for the full utterance.
Reduce Token Bloat
When using GPT, trimming context intelligently can improve speed:
- Limit system prompt verbosity
- Truncate long chat history
- Compress old turns via summarization
Also, set max_tokens aggressively (~50–100) if your assistant gives short, command-style replies.
Deployment Considerations
By now, you have a working assistant on localhost. Time to think about moving to staging or production.
Hosting LiveKit
- LiveKit Cloud: Fully managed with global latency routing, perfect for fast MVPs.
- Self-hosted: Run via Docker or Helm on your infra for full control. You’ll need to handle TURN servers, scaling, and token management.
Minimum requirements:
- UDP ports 7881/7880 open
- TURN/STUN fallback for NAT traversal
- Persistent Redis + Postgres for signaling layer
Whisper Hosting:
- Cloud Whisper API: Pay-per-minute and avoids GPU provisioning.
- Self-hosted Whisper: Use NVIDIA A10 or better for medium and large models.
- whisper.cpp: If latency is critical, compile and run the C++ version on CPU or Apple Silicon (Metal support).
GPT Model Hosting
Options:
- OpenAI API: Production-ready, scalable, but data leaves your infra.
- OpenRouter or Groq: API-compat layers for cost or performance trade-offs.
- Local LLMs: Requires quantized models (e.g., gguf format) and a solid inference backend (Ollama, vLLM, etc).
Security and Privacy
Voice assistants inherently deal with sensitive, often personally identifiable information (PII). If you plan to deploy beyond personal use or internal tools, harden every layer of the stack.
Audio & Transcript Handling
- Encrypt audio streams in transit via WebRTC/DTLS (handled by LiveKit).
- Avoid persisting raw audio unless explicitly needed for QA or logging.
- For GDPR/CCPA compliance, log opt-ins and implement data deletion APIs.
Whisper/GPT API Security
- Don’t expose your OpenAI API keys client-side, route all GPT calls through your backend with proper rate limiting and usage tracking.
- For local models, sandbox execution (especially Whisper or GPT running Python) using Docker or systemd-nspawn.
Token Safety
LiveKit tokens (JWT) should:
- Be time-bound (e.g., 15 mins)
- Be scoped per-room/user
- Use cryptographically strong secret rotation policies
Scaling the System
Once your assistant is running, supporting real users means supporting real concurrency.
LiveKit Rooms
Each conversation can happen in an isolated LiveKit room, letting you support multiple simultaneous assistants, users, or even multi party voice bots.
- Use room webhooks to track joins/leaves.
- Clean up inactive rooms to save memory.
Worker Processes
Split responsibilities across services:
- Transcriber workers: Ingest audio, transcribe using Whisper
- NLP workers: Handle GPT requests and memory state
- TTS workers: Queue up responses and synthesize voice
- Orchestrator: Coordinates all phases and manages context per session
Use message queues like Redis Pub/Sub, NATS, or Kafka for coordination and retry logic.
Load Balancing
Horizontally scale:
- Whisper inference nodes
- GPT proxy servers
- LiveKit SFUs with shared Redis/Postgres backend
Use k8s + autoscaling policies to respond to traffic spikes without degrading UX.
Debugging and Monitoring
To keep your assistant reliable, observability is non-negotiable.
Key Metrics to Track
- Audio stream latency (LiveKit round-trip)
- Whisper inference time per chunk
- GPT completion time (token throughput)
- TTS synthesis duration
- Total voice-to-voice latency
Logging & Tracing
- Annotate each message turn with full trace_id
- Log transcription accuracy (e.g., Levenshtein diff vs ground truth)
- Capture GPT failures, empty responses, or rate-limiting events
- Use tools like Grafana, Prometheus, and OpenTelemetry for full trace visibility
Conclusion
Building a voice assistant from scratch using LiveKit, Whisper, and GPT isn’t just technically feasible, it’s developer-friendly, scalable, and surprisingly elegant.
With the right architecture, you can achieve:
- Real-time transcription at <500ms delay
- Stateful, context-aware conversations powered by GPT
- Optional TTS for full-duplex speech interaction
- Modular components that scale independently
Whether you’re designing a customer support agent, an embedded assistant for a product, or a hands-free productivity tool, this stack gives you the flexibility and control that black-box solutions never will.
Keep in mind: latency wins user trust. Privacy earns adoption. And great conversation design drives retention.