VegaDūtaTutorial · Voice
How to set up a voice AI agent
In this tutorial you will give a VegaDūta agent a voice: it will listen through speech-to-text, answer through text-to-speech, and take conversations over a real phone line or directly in the browser. You choose the ear (locally-run Whisper or Sarvam), the voice (ElevenLabs or Sarvam — Indian-language voices via Sarvam's bulbul, in the console and in-app), and the wire (Twilio for phone calls, LiveKit for in-app voice).
The agent underneath is the same one you built for chat — same knowledge, same guardrails — so voice is a channel decision, not a rebuild.
By the VegaDūta team · Last updated
Step by step
Create the agent
Build the agent as you would for any channel — describe the job in the builder, add knowledge, test in the playground. Voice sits on top of an agent that already works in text, so get the answers right first.
Choose speech-to-text
Pick how the agent listens: locally-run Whisper, or Sarvam's speech-to-text. Sarvam is the natural choice for Indian-language voice notes and console or in-app voice; Whisper runs on the platform without an external provider. Phone calls through Twilio use Twilio's own relay speech with its default language.
Choose text-to-speech
Pick the agent's voice: ElevenLabs, or Sarvam — whose bulbul voices speak Indian English in live in-app voice calls today, and other languages through the agent's speak tool. Phone calls through Twilio use Twilio's relay speech instead. Connect the provider's key on the Providers screen if you have not already, then select the voice in the agent's voice settings.
For phone: connect Twilio voice
To put the agent on a phone number, connect Twilio voice in the channel settings. The agent then takes inbound calls, and DTMF menus are supported — callers can press keys to navigate, not just speak.
For browser or in-app voice: use LiveKit rooms
For voice inside a web page or app, the platform uses LiveKit rooms. A caller joins from the browser — no phone number involved — and recording is optional per room if you need a record of the conversation.
Test the call and check guardrails
Place a real test call, on the phone line or in the browser, and listen end to end: transcription accuracy, voice quality, latency. Guardrails apply to voice exactly as to any channel, so actions you gated behind approval still pause for a human even mid-call.
Troubleshooting
Voice adds moving parts; isolate which one is misbehaving before changing settings.
- Poor transcription of Indian-language voice notes or console voice: switch speech-to-text to Sarvam, which is built for them. This does not change phone calls, which use Twilio's relay speech.
- No audio on the agent's replies: confirm the text-to-speech provider's key is valid on the Providers screen and a voice is selected.
- Phone calls not connecting: verify the Twilio voice connection and the number's configuration on the Twilio side.
Frequently asked questions
Can an AI agent answer real phone calls?
Yes. Connect Twilio voice and the agent takes inbound calls on a real phone number, with DTMF menus supported so callers can press keys as well as speak. Browser and in-app voice runs separately over LiveKit rooms, with no phone number needed.
Does the voice agent support Indian languages?
In the console, in-app and for voice notes, yes: Sarvam's speech-to-text handles Indian-language audio, and the agent's speak tool can ask Sarvam's bulbul voices for a specific language. Phone calls through Twilio are different: the call speech is handled by Twilio's relay with its default language, and the provider and language are not selectable there today. Locally-run Whisper covers transcription and ElevenLabs covers text-to-speech elsewhere.
Can voice calls be recorded?
In-app and browser conversations run over LiveKit rooms, where recording is optional and set per room. Enable it when you need a record of the conversation; leave it off otherwise.
Do guardrails still apply when the agent is on a call?
Yes. Voice is a channel like any other, so the agent's guardrails and approval rules apply mid-call exactly as they do in chat. An action gated behind human approval still pauses and waits, even while a caller is on the line.
See it working in two minutes
The sandbox provisions a real tenant — describe an agent in one sentence and test it, no account, no card. Or browse ~90 industry workflow recipes to see what teams build.
