ElevenLabs

The ElevenLabs source connects to a Conversational AI agent, plays its voice, and reports state and audio to the visual, including interruptions.

import { ElevenLabsVoiceSource } from "@sinua/voice/elevenlabs";

// The agent's user input format must be PCM (e.g. pcm_16000), set in the agent's settings.

// A private agent: your endpoint signs a `wss://` URL with your API key (see Credentials),
// a new one on every connect.
export const privateAgent = new ElevenLabsVoiceSource({ credentialUrl: "/api/voice/elevenlabs" });

// A public agent needs no backend: its agent id is the credential.
export const publicAgent = new ElevenLabsVoiceSource({ credential: "agent_…" });

It sends transcripts of both speakers (transcriptTiming chars), synced to the played audio by default.

Credentials

• A public agent needs only its agent id. There's no secret anywhere.
• A private agent needs a signed URL, minted by your server with your API key (signElevenLabsUrl in @sinua/voice/server). It's valid for 15 minutes, and the source asks your endpoint for a new one on every connect. See Credentials.

Packages

Good to know

• The agent's input format must be PCM. In the agent's settings, set user input audio format to a PCM format (pcm_16000 and so on). With μ-law input, the source refuses to connect and says why.
• How a turn ends. The agent's turn ends when ElevenLabs signals the reply is complete and its audio has finished playing. If the network stalls mid-reply, the visual stays in thinking rather than flipping to listening. thinking shows between the end of the user's speech and the reply's first audio, so with a fast agent it can be short.
• A conversation can't be resumed. When it closes, the source goes back to idle; connect again to start a new one.
• Start from a user gesture on the web, since the browser only allows the microphone and audio playback after one. On iOS, add NSMicrophoneUsageDescription; on Android, declare RECORD_AUDIO. Say where the audio goes: the user's voice is sent to ElevenLabs, so your iOS purpose string (and, on Android, the explanation you show before the system prompt, which carries no app text) should say so. For example: "Your voice is sent to ElevenLabs so the assistant can hear you." App Store review expects the purpose string to explain the use.