Voice overview

A voice source turns a conversation into two streams the visual understands:

• the agent's state (initializing, idle, listening, thinking, speaking), which picks the look through your states;
• the audio: an overall level and per-band levels, which drive audioLevel (an orb's breathing) and the bars and waveforms of a signal.

Give the view a source and it does the rest: it follows the state, eases the audio at display rate so nothing steps, shows the mute cue when you set it, and flashes briefly when the user interrupts the agent (barge-in).

The two talking states move in opposite directions, which is what makes them tell apart at a glance: while the user speaks, the visual is drawn inward by the mic, and while the agent speaks it swells outward with its voice. That behaviour ships with the engine, so a visual follows a conversation without a design file; an FX Spec then changes whatever you want on top of it.

import { mount } from "@sinua/web";
import { LocalMicVoiceSource } from "@sinua/voice/mic";
import voiceOrb from "../spec/voice-orb.fxspec.json";

const canvas = document.querySelector<HTMLCanvasElement>("#orb")!;
const voice = new LocalMicVoiceSource();

// The view binds the source: level and bands drive the visual, and the source's
// AgentState picks the spec's lifecycle state. It never connects the source itself.
const fx = mount(canvas, { spec: voiceOrb, voice });

// Connect from a user gesture: the mic prompt and audio playback both need one.
document.querySelector("#talk")!.addEventListener("click", async () => {
  try {
    await voice.connect();
  } catch (err) {
    console.error("voice failed to start", err); // e.g. the mic permission was denied
  }
});

// On teardown: voice.disconnect(); fx.destroy();
export { fx, voice };

Sources

Each vendor source is its own package on iOS and Android, and its own entry point of @sinua/voice on the web (/livekit, /openai, /gemini, /elevenlabs, /mic, /tone), so your app only includes the SDKs it uses. On the web, install @sinua/voice (and livekit-client only if you use LiveKit), pass the source to the view as voice, and call connect() from a user gesture, such as a button tap: browsers only allow audio and the microphone after one. All of them report the same state names, the same audio shape, and interruption where the vendor signals it.

Credentials: never ship an API key

A voice source never needs your vendor API key. Keep the key on your server, where one endpoint mints a short-lived credential per session, and point the source at it:

• OpenAI: an ephemeral client secret (ek_…).
• Gemini: an ephemeral auth token (auth_tokens/…).
• ElevenLabs: a signed URL for a private agent, or just the agent id for a public one.
• OpenAI GPT-Live: no client credential at all; your server opens the session and answers the app's offer.
• LiveKit: a room token from your backend, or a room you've already connected.

The endpoint is yours, so you decide who gets a session and with which model, voice and instructions. @sinua/voice/server mints all four, the repository has ready-made endpoints for Next.js, Express, Cloudflare Workers and Supabase, and npx @sinua/voice dev-proxy serves them on your machine while you try things. See Credentials.

How the states look, and changing it

Every pattern follows the four states with the same built-in profile, on top of its own design:

The states add no particles.

A character has its own state language instead: it looks at you while listening, looks away while thinking and talks while speaking.

Your overrides go on top of the profile, in every state and on every platform. To make a view quieter still, soften the glow and drop the pulse:

import { mount } from "@sinua/web";
import { LocalMicVoiceSource } from "@sinua/voice/mic";

const canvas = document.querySelector<HTMLCanvasElement>("#signal")!;
const voice = new LocalMicVoiceSource();

// The voice states add a glow and a pulse; your overrides go on top of them,
// in every state. Here: a softer glow and no pulse.
export const fx = mount(canvas, {
  pattern: "waveform",
  voice,
  overrides: { glowStrength: 0.15, pulseStrength: 0 },
});

The keys are the materials' master strengths: glowStrength, pulseStrength (0 turns one off). With a spec instead of a pattern, the file decides: put the same values in its states.

Driving the visual yourself

If your app already listens to the agent, skip the source: pass a VoiceOverrides you feed yourself as voice. Call push({ level, bands }) for the audio, setState(...) for the agent's state and interrupt() for a barge-in. The view's easing and cues still apply.

The mute cue is yours to turn on, since sources don't report mute: voiceOptions: { muted: true } on the web, or the muted option of VoiceOverrides on iOS and Android.