Transcripts

A voice source can report what both speakers say as they say it, so you can put captions under the visual. It's a data layer only: Sinua draws no captions. Display only: Sinua keeps nothing beyond the current turn's text and sends it nowhere.

import { SharedVoiceSource } from "@sinua/core";
import { OpenAILiveVoiceSource } from "@sinua/voice/openai";

// Captions under the visual: the text of both speakers, live. Sinua draws nothing here,
// keeps nothing beyond the current turn and sends the text nowhere.
const bubbles = new Map<string, string>();

export const voice = SharedVoiceSource.of(new OpenAILiveVoiceSource({ sessionUrl: "/api/voice/openai-live" }));
export const stop = voice.onTranscript((u) => {
  bubbles.set(u.turnId, u.text); // `text` is the whole turn so far: replace, don't append
  if (u.final) console.log(`${u.role}: ${u.text}${u.truncated ? " (cut off)" : ""}`); // once per turn
});
// Subscribe before connect() and the first turn isn't missed.
void voice.connect();

• Where: onTranscript on a source or on SharedVoiceSource (Web; it returns its unsubscribe), listenTranscript on SharedVoiceSource (iOS, Android), and onTranscript on a React Native voice handle. Subscribing before connect() catches the first turn.
• Callbacks arrive on the main actor (iOS), the Main dispatcher (Android) or the event loop (Web), like every other callback.

The update

Every update has the same shape on every platform and vendor: { role, text, final, turnId, truncated?, startMs?, endMs? }.

• text is the whole turn so far, never a diff: replace what you show, don't append.
• turnId ("u3", "a4") stays the same for the whole turn and keeps counting for the source's lifetime, reconnects included.
• final: true comes exactly once per turn, and nothing follows it.
• Both speakers at once. User and assistant turns are independent, so both can grow at the same time.
• Barge-in. When the user cuts the assistant off, its turn ends with one update, truncated: true and final: true, whose text is the part that was actually played. An unspoken word doesn't flash up.
• When a turn ends. An assistant turn ends when the source stops speaking, after its short quiet tail, so a natural pause doesn't split it. A "let me check" followed by the answer gives two assistant turns.
• Text is NFC-normalized and otherwise left as the vendor sent it.

Synced or raw

By default the assistant's text appears with the played audio, word by word. With syncToAudio: false (Web option, iOS init, Android constructor) it appears as soon as the vendor sends it, with the vendor's own timing in startMs / endMs. The user's text shows as it arrives in both modes.

How close the synced text follows the audio on GPT-Live, measured against word timings (which carry ±100–200 ms of their own):

GPT-Live's word timings are approximate and how far they run ahead varies from session to session, so treat the English figures as a known limit. syncToAudio also exists on ElevenLabs, Gemini Live and OpenAI Realtime.

Which sources send transcripts

supportsTranscript says whether a source sends them, and transcriptTiming how it times the text.

• Realtime sends only the assistant's side by default. The user's side needs the session's input transcription, which OpenAI bills per minute, so it is off unless you opt in: transcribeUser: "gpt-4o-mini-transcribe" (or another model) turns it on once per session when you subscribe. If your backend already set input transcription on the session, its user text comes either way; Sinua never turns it on or off by itself.
• Gemini asks for output (and input) transcription in the Live setup. Its user transcript can come late, or not at all, on long speech.
• LiveKit: a Room takes one handler per text-stream topic. If your app already reads lk.transcription on a Room you attached, the source's transcript stays empty (with a console warning); read your own handler instead. Raw mode is the agent's setting there, not the source's.
• Raw mode shows the text as it arrives on every vendor. At a barge-in, chars keeps the characters played by then and none the text that had arrived.
• Only GPT-Live has been measured on live sessions; the others' timing follows from what each vendor sends.

For your own voice source, @sinua/voice exports TranscriptAssembler: feed it your vendor's events and it gives the same updates.