OpenAI Realtime

The OpenAI source opens a Realtime session over WebRTC with a short-lived client secret from your server, plays the agent's voice, and reports state and audio to the visual. When the user starts speaking over the agent, the visual flashes: the server cancels the response and the source reports the interruption.

import { OpenAIRealtimeVoiceSource } from "@sinua/voice/openai";

// Your endpoint mints a short-lived `ek_…` with your API key, which never reaches the
// browser (see Credentials). The source POSTs to it on every connect and reconnect,
// since an `ek_` works once. The model, voice and instructions are set on the server.
export const voice = new OpenAIRealtimeVoiceSource({ credentialUrl: "/api/voice/openai" });

It sends transcripts of both speakers (transcriptTiming none), synced to the played audio by default.

The user's side is off by default. It needs the session's input transcription, which OpenAI bills per minute, so Sinua never turns it on by itself. To opt in, pass transcribeUser: "gpt-4o-mini-transcribe" (or another model) on the source, on every platform; it's turned on once per session when you subscribe. Without it you get the assistant's side, plus the user's if your backend already enabled input transcription.

Credentials

Your server mints an ephemeral client secret (ek_…) with your API key, and sets the session's model, voice and instructions while doing it: mintOpenAIRealtimeCredential in @sinua/voice/server. Point the source at that endpoint with credentialUrl, the same on every platform. See Credentials.

Your own backend (sideband)

To keep tools and transcripts on your server, let it open the call: point callsUrl (callsURL on iOS) at your endpoint instead of OpenAI's. The source sends its SDP offer there; your server opens the Realtime call with your key (a multipart sdp + session request), answers with the SDP text and a 2xx status, and joins the same call over OpenAI's sideband WebSocket.

import { OpenAIRealtimeVoiceSource } from "@sinua/voice/openai";

// Your own backend opens the Realtime call with your key, so it can join it on OpenAI's
// sideband connection for tools and transcripts. The source sends its SDP offer there
// instead of to OpenAI, with your own short-lived token (any shape, never `sk-…`).
export const voice = new OpenAIRealtimeVoiceSource({
  callsUrl: "/api/voice/openai-call",
  credentialUrl: "/api/voice/openai-call-token",
});

The credential then goes to your endpoint, not OpenAI's: your own short-lived token, in any shape, but never a raw sk-… key. With OpenAI's default endpoint it stays an ek_…. React Native doesn't pass a callsUrl yet.

Requests, errors and audio

• Web, your request: headers adds headers to the call to your endpoint, as an object or a function called per request (for a CSRF token, say), and fetch replaces the fetch it uses.
• Web, errors: a failed request rejects with a VoiceHttpError that carries the status and the response body, so you can tell a 401 from a 500.
• iOS, the audio session: Sinua sets the session up before the call. audioSession: .speaker (the default) plays through the speaker, .receiver through the earpiece, and .unmanaged leaves the session to your app.
• When speaking ends: after a short reply the agent counts as speaking for about 0.7 s more, after a long answer up to 1.7 s, so the state doesn't flicker between phrases. Talking over it (barge-in) is still instant.

Packages

Good to know

• WARP (warp: true, off by default) pre-negotiates the event channel and sends its dcid, saving a round trip at startup; on iOS and Android it also turns on libwebrtc's DTLS 1.3, SNAP and SPED field trials. A callsUrl backend must forward dcid to OpenAI unchanged.
• One secret per connection. An ek_… secret works once. Give the source a way to fetch a fresh one (credentialUrl, or a provider), not a fixed string, or a dropped call can't reconnect.
• No second WebRTC on native. The iOS and Android sources use the WebRTC build that ships with LiveKit's SDK, so an app that has both doesn't carry two copies.
• Start from a user gesture on the web, since the browser only allows the microphone and audio playback after one. On iOS, add NSMicrophoneUsageDescription; on Android, declare RECORD_AUDIO. Say where the audio goes: the user's voice is sent to OpenAI, so your iOS purpose string (and, on Android, the explanation you show before the system prompt, which carries no app text) should say so. For example: "Your voice is sent to OpenAI so the assistant can hear you." App Store review expects the purpose string to explain the use.