Skip to main content

Function: audio()

audio(input, mode?, options?): UnifiedMessageInput

Defined in: packages/core/src/multimodal.ts:317

Create an audio input with a simple, flexible API

Supports multiple input types with automatic format detection:

  • Blob/File: Auto-detects format from MIME type
  • MediaAudio: Backward compatible explicit format
  • MediaStream: Live audio streams from microphone or other sources

Parameters​

input​

Blob | MediaAudio | MediaStream | File | { data: string | Uint8Array<ArrayBufferLike> | Blob | MediaStream; format: AudioFormat; }

mode?​

"transcribe" | "analyze"

options?​

AudioOptions

Returns​

UnifiedMessageInput

Examples​

const fileInput = document.querySelector('input[type="file"]');
const audioFile = fileInput.files?.[0];
if (audioFile) {
  bot.process(audio(audioFile, 'transcribe'));
}
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
bot.process(audio(stream, 'transcribe'));
const mediaRecorder = new MediaRecorder(stream);
mediaRecorder.ondataavailable = (e) => {
  bot.process(audio(e.data, 'transcribe'));
};
audio(audioBlob, 'transcribe', { minSize: 2048 }) // ~0.2 seconds
const base64Audio = "...";
bot.process(audio({ data: base64Audio, format: 'mp3' }, 'transcribe'));