Voice and multimodal input
Add speech transcription, file attachments, and typed media content to an Angular chat.
Open your coding agent in your project's folder, or in an empty folder for a new app.This runs in a coding agent on your computer.
CopilotChat includes a microphone control and an attachment composer. Voice
input is transcribed through Copilot Runtime and inserted into the composer as
text. Multimodal input sends image or document content parts with the user's
message.
What is voice and multimodal input?#
Voice input turns recorded speech into editable text before a message is sent. Multimodal input attaches typed image or document content parts to that message, so a compatible model can reason about more than text.
Accept voice input#
No component option is required to display the microphone. The browser asks for microphone permission, records the audio, and sends it to the Runtime transcription endpoint. The resulting text remains editable before the user sends it.
Serve production applications over HTTPS so browsers can grant microphone access.
Configure Runtime transcription#
Follow the Voice backend setup to add a dedicated Runtime route and transcription service. Keep speech-provider credentials separate from credentials used for agent chat. For the Built-in Agent showcase, set:
OPENAI_TRANSCRIPTION_API_KEY=your-api-key
# Optional: defaults to https://api.openai.com/v1
OPENAI_TRANSCRIPTION_BASE_URL=https://your-speech-provider.example.com/v1The route must omit transcriptionService when the dedicated credential is
absent. Copilot Runtime then reports audioFileTranscriptionEnabled: false on
/info, and the chat hides the mic instead of exposing a control that cannot
work. When the service is configured, the browser sends recorded audio to
/api/copilotkit-voice/transcribe.
Voice requests can call the same backend tools as typed requests. Register renderers only for the exact tool names your backend exposes:
export const voiceWeatherRendererConfigs: readonly RenderToolCallConfig<VoiceWeatherArgs>[] = VOICE_WEATHER_TOOL_NAMES.map((name) => ({ name, args: z.record(z.unknown()), component: WeatherToolCard as unknown as RenderToolCallConfig<VoiceWeatherArgs>["component"], }));The renderer registration is optional. It changes how matching tool calls are displayed, not how speech is captured or transcribed.
Configure attachments#
Define an AttachmentsConfig and bind it to the chat surface. The runnable
Showcase accepts images and PDFs up to 10 MiB:
const MULTIMODAL_ATTACHMENTS: AttachmentsConfig = { enabled: true, accept: "image/*,application/pdf", maxSize: 10 * 1024 * 1024,};<copilot-chat [attachments]="multimodalAttachments" />The accept value filters the file picker and maxSize provides immediate
client feedback. They are not server-side security controls. Validate file
type, size, and content again wherever uploads are stored or processed. Use
the configuration's upload callbacks when files should go to object storage
instead of traveling inline with a message.
Send media programmatically#
For a bundled sample or custom uploader, add a user message whose content
contains normal AG-UI content parts. The Showcase constructs a text part plus
an image or document part:
export function createMultimodalMessage( spec: Pick<SampleSpec, "filename" | "mimeType" | "autoPrompt">, base64: string, size: number, id: string,): MediaAgentMessage { return { id, role: "user", content: [ { type: "text", text: spec.autoPrompt }, { type: spec.mimeType === "application/pdf" ? "document" : "image", source: { type: "data", value: base64, mimeType: spec.mimeType, }, metadata: { filename: spec.filename, size }, }, ], };}Add that message to the selected agent and run the agent. Keep the MIME type
authoritative: use an image part for images and a document part for PDFs
and other documents. The chosen model and backend converter must support each
MIME type you accept.