Voice agents might just be the next revolution in accessibility and ergonomics. They allow consumers to interact with an AI-based system through speech, both as input and output.

A typical voice agent takes in audio streams from the user, either in batch or in real-time, and leverages a language model to process them, handling any required tool call in the process. The resulting output of the process is finally outputted to the user, typically as speech too.

Voice agents differ from traditional text-based agents in the technology required to operate them:

• They need access to a real-time model API that is able to handle speech
• There interface must support streaming of input and output for real-time interaction
• They require very-low-latency serving infrastructure to maintain minimal response times for human-like conversational flow

Blaxel helps you build your first voice agent in record time. Use this template to start prototyping today.

This template uses the Realtime API from OpenAI, a low-latency API that supports multi-modal inputs and outputs. The agent is built using a LangGraph ReAct agent which enables sophisticated reasoning and action capabilities. This agent supports tool calling by connecting pre-built or custom Blaxel Functions to the agent. WebSockets protocol ensures efficient bidirectional communication of audio between the agent and client.