You Speak to AI, and It Speaks Back. What Happens in Between?

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

You might have heard of audio-to-audio AI models. But have you wondered how they work behind the scenes?

An audio-to-audio model listens to your voice and responds with a voice. Unlike a traditional speech pipeline, it does not necessarily need to convert your speech into text before generating a response.

Instead, it can represent sound as a sequence of small codes. The model processes these codes to understand the input and generate a response, then converts the output codes back into audio.

When you speak, sound waves travel through the air. The microphone captures these vibrations and converts them into an electrical signal. An audio system then samples that signal many times per second, storing each measurement as a number.

For example, audio sampled at 24,000 samples per second contains 24,000 numerical samples for every second of audio.

That means a 10-second voice message contains 240,000 samples.

That's a lot of numbers for a model to process directly. More importantly, the raw waveform contains far more detail than the model needs to represent speech efficiently.

We need a way to compress the sound while preserving the information that matters.

A codec is a system that encodes data into a representation and decodes it back. Think of it as a tool for compressing and reconstructing information.

A neural audio codec does this for sound using neural networks. Examples include SoundStream, EnCodec, and Mimi.

A neural audio codec has two main components:

Think of a song and its sheet music. The recording contains a huge amount of information, while the sheet music provides a more compact representation of what needs to be played. A musician can use it to recreate the song, although the performance may not be identical to the original recording.

Audio codes serve a somewhat similar purpose. They represent the sound in a form that a model can process more efficiently.

The encoder does not necessarily represent the entire recording as one giant code. Instead, it divides the audio into small time steps called frames.

Consider a codec that produces 12.5 frames per second. Each frame corresponds to approximately 80 milliseconds of audio.

The encoder converts each frame into a compact representation. The model can then process these representations in sequence, much like a language model processes tokens.

But there is another detail: a frame can contain more than one code.

Some neural audio codecs use a technique called Residual Vector Quantization (RVQ).

The name sounds complicated, but the basic idea is straightforward: represent the audio using several layers of codes, with each layer refining the representation provided by the previous layers.

Together, these values represent 7.34 more accurately than the first approximation alone.

RVQ works with learned codebooks and vector representations rather than literally adding decimal numbers like this. The example illustrates the idea of progressively refining a representation.

Each frame can therefore contain a stack of codes, with each code contributing additional information about the audio.

Let's use a simplified example for one second of audio.

With these assumptions, the audio is represented by approximately 100 discrete codes instead of 24,000 waveform samples.

That is a reduction of roughly 240 to 1 in the number of represented values.

The decoder can use the resulting codes to reconstruct speech that preserves much of the original audio's important characteristics.

When someone speaks, the audio carries more than the words themselves.

There are two useful ways to think about this information:

A voice assistant needs to handle both. Understanding the words is not enough if the goal is to produce natural, expressive speech. Sound details matter too.

Different audio models organize this information in different ways. For example:

However, these are not universal rules. In an RVQ codec, the first code is not automatically a semantic representation, and later codes are not necessarily dedicated to emotion or voice characteristics. That behavior depends on the model's architecture and training.

The key idea is that a model must represent enough information to understand the input and generate the kind of audio response it is designed to produce.

Once the input speech has been encoded into codes, the main model can process them and generate codes for its response.

The overall process has three main stages.

The audio encoder converts the incoming waveform into a sequence of audio codes.

The main model processes the input representation and generates a sequence of output codes. Depending on the architecture, it may also use text tokens, separate audio representations, or other intermediate information.

Like an autoregressive language model, an audio model may generate tokens one step at a time. Other architectures can organize generation differently.

The audio decoder converts the generated codes back into a waveform, producing the model's spoken response.

The simplified pipeline looks like this:

Input waveform → Encoder → Audio codes → Model generates response codes → Decoder → Output waveform

The model does not have to treat audio as raw numbers throughout the entire process. The codec gives it a compact representation that is more suitable for modeling.

A traditional speech-based assistant might use a three-stage pipeline:

Speech → Speech recognition → Text model → Text-to-speech → Spoken response

This approach works well for many applications. However, converting speech into text can discard information that is not represented in the transcript.

An audio-to-audio model can preserve and process more of this information when its architecture and training support it. That can help it produce responses with more natural timing and vocal expression.

It may also reduce the need for separate speech-recognition and speech-synthesis stages, potentially lowering latency. The actual speed advantage depends on the model and the system in which it runs.

Audio-to-audio AI models do not have to process an entire sound waveform directly. A neural audio codec converts the waveform into compact codes, a model processes those codes to generate a response, and a decoder reconstructs the output as sound.

The important idea is that audio codes give AI models a compact representation of sound that they can learn to understand and generate.

This is one of the building blocks behind more natural voice interactions, where an AI system can respond to spoken input without relying entirely on a speech-to-text-to-speech pipeline.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.

LiveReview auto-approving a low-risk change while …