Vocal Isolation and Voice Conversion: How AI Covers Actually Work Under the Hood

I'm a music blogger, not an ML engineer, but I've spent enough time running AI cover pipelines (and reading the repos behind them) that I wanted to write down how the pieces fit together. If you've ever wondered what happens between "upload a song" and "hear it sung by a different voice", this is the plain-English version, with pointers to the open-source projects you can run yourself.

An AI cover pipeline has two very different ML problems chained together:

Then there's a boring but important third stage: put the converted vocal back on the instrumental.

A mixed song is a single waveform, the sum of every instrument. Vocal isolation asks: given that sum, estimate the vocal component. It's an underdetermined problem, since infinitely many combinations of sources could add up to the same mix. Models solve it by learning priors from thousands of multitrack recordings where the true stems are known.

The classic approach works in the time-frequency domain:

Deezer's Spleeter (2019) popularised this with a U-Net over spectrograms and pretrained 2-, 4- and 5-stem models. It's fast and still useful, but reusing the mixture phase is one reason older separators sound "phasey" or watery.

Demucs, from Meta's research team, started as a waveform-domain model: an encoder-decoder that takes raw audio in and outputs raw audio per source, so it learns phase implicitly. Later versions went hybrid, combining a spectrogram branch and a waveform branch, and the newest variant (HT Demucs) adds a transformer between them. The standard models output four stems (drums, bass, other, vocals), and there's an experimental six-stem model that adds guitar and piano.

Trying it locally is about two commands:

pip install demucs demucs --two-stems = vocals my_song.wav Enter fullscreen mode Exit fullscreen mode --two-stems=vocals gives you vocals.wav and no_vocals.wav, which is all a cover pipeline needs.

If you prefer a GUI, Ultimate Vocal Remover (UVR) is a free open-source desktop app that wraps several model families (MDX-Net, Demucs, VR architecture) and lets you ensemble them.

Knowing the failure modes saves you a lot of wasted compute downstream:

That's why a practical pipeline usually adds a cleanup stage after separation: dereverberation, echo removal and denoising. These are often separate models trained specifically for each artefact.

Voice conversion tries to split a vocal into:

Then it swaps the timbre and resynthesises audio.

Modern open-source systems don't use hand-made phoneme recognisers. They use self-supervised speech models such as HuBERT or ContentVec, which turn audio into frame-level embeddings that capture linguistic content. ContentVec in particular was designed to reduce how much speaker identity leaks into those embeddings, which is exactly what conversion needs.

Singing needs accurate F0, otherwise the converted voice warbles or jumps octaves. Pipelines use pitch trackers such as CREPE, Harvest or RMVPE (a model designed to estimate vocal pitch even from polyphonic music). Most tools let you shift the extracted pitch by a number of semitones, which is how you move a song into a target voice's comfortable range, for example when converting a male vocal to a female voice model.

so-vits-svc combines a SoftVC content encoder with a VITS -style decoder, a model architecture that pairs a variational autoencoder with adversarial training and a neural vocoder to produce waveforms directly.

RVC (Retrieval-based Voice Conversion) builds on similar VITS-style synthesis but adds a retrieval trick: during training it stores content features from the target singer in an index (using Faiss). At inference, each incoming content frame is blended with its nearest neighbours from that index. Pulling the input features toward real features from the target voice reduces timbre leakage from the source singer. An "index rate" parameter controls how strong that blending is.

Training an RVC voice model needs only minutes of clean, dry audio of the target voice, which is why so many community models exist. Data quality matters far more than quantity: reverb, music bleed or noise in the training set gets baked into the voice.

mix.wav └─► separation (Demucs / MDX) ─► vocals.wav + instrumental.wav └─► cleanup (de-reverb, de-echo, denoise) ─► vocals_dry.wav └─► F0 extraction + content encoding └─► voice conversion (RVC / so-vits-svc) ─► vocals_converted.wav └─► mix with instrumental (+ new reverb) ─► cover.wav Enter fullscreen mode Exit fullscreen mode Running this yourself is a great learning project, but it means managing GPUs, model checkpoints and a few Python environments that don't always agree on dependency versions.

If you'd rather not maintain that stack, hosted tools wrap the same stages behind a UI. I've been using Voicestars for this, mainly because each stage is exposed as its own tool, which maps nicely onto the pipeline above:

Usage is credit-based. The pricing page lists the per-feature cost, for example 50 credits per AI cover and per-minute rates for the audio tools. Voicestars doesn't publish which model architectures it uses internally, so I can't tell you whether it runs Demucs or RVC under the hood. What I can tell you is that the same input-quality rules apply: clean separation in, clean conversion out.

Run demucs --two-stems=vocals on a track you own, listen to the vocal stem solo, then push the same file through the Voicestars stems splitter and compare. Hearing where each one leaks is the fastest way to understand what these models are really doing.

Disclosure: this article was written in partnership with Voicestars.