A Journey Through Sound: Building an AI Transcription Pipeline with Dograh

“Language is humanity’s most spectacular open source project.”-Gretchen McCulloch

One person is explaining a technical problem. Someone interrupts with a solution. Another person switches from English to Hindi. There is background noise, a few seconds of silence, and sometimes two people speak at the same time.

Now take that one-hour conversation and ask a computer to turn it into a clean, searchable transcript.

It sounds simple: Audio → Speech-to-Text → Text

A useful transcription system has to deal with audio quality, silence, multiple speakers, timestamps, languages, technical vocabulary, and unreliable network or model responses.

That is where an AI transcription pipeline becomes useful.

In this article, we will explore how to build such a pipeline using Dograh as the voice-AI orchestration layer, along with audio processing, Voice Activity Detection (VAD), speech-to-text, transcript normalization, speaker handling, and downstream meeting intelligence.

Automatic Speech Recognition (ASR) converts spoken audio into text. A basic system looks like:

Audio ↓ Speech-to-Text Model ↓ Text For a real application, we usually need more stages: Meeting Audio ↓ Audio Validation ↓ Voice Activity Detection ↓ Speech-to-Text ↓ Speaker / Timestamp Processing ↓ Language Processing ↓ Transcript Cleanup ↓ Structured Transcript ↓ Summary / Action Items

VAD determines which parts of an audio recording contain speech. For example:

[Silence] [Speech] [Silence] [Speech] [Noise] The pipeline can concentrate on the speech regions instead of treating the entire recording equally.

The STT model converts the detected speech into text.

Finally, the transcript can be cleaned, normalized, translated, summarized, or converted into structured data.

“Speech and language technology … is really the hardest AI challenge we’re facing.”- Xuedong Huang

That observation explains why a seemingly simple feature such as “convert this meeting recording into text” quickly becomes a systems problem.

This separation makes the system easier to test and maintain.

A person can listen to a meeting and understand context naturally. A machine receives an audio signal. That creates several challenges.

Poor audio quality can directly affect transcription accuracy.

A long meeting may contain many periods where nobody is speaking. Processing every second unnecessarily increases work.

Consider: Saad: We can deploy the service tomorrow-

The system has to determine what was said and, depending on the pipeline, who said it.

A meeting may contain: Saad: Let's deploy the service today.

Sneha: Pehle worker ko test karna chahiye.

Manthan: Yes, then we can check the logs.

A multilingual pipeline needs to handle this deliberately.

Software meetings contain terms such as:

FastAPI PostgreSQL Docker Redis PyTorch Kubernetes WebSocket

These can be difficult for speech recognition systems, especially when the audio itself is noisy.

This is why transcription quality depends on the complete pipeline, not only the model.

Dograh is an open-source, self-hostable voice-AI platform. Its architecture separates different parts of a voice workflow, including speech-to-text, LLM processing, text-to-speech, and telephony.

A simplified view is: Audio ↓ STT ↓ LLM ↓ TTS ↓ Response

Dograh also supports configurable STT providers, allowing developers to choose the speech technology appropriate for their application.

This is useful because the requirements of a customer-support voice bot may be different from those of a meeting transcription system.

The important architectural idea is separation of responsibilities. Dograh can handle voice-AI orchestration while the application can handle domain-specific transcript processing.

Think about a detective investigating a complicated case.

The detective does not listen to every conversation once and immediately write a final report.

Instead: Evidence ↓ Filter useful information ↓ Identify people ↓ Record timestamps ↓ Connect events ↓ Write report

An AI transcription pipeline works similarly. The audio is the evidence. VAD identifies useful speech. STT converts speech into text. Speaker processing identifies participants. Timestamps tell us when something happened. The final transcript becomes structured evidence for another system. That system might then create:

Summary Decisions Action Items Deadlines

The transcript is therefore not always the final product. It is often the foundation for everything that comes after it.

For a meeting-notes application, a practical architecture can look like this:

This architecture also makes debugging easier.

If something goes wrong, we can ask: Was the recording bad? ↓ Did VAD fail? ↓ Did STT fail? ↓ Did speaker mapping fail? ↓ Did the LLM receive bad data?

Instead of simply saying: “The transcription is wrong.”

A preprocessing layer can convert audio into a predictable format. For example:

ffmpeg -i meeting.wav \ -ar 16000 \ -ac 1 \ -c:a pcm_s16le \ meeting-normalized.wav Enter fullscreen mode Exit fullscreen mode This converts the recording to:

The exact format should depend on the STT system being used. The goal is consistency.

Step 2: Detect Speech A VAD system can identify speech regions:

speech_regions = vad.detect(audio) for region in speech_regions: print( f"{region.start:.2f}s - " f"{region.end:.2f}s" ) Enter fullscreen mode Exit fullscreen mode Instead of treating an entire recording as useful speech, the pipeline can focus on regions where people are actually talking. This becomes especially useful for long recordings.

The speech segments are passed to the configured STT system. Dograh’s architecture treats STT as a separate component of the voice workflow, allowing different providers and models to be integrated into the larger system. This separation also makes it easier to test different transcription approaches without rewriting the application.

Different providers may return different response formats. Your application should convert them into a consistent internal structure.

from dataclasses import dataclass, asdict @dataclass class TranscriptSegment: speaker: str start: float end: float text: str def normalize_segments(raw_segments): result = [] for segment in raw_segments: result.append( TranscriptSegment( speaker=segment.get("speaker", "unknown"), start=float(segment["start"]), end=float(segment["end"]), text=segment["text"].strip(), ) ) return [asdict(item) for item in result] Enter fullscreen mode Exit fullscreen mode How It Works

{ "speaker": "spk_0", "start": 12.4, "end": 15.8, "text": "Let's deploy the service." } Enter fullscreen mode Exit fullscreen mode Our application converts it into a predictable format.The rest of the system does not need to know which provider generated the transcript.That makes the architecture easier to maintain.

A plain transcript might look like: Let's deploy the service tomorrow. We should test the worker first. The API is returning a timeout. A structured transcript is much more useful: [00:12] Saad Let's deploy the service tomorrow.

[00:18] Sneha We should test the worker first.

[00:25] Manthan The API is returning a timeout. Now the same data can support:

Speaker Diarization vs Speaker Identification

Diarization can produce: Speaker 0 Speaker 1 Speaker 0

But that does not automatically mean: Speaker 0 = Saad Speaker 1 = Sneha

The application needs additional information to map technical speaker labels to real participant names. This distinction is important when building meeting applications.

For multilingual meetings, preserve the original text whenever possible.

For example: Original: "Pehle worker ko test karna chahiye."

Normalized: "We should test the worker first." Keeping both gives the application: Original Transcript + Normalized Transcript The original remains the source of truth.

Diarization is not automatically speaker identification.

Diarization tells us that two different voices exist. Identification tells us who those voices belong to.

“Speech is the most natural way humans communicate.”- David Leroy

The engineering challenge begins when we ask a machine to understand that speech under real-world conditions.

Imagine we are building a meeting-notes bot. The requirements are:

The workflow becomes: Meeting Starts ↓ Bot Joins ↓ Audio Recording ↓ Audio Validation ↓ VAD ↓ STT / Dograh ↓ Transcript ↓ Speaker + Timestamp Processing ↓ Summary + Action Items ↓ Database ↓ Cleanup A structured result might look like:

{ "meeting_id": "meeting_123", "segments": [ { "speaker": "Saad", "start": 12.4, "end": 18.2, "text": "Let's test the transcription worker today." }, { "speaker": "Sneha", "start": 18.5, "end": 24.0, "text": "I already checked the worker logs." } ], "action_items": [ { "owner": "Saad", "task": "Test the transcription worker" } ] } Enter fullscreen mode Exit fullscreen mode The application now has structured information instead of one large text file.

Different applications require different architectures.

Modular Each stage can be tested independently.

Flexible STT providers and processing components can be changed without redesigning the entire application.

Self-Hostable Dograh can be deployed within infrastructure controlled by the development team.

Better Downstream Processing Structured transcripts are easier for LLMs and databases to consume.

Accuracy Is Not Guaranteed Speech recognition can still make mistakes, particularly with noise, accents, technical terms, and overlapping speech.

Infrastructure Complexity Self-hosting introduces responsibilities such as:

Speaker Mapping Knowing that two speakers exist is different from knowing their names.

Multilingual Complexity Mixed-language conversations may require additional language detection, translation, and normalization.

Numbers are useful only when we understand what they measure.

1. Whisper Was Trained on 680,000 Hours of Audio Statistic: OpenAI reports that Whisper was trained on 680,000 hours of multilingual and multitask supervised audio data collected from the web. Why it matters: Large and diverse training data was one factor behind Whisper’s reported robustness across accents, background noise, technical language, and multiple languages. Year: 2022 Source: OpenAI- Introducing Whisper

2. Whisper’s Training Data Covered 98 Non-English Languages Statistic: The Whisper model card reports that approximately 117,000 hours of the training data represented non-English audio paired with corresponding transcripts, covering 98 languages. Why it matters: Multilingual training is particularly relevant to applications where users switch languages or where English is not the only language being spoken. Year: Model card, current repository documentation Source: Whisper Research Paper

3. pyannote Community-1 Processes AMI Audio at About 31 Seconds per Hour Statistic: The pyannote project reports approximately 31 seconds of processing time per hour of AMI audio for its community-1 diarization pipeline under the published benchmark setup. Why it matters: Diarization does not necessarily require processing time comparable to the duration of the recording. Hardware, pipeline configuration, and workload can dramatically affect throughput. Benchmark setup: NVIDIA H100 80GB HBM3, benchmark updated September 2025. Source: pyannote.audio GitHub

4. WER Is the Standard Basic Metric for Speech Recognition Statistic: Word Error Rate is calculated as: WER = (Substitutions + Deletions + Insertions) / Reference Words Why it matters: A transcript containing fewer word-level errors is generally more accurate, but WER does not treat all mistakes equally. An incorrect person’s name can be much more damaging than an incorrect filler word. Source: AssemblyAI- Speech-to-Text Benchmarks

1. Transcription and Translation Are Different Tasks An ASR system can produce text in the language that was spoken. Translation changes that text into another language. For example: Hindi Speech ↓ Hindi Transcript ↓ English Translation Combining the two concepts too early can make debugging much harder.

2. Speaker Diarization Does Not Necessarily Know Names A diarization system can identify: SPEAKER_00 SPEAKER_01 SPEAKER_02 but that does not automatically mean: SPEAKER_00 = Alice SPEAKER_01 = Bob SPEAKER_02 = Charlie Additional identity information is required for reliable name mapping.

3. pyannote Supports Exclusive Speaker Diarization Recent pyannote releases include an exclusive diarization output intended to make reconciliation between speaker timestamps and transcription timestamps easier. This is particularly useful because ASR and diarization systems do not always produce perfectly matching segment boundaries.

4. Dograh Is More Than an STT Layer Dograh describes itself as an open-source voice AI platform rather than simply a transcription API. Its platform includes workflow construction, STT, LLM, TTS, telephony integrations, and MCP capabilities. That makes it possible to think about transcription as one part of a larger voice-agent architecture.

5. Noise Is Not the Same as Silence A silent section is relatively easy to identify. Noise is more difficult.

A microphone can capture: Keyboard Fan Traffic Echo People talking in another room

That is why speech detection and audio preprocessing matter.

6. Real-World Audio Is Different From Benchmark Audio

A clean test file can produce impressive results. A real meeting can contain: Accent + Noise + Cross-talk + Code-switching + Technical terminology A production evaluation should therefore use audio that resembles the application’s actual environment.

For speech systems, that “open source project” comes with enormous variation: accents, dialects, vocabulary, speaking styles, and context.

What is an AI transcription pipeline? It is a series of processing stages that converts audio into structured text. It can include audio validation, VAD, STT, speaker handling, language processing, and post-processing.

Why use Dograh? Dograh provides an open-source, self-hostable voice-AI platform with configurable components such as STT, LLM, and TTS.

Is Whisper enough for production? Whisper can be a strong STT component, but production quality depends on the complete pipeline, including audio processing, VAD, language handling, speaker processing, and monitoring.

Does transcription automatically identify speakers? No. Transcription determines what was said. Speaker diarization determines which voice spoke. Mapping that voice to an actual person’s name is another step.

Why use VAD? VAD identifies speech regions and can reduce unnecessary processing of silence and non-speech audio.

How should multilingual meetings be handled? Keep the original transcript and, when required, generate a separate normalized or translated version.

Should meeting recordings be stored permanently? Not necessarily. Retention should be based on the application’s privacy, security, and business requirements.

Can the STT provider be changed later? Yes, if provider-specific code is isolated behind a common interface and the application uses its own internal transcript format.

Building an AI transcription system looks simple from a distance: Audio → Text

Once you build one for real meetings, the complexity becomes obvious. People interrupt each other. Languages change. Microphones introduce noise. Silence fills recordings. Models make mistakes. And the transcript itself is often only the beginning. A better approach is to treat transcription as a pipeline. Audio validation prepares the input. VAD finds useful speech. Dograh can orchestrate the voice workflow and STT layer. Normalization creates consistent data. Speaker and timestamp processing add context. An LLM can then turn that structured transcript into summaries, decisions, and action items.

The biggest lesson is simple: Do not build your application around a transcription model. Build a pipeline around the problem you are trying to solve. The model converts sound into words. Good engineering turns those words into useful information.

About the Author: Saad Ansari is an AI Engineer at AddWeb Solution, working on AI-powered applications, LLMs, automation, and backend systems.