Flutter Voice AI: Real-Time Speech Feedback

Posted on Nov 1 • Originally published at uzairxdev.online

Adding voice to a mobile app sounds like a solved problem until you ship one. A recognizer that works beautifully in a quiet-room demo turns flaky on a crowded bus, partial transcripts flicker in and out of existence, and users end up repeating themselves until they give up. I learned this building Fonetic AI, an app that helps people learn English by listening to their pronunciation, running real-time conversational practice, and giving actionable feedback on their mistakes. The frontend is Flutter and Dart, the speech work goes through AI speech APIs with Firebase on the backend, and the app ships more than 100 interactive lessons where every spoken answer is scored in real time.

This article distills the architecture that made real-time speech feedback actually work in production: a partial-results pipeline built on the speech_to_text package, word-level scoring against an expected phrase, an explicit session state machine, and the permission and performance pitfalls that will waste your week if you skip them.

A pronunciation exercise is a tight loop: show a prompt phrase, capture the user's speech, compare what was recognized against what was expected, and give feedback while the moment is still fresh. The single biggest architectural decision is this: feedback must come from partial results, not from waiting for the recognizer's final verdict.

When you call listen()\ with partialResults: true\, the recognizer streams hypotheses to your onResult\ callback several times per second. Each hypothesis improves on the last. The final result is only delivered after the recognizer decides the user has stopped speaking, which adds one to three seconds of dead air. For a language learner that pause kills the experience; the feedback needs to land while they are still in speaking mode.

The speech_to_text package is the community standard and wraps the platform recognizers on Android and iOS. Add it to your dependencies, then handle platform permissions before anything else.

On Android, declare the microphone permission in AndroidManifest.xml\:

\ xml <uses-permission android:name="android.permission.RECORD_AUDIO" /> \ \

On iOS, add both usage descriptions to Info.plist\. The app will crash at runtime without them:

\ xml <key>NSMicrophoneUsageDescription</key> <string>We need microphone access to hear your pronunciation.</string> <key>NSSpeechRecognitionUsageDescription</key> <string>We need speech recognition to give you pronunciation feedback.</string> \ \

Initialize the recognizer once, ideally at app or feature startup, and prefer on-device recognition when the model is available — it removes network latency from the feedback loop:

\ `dart import 'package:speech_to_text/speech_to_text.dart';

Future initSpeech() async { return speech.initialize( onStatus: (status) => debugPrint('status: $status'), onError: (error) => debugPrint('error: \${error.errorMsg}'), options: [SpeechToText.onDevice], ); } ` \

Start listening with a generous listenFor\ window, a short pauseFor\ so silence ends the session quickly, partial results on, and an explicit locale. Locking the locale to the lesson language (for example en-US\ ) noticeably improves accuracy over auto-detection:

\ dart await speech.listen( onResult: onSpeechResult, listenFor: const Duration(seconds: 30), pauseFor: const Duration(seconds: 3), partialResults: true, localeId: 'en-US', ); \ \

The onResult\ callback fires for both partial and final hypotheses. Check result.finalResult\ to tell them apart:

\ dart void onSpeechResult(SpeechRecognitionResult result) { if (result.finalResult) { scoreUtterance(result.recognizedWords); } else { highlightMatchingWords(result.recognizedWords); } } \ \

One performance rule matters more than any other here: partial results arrive many times per second, so never run setState\ on the whole screen for each one. Drive the live transcript through a ValueNotifier\ and rebuild only the transcript widget:

\ `dart final transcript = ValueNotifier('');

void highlightMatchingWords(String words) { transcript.value = words; }

// In the widget tree: ValueListenableBuilder( valueListenable: transcript, builder: (context, value, _) => Text(value), ) ` \

This keeps the animation jank at zero even on mid-range devices, a lesson I also applied when profiling frame budgets for another on-device AI feature. If you want the full breakdown, see my guide on Flutter on-device AI with TensorFlow Lite.

For scripted lessons you already know the target phrase, which makes scoring a word-alignment problem rather than a machine-learning one. The standard metric is word error rate: the number of word-level insertions, deletions, and substitutions divided by the number of words in the target phrase. It is exactly the Levenshtein distance computed over word lists:

double wordErrorRate(List expected, List actual) { final m = expected.length; final n = actual.length; final dp = List.generate(m + 1, (_) => List.filled(n + 1, 0)); for (var i = 0; i <= m; i++) dp[i] = i; for (var j = 0; j <= n; j++) dp[j] = j; for (var i = 1; i <= m; i++) { for (var j = 1; j <= n; j++) { final cost = expected[i - 1] == actual[j - 1]? 0: 1; dp[i][j] = [ dp[i - 1][j] + 1, dp[i][j - 1] + 1, dp[i - 1][j - 1] + cost, ].reduce(min); } } return m == 0? 0: dp[m][n] / m; } ` \

A word error rate of 0 is a perfect repetition; anything under about 0.2 feels like a clean pass for a learner. The same alignment also tells you which words were wrong, which is what drives the actionable feedback in Fonetic AI: the mismatched words get highlighted in the target phrase so the learner sees exactly where their pronunciation drifted, instead of getting a single vague score.

Normalize both sides before comparing — lowercase, strip punctuation — or a trailing comma will count as a pronunciation error:

\ `dart List normalize(String phrase) => phrase.toLowerCase().replaceAll(RegExp(r'[^a-z ]'), '').split(' ').where((w) => w.isNotEmpty).toList();

// The same alignment identifies the mispronounced words: any position // where the recognized word differs from the expected word is flagged. List misalignedWords(String target, String recognized) { final expected = normalize(target); final actual = normalize(recognized); final aligned = []; for (var i = 0; i < expected.length && i < actual.length; i++) { if (expected[i]!= actual[i]) aligned.add(expected[i]); } if (actual.length < expected.length) { aligned.addAll(expected.sublist(actual.length)); } return aligned; } ` \

A voice session has more states than a boolean can express: idle, requesting permission, listening, scoring, scored, and error. I model AI request flows as explicit state machines everywhere — it is the same pattern I use for text-based AI calls, described in my AI API integration guide. A ChangeNotifier\ holding an enum keeps the UI honest:

\ `dart enum VoiceSessionState { idle, listening, scoring, scored, error }

class VoiceSession extends ChangeNotifier { VoiceSessionState _state = VoiceSessionState.idle; VoiceSessionState get state => _state;

List _wrongWords = const []; List get wrongWords => _wrongWords; bool _passed = false; bool get passed => _passed;

Future start() async { _state = VoiceSessionState.listening; notifyListeners(); await speech.listen( onResult: onSpeechResult, listenFor: const Duration(seconds: 30), pauseFor: const Duration(seconds: 3), partialResults: true, localeId: 'en-US', ); }

void onSpeechResult(SpeechRecognitionResult result) { if (!result.finalResult) { transcript.value = result.recognizedWords; return; } _state = VoiceSessionState.scoring; notifyListeners(); final score = wordErrorRate( normalize(targetPhrase), normalize(result.recognizedWords), ); _wrongWords = misalignedWords(targetPhrase, result.recognizedWords); _passed = score < 0.2; _state = VoiceSessionState.scored; notifyListeners(); } } ` \

Note that scored\ does not mean "passed": the score and the wrong-word list travel together, and the UI decides whether to celebrate or coach. The state machine's job is only to guarantee the UI never shows listening controls and a score at the same time.

Voice features have a long tail of platform edge cases. Handle all five before release:

Scoring is cheap — a word-level edit distance over a sentence takes microseconds — but the plumbing around it is where jank hides. Three rules held up in production:

Fonetic AI's contributions list calls out a gamified learning experience, and it exists for an engineering reason, not just a product one: pronunciation practice is high-friction, and the loop needs a reward to survive repetition. The architecture makes this cheap. The score from wordErrorRate\ maps directly to points, the wrong-word list maps to retry prompts, and lesson completion (the app ships over 100 interactive lessons) maps to streaks. Store the per-lesson best score in Firestore keyed by the user's UID so progress survives reinstalls — the Firebase side follows the standard patterns from my Firestore and authentication guides.

A subtle detail: celebrate attempts, not just passes. A learner who scores 0.35 on their first try and 0.28 on the second is improving, and the UI should say so. The state machine already carries the score; comparing it against the stored best before showing feedback turns a dry evaluation into a reason to try again.

Before you call the voice feature done, verify these eight items on a real mid-range device, not just the emulator:

Real-time voice in Flutter is a solved infrastructure problem and an unsolved experience problem. The recognizer gives you the words; everything your users will remember — the live highlighting, the exact wrong word, the encouragement to retry — is the pipeline you build around it. That pipeline is where Fonetic AI won its production users, and it is the part no package gives you for free.

Get in touch and let's talk about your project.

Originally published at uzairxdev.online