How We Built AI-Powered Japanese Shadowing Practice

Posted on Oct 24 • Originally published at Medium

Originally published on Medium. Cross-posted here for the dev community.

Here's the spec a language learner would give you:

"Take any video. Let me loop one sentence at a time, repeat it out loud, and tell me which sounds I got wrong."

Sounds like a weekend project: grab captions, add a loop button, call a speech API. In practice, each of those three steps breaks in its own way for Japanese — no spaces between words, sentences that run for fifteen seconds, captions timed for reading rather than audio, and two writing systems for the same word.

We're the team behind Corodomo, a web and mobile app that turns videos into Japanese, English, Chinese and Korean lessons. This post walks through the approach we took for segmentation, timestamp sync, pronunciation scoring and dictation checking — not our full internals, but the decisions and the lessons, which should be useful to anyone working with speech, subtitles or learner feedback.

Learners don't shadow a video; they shadow a line until it feels natural. So the product depends on cutting audio into pieces someone can hold in short-term memory and repeat in one breath.

We generate our own transcript with speech recognition, because many videos have no subtitles and auto-captions are split by time, not meaning. We keep word-level timing, because every later step depends on it.

Text-only splitting ignores where the speaker breathes. Pause-only splitting cuts every hesitation. Using both signals produced noticeably more natural loops in our testing.

Off by 300 ms and the learner loses the first mora, or hears the start of the next line. What helped:

Point 4 was the best return on effort. Users forgive a wrong boundary they can fix in two taps.

"72/100" tells a learner nothing. For Japanese the natural unit is the syllable/mora, so each attempt shows per-syllable scores plus an overall score.

record attempt ──► align to reference text ──► assess each unit ──► aggregate (per syllable) (sounds + pitch) (sentence score) Enter fullscreen mode Exit fullscreen mode Lessons:

すこし vs 少し is not a listening error. しょう vs しよう is. So we normalise both strings (script, punctuation) before diffing, while keeping distinctions like small kana and long vowels that reflect what was actually heard. Answers in kana or kanji are both accepted. Then we highlight the difference side by side.

Corodomo runs on web, iPhone, iPad, Mac and Android. You can start learning without signing up; importing your own videos, pronunciation checks and AI speaking practice need a free account, and the free plan includes one hour of AI usage per month.

We're a two-person core team in Da Nang, Vietnam (plus a few part-time collaborators). If you've built anything with forced alignment, Japanese tokenisation or pronunciation assessment, I'd love to hear how you handled the edge cases — comments are open.