My Android AI app has 400 unit tests. None of them can tell me if the model's answer is any good

I'm a student building an open-source Android AI app on my own, and at some point I hit 400 unit tests with zero failures. That sounds great until you ask what those tests actually prove.

Quick context: Kryzz AI is an MIT-licensed Android app (Kotlin + Jetpack Compose) for chat, voice and agent tasks. It doesn't run models on the phone. You bring your own OpenRouter key and the models are hosted by whichever provider you pick. Chats, memory, files and keys stay on the device, and there's no analytics or telemetry.

An AI app has a lot of plain, boring code around the model, and that's exactly where tests pay off. Looking at the test folder, most files fall into a few groups:

These catch real bugs. A change to the plan parser that breaks on one weird model output shows up in seconds instead of in someone's chat.

So "400 tests, 0 failures" means the app is less likely to fall over. It doesn't mean it's good. For that I still need people actually using it.

If you build apps on top of LLMs: how do you check answer quality without paying for a model call on every test run? Recorded responses, a small eval set, or just trying it yourself?

Honest caveats: you need your own OpenRouter key, it's not on the Play Store, and the release build is emulator-tested only so far. Bug reports, especially from real devices, are very welcome.