Building an Android AI Agent, Part 10: Handling Unexpected Screens Mid-Task

In Part 9 we built a flat task executor that walks a step list with a cursor, pauses for confirmation, and keeps state on the client.

The assumption baked into that executor: when the agent arrives at a step, the expected screen is there.

A permission dialog appears. The app updated overnight and the UI changed. A loading spinner is still running. A previous step succeeded but triggered a side effect — an ad, a rate prompt, a "what's new" overlay.

The agent needs to handle this without asking the user every time, and without silently doing the wrong thing.

Treating all three the same way is the mistake. A spinner should be waited out. A permission dialog should be handled inline. A hard failure needs a new plan.

data class ScreenSnapshot ( val packageName: String?, val activityName: String?, val visibleTexts: List < String >, val interactableIds: List < String > ) fun captureSnapshot ( rootNode: AccessibilityNodeInfo?): ScreenSnapshot { val texts = mutableListOf < String >() val ids = mutableListOf < String >() fun traverse ( node: AccessibilityNodeInfo?) { node?: return node. text?. toString ()?. let { texts. add ( it ) } node. viewIdResourceName?. let { ids. add ( it ) } for ( i in 0 until node. childCount ) traverse ( node. getChild ( i )) } traverse ( rootNode ) return ScreenSnapshot ( packageName = rootNode?. packageName?. toString (), activityName = null, // filled from AccessibilityEvent if available visibleTexts = texts, interactableIds = ids ) } Enter fullscreen mode Exit fullscreen mode Before executing each step, the executor captures a snapshot and compares it against what the step expects.

object ObstructionDetector { private val DISMISS_TEXTS = setOf ( "not now", "maybe later", "skip", "no thanks", "dismiss", "close", "cancel", "got it", "ok" ) private val PERMISSION_TEXTS = setOf ( "allow", "deny", "while using the app", "only this time", "don't allow" ) private val LOADING_IDS = setOf ( "android:id/progress", "progressBar", "loadingSpinner" ) sealed class ObstructionType { object None: ObstructionType () object Loading: ObstructionType () data class Dismissible ( val dismissText: String ): ObstructionType () data class Permission ( val allowText: String ): ObstructionType () object Unknown: ObstructionType () } fun detect ( snapshot: ScreenSnapshot ): ObstructionType { if ( snapshot. interactableIds. any { id -> LOADING_IDS. any { id. contains ( it ) } }) { return ObstructionType. Loading } val lower = snapshot. visibleTexts. map { it. lowercase () } val permission = PERMISSION_TEXTS. firstOrNull { p -> lower. any { it. contains ( p ) } } if ( permission!= null ) { val allowText = snapshot. visibleTexts. firstOrNull { it. lowercase (). contains ( "allow" ) }?: "Allow" return ObstructionType. Permission ( allowText ) } val dismiss = DISMISS_TEXTS. firstOrNull { d -> lower. any { it. contains ( d ) } } if ( dismiss!= null ) { val dismissText = snapshot. visibleTexts. firstOrNull { it. lowercase (). contains ( dismiss ) }?: dismiss return ObstructionType.

Dismissible ( dismissText ) } return ObstructionType. None } } Enter fullscreen mode Exit fullscreen mode Handling obstructions inline suspend fun handleObstruction ( obstruction: ObstructionDetector. ObstructionType, service: StareBrainService ): Boolean { return when ( obstruction ) { is ObstructionDetector.

ObstructionType. Loading -> { delay ( 1500 ) true // retry the step } is ObstructionDetector. ObstructionType.

Dismissible -> { service. clickText ( obstruction. dismissText ) delay ( 300 ) true } is ObstructionDetector. ObstructionType. Permission -> { // Permissions are always user decisions — don't auto-allow false // escalate to confirmation flow } is ObstructionDetector.

ObstructionType. Unknown -> { false // escalate to LLM } is ObstructionDetector. ObstructionType.

None -> true } } Enter fullscreen mode Exit fullscreen mode Permissions are never auto-handled. The agent dismissing a "allow microphone access" dialog on behalf of the user is a trust violation, even if it's technically possible. It escalates to confirmation instead.