Giving an AI Agent a Real Phone: What ADB Automation Actually Looks Like

The browser was easy. The phone was humbling.

My agent already drives a desktop: screen, shell, browser. So pointing it at an Android device over ADB sounded like a weekend. It took two weeks, and every hard part was the same lesson as the desktop - the interface is the easy bit, state is the hard bit.

adb shell uiautomator dump returns the view hierarchy as XML with bounds for each node. adb shell input tap/swipe/text injects events. adb exec-out screencap -p gives you a PNG. That is enough to drive any app a human can drive by hand.

It is also enough to drive the wrong thing confidently.

1. FLAG_SECURE screens return a zero-byte screenshot. Password fields, banking apps, some onboarding flows. screencap exits 0 and writes nothing. If your loop treats an empty image as "screen unchanged", it will tap happily for ten minutes. Check the byte length and the PNG header, and treat a secure screen as a signal to switch strategy.

2. The UI tree lies about what is tappable. Jetpack Compose and Flutter apps often expose a single node covering the whole screen, so every coordinate resolves to the same element. The fix is not bigger coordinates: use the keyboard ( adb shell input keyevent ) to move focus between fields, and verify by reading the tree back after every action rather than assuming the tap landed.

3. gettext does not exist in the shell. Clipboard is the only sane way to type anything non-trivial: am broadcast -a clipper.set -e text "..." (or a clipboard helper), then input keyevent 279 to paste. Typing Chinese or any non-ASCII via input text is a trap.

Never act on a cached view of the screen. Read, act, read again. It is slower per step and faster per task, because you stop discovering five steps later that the third tap went to the wrong window.

With that in place the phone stops being a manual chore: the same agent that runs a shell script can open an app, log in once, and do the repetitive thing - not something you'd hand a stranger, but exactly the kind of work you resent doing yourself.

If you want the desktop side of this story: https://agent-home.ai

What is your go-to trick for driving an app that refuses to expose a usable accessibility tree?