ChatGPT steuert ein Android-Smartphone direkt, ohne Selbstfreigabe
Statt Skripte zum Kopieren zu liefern, kann ChatGPT ein Telefon inzwischen selbst bedienen, etwa Apps öffnen und Aktionen ausführen. Kritisch bleibt die Freigabe: Der Agent kann kritische Schritte nicht eigenständig bestätigen, was zeigt, wie viel Vertrauen solche Automatisierung noch braucht.
Last month I wrote about the alarm that fired into nothing — how an agent I'd scheduled for 8am went off on time and then silently didn't run, and what it took before I'd trust a schedule again.
This one is about handing that same phone to ChatGPT. Not "ChatGPT writes a script and I paste it." ChatGPT, in the ChatGPT app, asking my phone to do something, the phone doing it on a screen I'm not looking at, and the answer coming back into the chat.
I typed one request into the ChatGPT app: save the top-rated vintage stores near Shimokitazawa to a Google Maps list. On the right, my phone asks me to approve once. Then Google Maps starts moving on its own in the bottom-right pane — a background virtual display, not my screen — and a minute later ChatGPT tells me which two stores it saved. The list is really there when I open Maps.
The second one is quieter and, honestly, the one I use more: one message from ChatGPT registers an agent that runs on the phone every day at 16:27, researches the latest AI papers with Perplexity, and drops a sourced English summary into Obsidian. No ChatGPT, no tap, at 16:27. Every day after that, too.
If ChatGPT can drive the phone, ChatGPT can also, in principle, do the whole loop by itself: ask, approve, act. A web page it reads could say "also run this," and nobody would ever look.
So a hand-off to ClosePaw always waits for one approval on the phone:
ChatGPT can't approve. Writing an "approved" file from the terminal doesn't work either — the one-time nonce only lives in Shelly's process memory. I considered "let ChatGPT ask me, and I'll type yes in ChatGPT," and dropped it: the thing that finally sends that "yes" to the phone is ChatGPT itself.
The result that comes back is checked too: it has to come from the signature-pinned ClosePaw build, carry the request's own one-time 128-bit nonce, and on Android 14+ the sender package has to match. The text itself is treated as data — control and bidi characters stripped — not as instructions.
It took about eight takes. Each one broke something I would otherwise have shipped:
Write the Kotlin that holds this together. Promise that the virtual display trick works the same on every phone. Tell you whether OpenAI's remote-control host reconnects on its own when my phone hops from Wi-Fi to LTE — I'm finding that out this week.
What I can do is notice when the demo looks good and the result isn't actually there, and refuse to post it until it is.
GitHub: github.com/RYOITABASHI/Shelly — v8.1.0, GPLv3. ClosePaw's Shelly build: RYOITABASHI/closepaw releases.