How to give an AI agent a real iPhone with a $7 ESP32
Most phone automation needs a computer on a cable, a developer account, or a jailbreak. ESP32 iPhone Agent takes a different route: a small ESP32-S3 board plugs into the iPhone and acts as the agent's eyes and hands, while an iPhone app runs the vision model loop. At run time it is just the phone a
Posted on Oct 24 • Originally published at dedupedai.com
Most phone automation needs a computer on a cable, a developer account, or a jailbreak. ESP32 iPhone Agent takes a different route: a small ESP32-S3 board plugs into the iPhone and acts as the agent's eyes and hands, while an iPhone app runs the vision model loop. At run time it is just the phone and the board.
This guide walks through the setup from the project's README, start to finish, so you can type a goal like "Open Settings and go to Wi-Fi" and watch a vision model do it on a real iPhone.
The board opens its own Wi-Fi hotspot and shows up in the iPhone's Screen Mirroring list as ESP32 Agent. The phone mirrors its screen to the board over AirPlay, and the board forwards that stream to the app over Bluetooth LE. The app sends screenshots to any OpenAI-compatible vision model, which plans the next action. The board is also a standard USB mouse and keyboard, so with AssistiveTouch on, a click is a tap and a drag is a swipe. Loop until done.
Plug the board into your Mac. Install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:
arduino-cli lib install "NimBLE-Arduino@2.5.1" hardware/flash.sh # optional: [serial port] [--debug] Enter fullscreen mode Exit fullscreen mode Always build with flash.sh. It applies the BLE buffer sizes the firmware needs.
cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # set DEVELOPMENT_TEAM and IA_BUNDLE_ID open ios/IPhoneAgent.xcodeproj # run the IPhoneAgent scheme on your iPhone Enter fullscreen mode Exit fullscreen mode A free Apple ID can sign it, but you have to re-sign every 7 days. The one-tap hotspot join needs the Hotspot Configuration capability (a paid account). Without it, the app shows the hotspot password so you can join by hand.
Plug the board into the iPhone and open ESP32 Agent. Setup checks each step itself:
Type a goal such as "Open Settings and go to Wi-Fi" and press Start. Stop it any time from the Dynamic Island. Once you have tasks you repeat, Siri, Shortcuts, widgets and Control Center can run saved tasks.
Every run is kept on the phone. Open Runs, pick a run and replay it step by step with the taps and swipes drawn on top. To debug with another agent, use Runs, open a run, ⋯, Export trace to get the screenshots, actions and annotations.
The planner and actor prompts are plain text files in prompts/ and are bundled with the app. Edit them, rebuild the IPhoneAgent scheme, and compare the new run against the old one in Runs. If you suspect the board rather than the app, hardware/tools/board_test.py plays the app's role over BLE from a Mac (see hardware/tools/README.md and docs/ble-protocol.md ).
Only automate devices and accounts you own or are allowed to use. The agent acts with real hands on the phone, so watch it and keep the stop button within reach.
We wrote up five concrete use cases for this repo, with steps and copy-paste agent prompts:
Project credit: ESP32 iPhone Agent by Henry Yuan ( https://github.com/onehumanbeing/esp32-iphone-agent ), MIT licensed, with the AirPlay part under GPL-3.0.
Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com