AirPlay mirroring
The phone mirrors its screen to UxPlay running on your computer. A GStreamer pipeline turns the stream into JPEG frames for the agent and for the web page.
Open source · MIT · No jailbreakLet an AI model use a real iPhone.
Let an AI model use a real iPhone.
With a $7 ESP32.
The agent sees the phone through AirPlay screen mirroring and touches it through a tiny ESP32-S3 that plugs into the USB port as a mouse and keyboard. Any OpenAI-compatible vision model can be the brain.
Sign in with Google to start a run. · Fork it, flash it, self-host it.
~$7ESP32-S3 boardMac or LinuxSetup computerNot neededApple dev accountNoneApp on the phone
plan Open Whiteboard and draw a smiley face
see screenshot 1179x2556 → actor model
tap (589, 1421) precise d=0.4px 1.3s
draw smile path, 24 points 2.6s
check screen changed → continue 0.8s
act home → Settings → Wi-Fi 1.1s
step handling next screen▍
Why this exists
No Mac. No Xcode. No signed helper app.
Most iPhone automation drives the phone through XCTest / WebDriverAgent. That needs a Mac, Xcode, a signed helper app and an Apple developer account - a free account's signature expires every 7 days. This project uses the same paths a person uses: the screen and a pointer. The phone needs nothing installed.
Feature ESP32 iPhone AgentXCTest / WebDriverAgent tools
Apple developer account Not neededNeeded (to sign the runner)
Xcode Not neededNeeded
Computer Mac or LinuxMac
App installed on the phone NoneTest runner app
Jailbreak Not neededNot needed
Extra hardware ESP32-S3 board (~$7)USB cable
Works in any app, incl. games Yes - it only sees pixelsMostly - depends on the accessibility tree
How it works
Eyes, hands and a brain
Four small pieces, all of them understandable in an afternoon.
A USB HID bridge
An ESP32-S3 plugged into the phone shows up as a standard USB mouse and keyboard. With iOS AssistiveTouch on, the mouse moves an on-screen pointer - a click is a tap and a drag is a swipe. The board connects to the computer over Wi-Fi and executes simple line commands; all logic stays on the computer.
A planner and an actor
A planner model writes a short plan from the goal and the current screen. An actor model then looks at each new screenshot and picks the next action: tap, swipe, draw a path, type, press a key, go home, or finish. Both roles can be the same model - a cloud API, or a model you run yourself.
Closing the loop
The pointer is relative, so the server tracks where it is. Before each tap it wiggles the pointer, finds it on the mirrored screen and corrects any miss, refining the calibration as it goes. After each action it checks whether the screen actually changed - a missed tap is noticed and retried instead of built upon. Repeated actions trigger a re-plan; a dropped link or a bad reply is recovered from.
The web UI
Everything the agent does, on one page
The hosted Studio wraps the open-source web UI with queueing, model routing and history - the same panels, no local server required.
- Live phoneClick to tap, drag to swipe, scroll to scroll - with manual Home, Back, typing and keys.
- Live logEvery plan, thought and action as it happens, with timings.
- Step wallThe screenshot the model decided on for every step, with its tap or stroke drawn on top, plus the reasoning and how long thinking and acting took. Click a card to zoom in.
- Tap markersTaps are marked with a red ring and a spreading ripple; swipes and strokes with a yellow path. The Motion switch turns the animation off.
- Emergency stopPress Esc to stop the agent and release the pointer. Clear history empties the log.
- English firstA clean English interface. The open-source UI also ships a Simplified Chinese translation.
agent run · set a repeating alarm esc
plan Open Clock, set an alarm for 7:00 AM that repeats every day, then save it
step 1 home → Clock tap (87, 512)
step 2 tap Alarm tab tap (295, 142)
step 3 type 0700 into the time wheel key x4
step 4 tick the Repeat days tap x4
step 5 Save tap (354, 812)
result alarm saved · screen changed · finished 11 steps · 46s
Hardware
A $7 board and the phone in your pocket
The phone powers the board. No capture cards, no docks, no soldering.
ESP32-S3 board · ~$7
Tested with the Seeed Studio XIAO ESP32S3 (plain version without pre-soldered headers - not the Sense camera version, not the Plus). About $7 from Seeed, or ¥40-50 on Taobao ("XIAO ESP32S3"). Any ESP32-S3 board whose USB-C port is wired to the S3 native USB works.
iPhone
USB-C models plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board over the cable.
Computer
macOS or Linux, on the same network as the phone; it runs UxPlay and the server. Use a USB-C hub with power pass-through if the phone should keep charging while the board is plugged in.
Vision model
Any OpenAI-compatible chat API that accepts images. Running a model locally is optional - the project was developed against a local Qwen3.8-Flash setup at about 1.3 s per action and 2 s per plan.
Quick start
From unboxing to first task in one evening
Four steps: flash the board, run the server, set up the phone once, then give it a task.
Flash the board
Install arduino-cli and the ESP32 core (3.x), copy the secrets template and edit it with your 2.4 GHz Wi-Fi name/password and the IP of the computer running the server.
cp hardware/hid_bridge/secrets.example.h hardware/hid_bridge/secrets.h
# edit secrets.h: 2.4 GHz Wi-Fi + the server's IP address
hardware/flash.sh # optional: [serial port] [--debug]Install and start the server
UxPlay mirrors the phone; the Python server plans, acts and serves the web UI on port 8890.
# UxPlay — macOS:
brew install uxplay
# UxPlay — Debian / Ubuntu:
sudo apt install uxplay gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-libav
cd server
python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env # set LLM_BASE_URL / LLM_MODEL / LLM_API_KEY
set -a && . ./.env && set +a && .venv/bin/python -m hidagentOpen http://<computer-ip>:8890 in a browser.
Set up the phone (once)
Turn on AssistiveTouch, plug the board into the phone, mirror the screen, then calibrate the pointer from the web page.
1. Settings → Accessibility → Touch → AssistiveTouch: on
2. Plug the board into the phone; a round pointer appears
3. Control Center → Screen Mirroring → iPhone Agent
4. Web page → Manual controls → CalibrateCalibration moves the pointer, finds it on the live screen and corrects itself in a few seconds.
Give it a task
Type a goal - for example "Open Settings and go to Wi-Fi" - and press Start, or click one of the examples. Every step lands on the step wall with its screenshot and timings.
"Set an alarm for 7:00 AM that repeats every day, then save it"Press Esc at any time for the emergency stop.
Or let an AI coding agent set it up
Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine: flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key.See it work
Drawings, alarms and everything you can touch
Two recorded runs from the project and the desk setup itself. Every step keeps its screenshot, its tap or stroke and its timing.
Example 1 - draw on a canvas
"Open Whiteboard and draw a smiley face": the agent searches for the app, opens it and draws the face with continuous strokes. Every step's screenshot, with its tap or stroke drawn on top, stays on the step wall.
Example 2 - set an alarm
"Set an alarm for 7:00 AM that repeats every day, then save it": the agent opens Clock, types 0700 into the time wheel, ticks the days under Repeat several taps at a time, and saves. Left: the real iPhone with the ESP32 plugged in; right: the web UI.
The whole desk
A XIAO ESP32S3 plugged into an iPhone, next to the web UI. The phone powers the board - no capture hardware, no docking station, no Mac in the loop.
Open source
MIT licensed, end to end
Firmware, server and web UI are all in one repository. Read it, fork it, or self-host the whole thing.
MIT licensed
All of it: the ESP32-S3 firmware, the Python server and the web UI. Fork it, ship it, or build your own hosted version.
Two folders
hardware/ holds the ESP32-S3 firmware (Arduino) and the flash script. server/ holds the agent, web UI, screen capture and board link - Python and FastAPI.
Contributing
Issues and pull requests are welcome. Run the checks before sending a change: cd server && pip install -e ".[dev]" && ruff check . && pytest.
Star on GitHub
Built on UxPlay (AirPlay receiver) · Arduino ESP32 core and TinyUSB (USB HID) · FastAPI.
Limitations, honestly
- iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them.
- Mirroring stops when the phone locks - keep auto-lock off while the agent works.
- Keyboard input is ASCII only.
- Re-calibrate after changing AssistiveTouch tracking speed.
- Landscape apps are not supported yet: drawings and taps there land in the wrong place.
- On Linux machines with several network interfaces, limit Avahi (mDNS) to the interface on the phone's network if the phone does not list the receiver.
Questions, answered
Do I need a Mac, Xcode or an Apple developer account?
No. The phone needs nothing installed - no helper app, no signing, no developer account. Your computer just runs UxPlay and the server, and it can be Linux.
Does this require a jailbreak?
No. The agent uses exactly the paths a person uses: the mirrored screen for eyes and a USB mouse/keyboard for hands, with iOS AssistiveTouch moving an on-screen pointer.
Which iPhones are supported?
USB-C models plug straight into the board. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board, and a USB-C hub with power pass-through keeps the phone charging.
Which model can be the brain?
Any OpenAI-compatible chat API that accepts images - a cloud model, or one you run yourself with vLLM, SGLang, llama.cpp or Ollama. Planner and actor roles can use the same model or different ones.
What does the hosted Studio add?
The Studio is the same web UI with queueing, cloud vision-model routing and session history, so you can run the agent without installing anything locally. The open-source server stays MIT-licensed and fully self-hostable.
Does it work in games and apps without accessibility trees?
Yes - it only sees pixels and acts with a pointer, so anything you can touch, it can touch. The known exceptions are screens iOS refuses to mirror, like passcode entry, Apple Pay and DRM video.
How does billing work?
Plans are billed by Stripe, monthly or yearly. You can cancel at any time from the billing portal and keep access until the end of the period.
What happens during a run?
A planner writes a short plan from your goal and the current screen; the actor looks at each new screenshot and picks the next action. After every action the agent verifies that the screen actually changed, so a missed tap is retried instead of built upon.
Run your first task tonight
Flash a $7 board, plug it in, and let a vision model take it from there. Or start with the hosted Studio and skip the setup.