Test Automation with agent-device (Part 2)

agent-device drives real devices with the same commands it uses for simulators and emulators. A real iPhone, though, needs a signed test app installed on the device to drive the screen, and a real Android device has to be selected explicitly depending on how it is connected. This post covers why real devices were needed, how to prepare an iPhone and an Android device, and what to watch out for when handling apps on them. It is based on agent-device 0.21.31 (2026-10-24).

Installation and the basic loop are in Part 1.

Why real devices

Simulators and emulators are enough to check app screens, but they cannot reproduce how a build shipped through the store behaves on a real user's device. Three things had to be checked on real devices.

Store-build updates and reinstalls. Installing a TestFlight build over the existing App Store version (the update path), and deleting and reinstalling the app and then restoring a backup, can only be checked on a real device. Whether the backup file was overwritten was checked in the device's Files app under iCloud Drive, by comparing file size and modification time before and after the install.

System dialogs on first launch. The order in which system dialogs such as App Tracking Transparency (ATT) appear relative to the app's own screens has to be seen on a real install. A reinstall also resets that consent history, so first work out whether the state you want to test can actually be produced on the device.

Real ads. The Google Mobile Ads SDK automatically treats an emulator as a test device. logcat prints This request is sent from a test device., and even a release build with production ad unit IDs shows only test ads. That means a no-fill (the ad server finds no ad to serve) cannot be reproduced on an emulator either. On a real device that is not registered as a test device, logcat prints a setTestDeviceIds hint and real ads are requested. In that case, never tap the ad (invalid clicks), watch a production ad to the end at most once per session, and judge the rest from the ad-load results in logcat.

Preparing a real iPhone

On a real iPhone, agent-device installs an XCTest runner on the device. The XCTest runner is a test app that reads the screen and taps on your behalf, and it has to be signed with a development certificate to install. The first run takes a while because the runner gets built, and the cable has to stay connected throughout.

Automatic signing. According to the agent-device docs, if an Apple developer account is signed in to Xcode, the team ID is all you need. To change the runner's bundle ID, also set AGENT_DEVICE_IOS_BUNDLE_ID.

export AGENT_DEVICE_IOS_TEAM_ID=<team id>

Manual signing. If Xcode has no account, or the existing development profile is Xcode-managed, automatic signing will not work. In that case, create a development profile for the runner yourself with an App Store Connect API key. fastlane's sigh creates a profile for a wildcard app ID (*) containing the registered devices.

# key.json: key_id, issuer_id, key (contents of the .p8 file), in_house: false
fastlane sigh --development --app_identifier "*" --team_id <team id> \
  --api_key_path key.json --provisioning_name "<profile name>" \
  --output_path <folder> --skip_install --skip_certificate_verification

Putting the .p8 file path in key.json (key_filepath) failed; putting the file contents in key worked. Copy the generated profile to ~/Library/MobileDevice/Provisioning Profiles/<UUID>.mobileprovision, and delete the key file once you are done.

Then pass the signing details as environment variables. The signing identity is the SHA-1 value shown by security find-identity -v -p codesigning, and the profile was specified by the name given to sigh.

export AGENT_DEVICE_IOS_TEAM_ID=<team id>
export AGENT_DEVICE_IOS_SIGNING_IDENTITY=<certificate SHA-1>
export AGENT_DEVICE_IOS_PROVISIONING_PROFILE="<profile name>"
export AGENT_DEVICE_STATE_DIR=<separate state folder>

AGENT_DEVICE_STATE_DIR is the agent-device daemon's state folder. A separate one was set to keep this work apart from the default daemon used by other work. Bundling these variables and the working directory (cd) into one wrapper script runs every command in the same environment.

Mac and device settings. Besides signing, three things have to be right for the runner to work.

1. Developer tools security on the Mac must be enabled. If it is off, the runner stops at Developer mode is disabled for Apple development tools. This is a Mac-side setting, separate from Developer Mode on the device; run sudo DevToolsSecurity -enable with admin rights.
2. iPhone Settings → Developer → "Enable UI Automation" must be on. If it is off, the runner installs and the test starts, but nothing responds.
3. Keep the screen on and unlocked. When the runner timed out at "Starting XCTest runner", the device was locked. A retry after unlocking never happened, so the cause is only circumstantial, but checking the lock state first with xcrun devicectl device info lockState --device <UDID> is the safer habit.

Handling apps on an iPhone

Listing and deleting installed apps on a real device is done with Xcode's devicectl.

xcrun devicectl list devices
xcrun devicectl device info apps --device <UDID>
xcrun devicectl device uninstall app --device <UDID> <bundle-id>

The runner is removed the same way. Its bundle ID is com.callstack.agentdevice.runner.uitests.xctrunner, and running open after deleting it builds and installs the runner again.

When installing a test build from TestFlight while the App Store version of the same app is already installed, a warning says "You may lose the app's data." The Install button in that warning overwrites the app. It is safer to have the agent stop at this warning and ask a human.

Deleting an app wipes its on-device data. If the app has a backup, take a screenshot showing that backup is on in the app's settings and that the last backup time is just now before deleting; otherwise, stop instead of deleting.

One more screen-handling note: tapping a toggle in a settings screen by ref sometimes hits the middle of the row and does not flip it. In that case, use the element's rect from snapshot --json and tap the right edge of the toggle by coordinates.

Connecting a real Android device

A real Android device connected over USB debugging behaves exactly like an emulator. With wireless debugging, the device may not show up in adb devices right after connecting. It appears only after the _adb-tls-connect service shows up in adb mdns services, so do not conclude it is missing; check again a few seconds later.

A wireless serial is a long mDNS name and easy to mix up with an emulator. Add --serial to every adb and agent-device command. agent-device sessions are created per working directory, so if an emulator was already in use from the same directory, you get an "already bound to android device" error. In that case, open a separate session with --session <name>.

adb mdns services
agent-device open <bundle-id> --platform android --serial <serial> --session <name> --foreground
agent-device close --session <name>
adb disconnect <serial>

Three things commonly go wrong on Android:

• Quote adb shell commands as one string. Unless the whole thing is wrapped, as in adb shell "cmd notification post -t 'title' <tag> 'body'", adb splits the arguments on spaces.
• Buttons hidden by the keyboard are not found. Close the keyboard with adb shell input keyevent KEYCODE_BACK and snapshot again.
• Quote selectors that contain spaces. Use the label="..." form; passing flags as one variable blob gives "Unknown flag".

When something other than the app, such as an interstitial ad, is on top, a snapshot alone makes it hard to tell what state the app is in. Check the system state with adb as well.

adb -s <serial> shell dumpsys activity activities | grep topResumedActivity   # what is in front right now
adb -s <serial> shell pidof <package>                                          # is the app process alive
adb -s <serial> logcat -d | grep -E "FATAL|AndroidRuntime"                     # traces of a crash

Match the snapshot against whether the front screen is the ad or the app, whether the app is still alive behind it, and whether a crash left a record. Ad screens put many webview and link nodes into the accessibility tree, so filter those out when reading a snapshot.

References

• agent-device repository: https://github.com/callstack/agent-device
• agent-device agent setup docs: https://oss.callstack.com/agent-device/docs/agent-setup
• fastlane sigh docs: https://docs.fastlane.tools/actions/sigh/
• Android wireless debugging: https://developer.android.com/tools/adb#wireless