Parallax beta.10: when the agent's tap misses, it sees what was on the screen

An iPhone 13 in the Parallax window: two clicks on count+ leave dots, a drag leaves a line and scrolls the list, then the agent swipes the
list and taps count+.

Parallax is a desktop app and an MCP server that give you and your coding agent the same live view of a running React Native app: the
screen, the component tree, logs, network and state. beta.9 let the agent tap, swipe and type on a real iPhone. beta.10 is about what
happens when one of those actions misses, plus faster swipes on the iPhone and a mark for your own clicks in the window.

What a miss now shows

A miss is one of these:

• a tap, a swipe or typing after which nothing on the screen changed;
• an id that isn't there (element_not_found), including an element that exists but is off screen.

Until now, the agent got the miss and had to look again to find out why. Now the answer carries a short snapshot of the screen: the app in
front and up to 15 elements on the screen. It is taken from the read the action had already made, so it costs no extra time. The window's
Activity shows you the same snapshot.

We word it as "No element appeared, disappeared or moved." rather than "Nothing changed". A switch that flipped, or a text that changed
inside the same element, looks the same to us, and the words shouldn't claim more than we checked.

The idea came from @progerallworld, who replied to the beta.9 post with "Can a failed tap show what the
agent expected to be visible and what was actually on screen?" Thanks for the nudge. We took the second half: the answer shows what was
there. We chose not to add a parameter for what the agent expected. After each action the agent already gets what changed
on the screen and can compare it with what it expected, agents rarely fill an optional parameter, and every parameter costs tokens in each
session's tool list, which we keep under about 2,500. So the screen is attached on its own, and only when an action misses.

This works everywhere: simulators, Android phones and real iPhones.

Faster swipes on a real iPhone

The agent's swipe on a real iPhone went from a median of 3.5 s to 2.1 s.

We measured where the time went. After a quick swipe the list flings, and WebDriverAgent waited for that scroll to stop: about 3.0 s,
whether the finger moved for 0.35, 0.6 or 0.7 s. Holding still before lifting didn't help. A move that lasts 1 s doesn't fling, so the
swipe takes 1.78 s every time.

There is a side effect. The list now moves by about the swipe's length (18 rows in our test instead of 38). For an agent, which needs to
know where the list will stop, that is more predictable.

Drags in the window got faster too. WebDriverAgent spends about 0.25 s on each point of a path before the finger moves, so a 20-point drag
took 7 s. The window now sends the start, the middle and the end: about 2 s.

A mark for your clicks

When you click on the device's picture, a dot appears where you clicked the moment you release. A drag leaves a line along its path. The
mark fades after about 0.7 s, or when the core answers, and never stays more than 2 s. It shows on every device, stays static with
reduced motion, and the mouse wheel draws nothing.

Fixes

• A window opened before the iPhone's helper was up ignored every click until you restarted it. Fixed.
• Taps on an iPhone that landed were flagged "may not have reached the device", because we watched for a change for only 500 ms. A real iPhone now gets 3 s.
• If the Parallax window is closed and the agent names an iPhone, it now says to open the Parallax window, not "no such simulator".

For debugging, PARALLAX_TIMING=1 prints one line per action with its phases.

What didn't change

A tap on a real iPhone still takes about 1.0 s, and about 0.6 s of each tap is WebDriverAgent and XCTest themselves. Typing takes 0.6 to 0.7 s. Simulators
and Android phones are unchanged, apart from the miss snapshot and the click mark.

The limits

• The miss check looks at elements, not their contents: a flipped switch or changed text inside the same element counts as no change.
• The snapshot holds at most 15 elements.
• The swipe numbers come from one iPhone 13 and one test app (fx-b086).
• macOS 26 or later, on Apple silicon only.

The desktop app is in closed beta: ask for an invite at https://parallax-rn.dev/download. Corrections to anything above are welcome.