Dev.to AI 🤖 Ai Mon, 26 Oct 2026 23:39:32 +0000 👁 0 📖 14 min read

Red, then green: 11 bugs TestSprite found in my NASA video research agent

TL;DR Ephemeris is a research agent over 87 NASA clips. I put 85 TestSprite tests behind it (34 browser, 51 API), and they found 11 real bugs. The centre of the story is PR #13: the first commit is only a regression te

TL;DR

Ephemeris is a research agent over 87 NASA clips. I put 85 TestSprite tests behind it (34 browser, 51 API), and they found 11 real bugs.
The centre of the story is PR #13: the first commit is only a regression test and goes red, the second carries the fixes and goes green.
Every pull request gets a preview of its own and a TestSprite check before it can merge. Below: how I set that up, what the tests caught, and the steps to copy it.

This is my write-up for TestSprite Season 4.

The problem: an archive in the wrong order

NASA published almost nothing online before the year 2000. So a clip uploaded in 2015 is routinely discussing results from 1976. Sort the video library by its own metadata and you get the order things were posted, which is not the order anything happened.

That matters if you want to understand how a field changed its mind. The real value of an archive like this is somebody on camera in 1965 saying what they thought Mars was, and being wrong, and somebody in 2015 saying what we know now. Titles and tags do not capture that. The truth is in the spoken words and in what was printed on the screen behind them.

Today you can find a clip called "Mars report". You cannot ask how our understanding of water on Mars changed and get back the exact moments that answered it. That is what I built Ephemeris to do.

What Ephemeris does

You ask a question in plain English and get three things:

A cited answer, ordered by when events actually happened, not by when NASA uploaded the file. Every claim carries a citation number that maps to a moment in a clip.
An evidence reel, one video cut from the exact seconds that support the answer, with the citation number, the year, how that year was determined, the mission and the NASA identifier burned into the frame so it survives an export.
A 3D solar system whose camera follows the reel. Each moment stands on the body it concerns, and the camera moves through the decades as the reel plays.

The corpus is 87 real NASA public-domain clips: 240 minutes, 1,484 scenes, 1957 to 2025. Nothing is synthetic and nothing is mocked. The live app is at https://ephemeris-taupe.vercel.app, and the code is at https://github.com/crypticsaiyan/Ephemeris. The four preset questions on the landing page are real pipeline output and need no key.

How it works, briefly

The tests only make sense if you know what they are guarding, so here is the short version.

Teaching the archive to read

The footage is ingested into VideoDB, which gives each clip a chained analyzer graph. Spoken words and on-screen text (OCR) both feed a vision-language pass that describes each scene while reading what was said and printed during it. Fusion happens at ingestion, not as a merge of three searches at query time. The OCR is load-bearing: NASA footage carries burned-in lower-thirds and mission clocks, and where narration is sparse it is the only textual signal there is.

The result is five indexes: transcript (what was said), scene_semantic (what is visible), scene_facets (what kind of scene it is), ocr (what is printed on screen) and mission_meta (who, where and when). One vision-language pass backs two of them, so the second surface costs no extra inference.

Four ways to know a date

Publication date is useless for this archive, so every scene carries a date and a record of how it was known:

Source	Meaning	Trust
`scene`	the scene states a year, in speech or on screen	highest
`mission`	the stated year was impossible for the mission, so its operating window decided	corrected
`video`	the clip as a whole is anchored to an era	medium
`published`	NASA's publication date	always correct, rarely interesting

The correction matters. A compilation dated once at the top hands that year to every scene beneath it, and one Curiosity segment came back as 1990, twenty-one years before launch. Across the archive, 20.7% of dated scenes with a known mission fell outside its operating window, and 15.7% of the archive is re-dated. A date the scene stated itself is never overruled, because archive footage is full of retrospect.

Answering a question

A search box takes a query and returns clips. Ephemeris takes a question, decomposes it into sub-questions and alternate phrasings, and runs two gates before any retrieval: is the question answerable at all, and does the corpus hold the world it names. Then it retrieves from each index separately, joins runs of touching cells into passages, caps the evidence at two moments per clip, snaps each window to sentence boundaries using word-level timings, orders the result by the era each moment discusses, and asks the model for a cited answer with a chronology and caveats. Last, it compiles the reel.

Several of those choices were forced by measurement:

Query expansion is load-bearing. "Twin rovers landing in 2004" retrieves nothing, because the archive says "the first of two rovers". That phrasing hits rank 1 at 0.783.
Indexes are queried one at a time. Scores are not comparable across indexes, and a combined call dropped a 0.6899 transcript hit out of 20 results.
Diversity is enforced. Without a per-clip cap the chronology comes from one dense clip.
Refusing is a feature. Given gibberish, an early version split it into plausible sub-questions and answered with confidence. Asked about Venus, it once reasoned from Artemis reentry footage while conceding no clip showed Venus. There are zero Venus scenes, and the coverage gate now ends that run in 19 seconds with what the archive does cover.

Retrieval is checked against ground truth built from NASA's own published caption files, so the evaluation does not grade the pipeline on its own output: 35 of 36 cases, mean recall 0.909.

The scene that follows the reel

In the browser, one activeEvidenceIndex is the entire sync bus. Inline [n] markers, timeline needles, shot rows, the craft standing in the 3D scene and the reel's playhead all read from it, so clicking any one of them moves all of them. That single shared index is also why so much of the interface is testable: if one thing selects a moment, you can assert that the other four followed.

Making every pull request testable

The first change was architecture, not tests.

The app ran as one container, and the Next.js route spawned the Python agent as a child process. That rules out a preview deployment per pull request, and a pull request check should test the pull request's own code, not main. In PR #7 I split the Python agent into its own small HTTP service on Render's free tier, behind a shared token, and moved the web app to Vercel. The web route now validates, rate-limits, and forwards the question to the service, streaming the events back unchanged. Now every pull request gets a Vercel preview with its own URL, and that URL is what TestSprite tests.

Setting up TestSprite

This is what I actually did, in order.

1. Generate a suite from the live app. TestSprite explores the URL with several agents and proposes a test plan. Mine proposed 62 tests across 8 use cases, with a lot of repetition: "switch between presets" appeared five times and "chronology is in order" three. I trimmed it to 20. A pull request run costs roughly one credit per browser test, so a deliberate suite is also a cheaper one.

2. Write assertions about what a user can see. I uploaded a short product requirements document first, with the real labels, the preset names, and what counts as a pass. Then I edited the generated steps so each test passes for a reason. The reel test, for example, passes if the player plays, or says the stream is slow, or says the browser cannot play it, with a link; it fails only on a bare "loading" spinner forever.

3. Run it once before wiring it up. A trigger re-runs an existing suite. If the suite has never been generated and run, the trigger has nothing to start. Generate it, run it, and only then add the trigger.

4. Connect the GitHub App and add a trigger. The pull request trigger listens for the Vercel commit status, so it fires when the preview is live, not when the build starts. Its target URL pattern is the preview hostname with a branch placeholder, for example https://my-app-git-{branch-slug}-my-team.vercel.app. Then switch on the merge gate, so a red check blocks the merge.

5. Drive it from the command line. The TestSprite CLI lets me read a failure the same way I would read a stack trace. testsprite test failure summary <id> prints the failing step and a root-cause hypothesis. testsprite test plan put <id> replaces a test's steps from a file. That made the loop quick: read the failure, decide whether the app or the test is wrong, fix one of them, push.

6. Add an API project. A second TestSprite project plans its tests from docs/openapi.yaml and from probes of the live API. It started with 7 tests and grew to 51, and runs against production through a push trigger.

The pull request that proves it: PR #13

PR #13 is built so the history proves the tests can fail. I had written 26 new tests and ran them against production before touching any code. Two failed for real reasons. So the pull request opens with a commit that contains only a regression test, and the fixes come in the second commit.

Commit	Python CI	TestSprite browser suite
1: regression test only	red	16 of 18 pass
2: the fixes	green	18 of 18 pass

I left the red run in the history on purpose. A test you have never seen fail is a test you cannot fully trust, and a red run followed by a green one is the cheapest evidence that it is real. TestSprite rewrites its comment on the pull request as each run finishes, but the red run stays on the first commit's check, which links to its report.

Three bugs were behind the red:

The ask service corrupted the next request after a refusal. The service checks the auth token before it reads the request body, on purpose, so an unauthorised caller costs nothing. That meant a 401 went out with the JSON body still unread. On a connection that stays open, those bytes are parsed as the next request line, and a proxy that reuses upstream connections hands that failure to a different caller. TestSprite's API test sends three refusals in a row and hung on the second. The service now closes the connection after any refusal, and a Python regression test fails without that line. That test is why commit 1 turns Python CI red as well as TestSprite.

The Venus and Mercury panels said "reading corpus summary" forever. The 3D scene lets you focus either planet, but the corpus has no footage of them, so the summary file has no entry for those bodies. The panel could not tell "no footage" from "still loading". Now a loaded summary with no entry for that body says there is no footage of it.

The first question after the free Render host slept failed with a 502. The service sleeps when idle and takes about 40 seconds to wake, and the route gave up on the first 502. I had only ever tested it warm. The route now retries 502, 503, 504 and refused connections for up to 75 seconds, and the trace tells the visitor the host is starting up.

Climbing from 13 to 18: PR #8

PR #8 is the other one worth reading, because the suite went 13 of 18, 16 of 18, then 18 of 18 across three commits, and every step was a real fix.

The planet names could not be clicked. The names floating over the 3D scene were plain text sitting over an invisible picker sphere, so clicking a name did nothing. I turned them into buttons. The test then failed again, differently: the buttons were visible but no click reached them. I reproduced it in headless Chrome. The button was there, the click timed out, and elementFromPoint at its centre returned the wrapper, not the button. The label rule set pointer-events: none so floating names never block a drag on the scene, and the buttons inherited it. A more specific rule fixed it, and a direct check showed the body panel opening with its counts.

Typing stalled. Two tests were blocked because the page was too slow to type into. I measured it. On a browser drawing the 3D scene on the CPU, the orrery at device pixel ratio 1.75 cost 53 ms a frame, and every keystroke in the Ask box waited behind a frame, about 110 ms a key. The scene now checks once whether the renderer is software and, if so, draws at half resolution without antialiasing: 17 ms a frame, about 25 ms a key. I had never noticed it on a machine with a GPU.

A browser without H.264 sat on "loading stream" forever. The player now says it cannot play the reel, or that the stream is slow, and gives a link. A spinner that never ends is the worst of the three possible outcomes, and the test treats it as a failure.

The Asked list vanished when empty, so a first visit showed no sign of where runs go. An empty row now says so.

An over-limit question was refused only after submit. The page now counts characters live, disables submit and says "question too long".

The others

A value that vanished (PR #11). A long value set from code disappeared from the Ask box. It was a controlled React input, and React discards a value that arrives without an input event, which is how some assistive tools, password managers and test drivers fill a field. The box is now uncontrolled, and the length is read from the field on submit, so an over-limit question is refused however it got there.
A 500 where a 400 belonged (PR #16). /api/ask answered 500 with an empty body when the question was a number, an array, an object or a boolean, because the route called .trim() on whatever the JSON held. A visitor cannot cause it from the page, but any API client can. I wrote one API test per JSON type and ran them against production first, so the red run is in the portal's history. The route now answers 400 "question must be text", before the quota is touched.

A suite that grew from 24 to 85

I started with 24 tests. By the end there were 85, and the growth came from asking what each area promises:

Area	Tests	What is asserted
Browser, presets and answers	34 in total	each preset loads its own answer, chronology and reel; citations select their evidence row; the timeline ruler has a needle per moment
Browser, scene		focusing a body opens its panel; the camera strip flips from following the reel to manual orbit; hide notes clears the side panels and show notes restores them
Browser, live ask		one real question through the whole agent, saved to the browser, surviving a reload
API, preset files	51 in total	every preset is a complete result: eight evidence rows in ascending era order, one reel shot per row, citations that point at rows, a numbered trace, a reachable stream
API, refusals		invalid JSON, empty, whitespace-only, over-length and non-text questions each get the exact documented 400, as JSON, with no path or key in the body
API, ask service		health check; no token or a wrong token is a 401; no other routes
API, the site		the page contains no key names or service address

Alongside those, eight offline Python suites run in CI on every push, plus the browser-side checks. The one test that spends VideoDB credits is the live ask, and the visitor quota of five questions an hour bounds it.

What I took from it

Give every pull request a preview of its own. Splitting the agent into a service is the change that made a pull request check meaningful, because the tests ran against the pull request's code and not against main.
Write the regression test as its own first commit. Seeing the red run before the fix, as in PR #13, is how I know the test is real.
Assert what a user can see, and let the test fail on a missing explanation. "Plays, or says why not" is a stronger contract than "plays", because it forbids the silent spinner.
Measure before you fix. The 53 ms a frame came from a measurement, not a hunch, and the fix could be checked against it.
Run new tests against production before you fix anything. The red run is the evidence that the test would have caught the bug.
Keep branch names short. The Vercel preview hostname is built from the branch name and is limited to 63 characters. A longer name will not match the URL pattern the trigger expects, so the check never starts.

Copy this setup

Make sure every pull request gets its own preview URL, and that the app on it can run without a local backend.
Write a short requirements document with the real labels and what counts as a pass, and generate a suite from the live URL.
Trim duplicates, edit the steps, and run the suite once by hand.
Add a pull request trigger on the Vercel commit status, with the preview hostname as the target pattern, and turn the merge gate on.
Open a pull request that contains only a failing test, watch it go red, then add the fix and watch it go green.
Plan an API project from your OpenAPI file, and run new tests against production before you fix what they find.