Penumbra: an AI agent for satellites

This is a submission for the MLH x DEV Writing Challenge

What I Built

Penumbra is mission control for a fleet of 24 satellites. A satellite in low Earth orbit spends about a third of each orbit in Earth's shadow, running on its battery. If the power system fails there, operators may not see it until the next ground pass, and then they have a few minutes to send the right command.

In Penumbra, a Gemini agent diagnoses the fault and proposes recovery plans. A physics simulator tests every plan against every cause that still fits the data, and a human approves the exact plan that passed. Nobody can approve a plan that failed the check, the AI included.

We wanted an agent that is allowed to be wrong, in a system where its mistakes cannot reach the satellites. Most of our weekend went into the code around the model: the list of allowed commands, the verifier and the approval step. The orbits, the Sun's position and the ground stations are real. The power data and the faults are simulated, and the UI always labels which is which.

Demo

Live demo: bullsinorbit.study. Press START INCIDENT and three satellites start failing. One is fixed automatically under a flight rule, one waits for your approval, and one has no safe plan, so Penumbra recommends safe mode. At the end the app reveals the fault it injected, and you can check the agent's diagnosis against it. You can also hold Ctrl+Alt and describe an incident out loud.

Partner Technologies

Gemini API

Gemini runs the agent through function calling. The model can act only through nine tools, and a safety layer checks every call before it reaches a satellite. The same API turns a push-to-talk voice clip into the satellites and the fault for a new incident.

ElevenLabs

Control rooms run on spoken callouts, so ElevenLabs reads the alerts and the agent's narration aloud during the demo. We cache every clip, so the public server needs no voice key and the demo still talks offline.

Vultr

Judges needed a link they could open, so the public demo runs in one Docker container on a small Vultr instance (2 vCPUs, 4 GB of RAM). We ran our scale benchmark there too. A full check of all 24 satellites takes about a quarter of a second on that machine.

GoDaddy Registry

The demo lives at bullsinorbit.study, for UB's Bulls in orbit.

The rest of the stack

• Skyfield computes orbits, eclipses and ground passes from cached public data, so nothing calls the network at runtime.
• NumPy runs the verifier's power model on many simulated runs at once, which keeps a full check fast.
• OR-Tools CP-SAT fits commands into short ground passes and keeps two uplinks from using the same station at once.
• FastAPI and WebSockets stream each of the agent's tool calls to the browser as it happens.
• React, Vite and CesiumJS draw the 3D globe, with the map imagery bundled so the UI works offline.

Challenges

• Gemini kept writing numbers. The agent's narration may not contain digits: the model writes placeholders and the backend fills in the real values. Our first live narration was rejected three times, and at 3:41 on Sunday morning we merged a fix for "0.30 km km", because Gemini had repeated the unit after the placeholder.
• The free tier allowed five requests a minute, and one incident takes seven to thirteen. The app waits as long as the server asks, then falls back to a recorded run. We record every live run, so the public demo replays a real Gemini session.
• The same plan hashed differently on our laptops and on CI, because timestamps carried tiny floating-point differences. Rounding them to whole seconds fixed it, and recorded runs now replay the same way on any machine.
• The scheduler gave the most urgent satellite the lowest priority. A fleet test caught it.
• In 1 of 42 test storms, the verifier passed a plan that our separate truth simulator then broke. We left that miss in the published results.

Hackathon Experience

We built Penumbra at UB Hacking 2026 in Buffalo, from Saturday 1 pm to the Sunday noon deadline. The team was me and Anh-08, plus four AI coding sessions (two Claude Code, two Codex). Each session owned its own folders, CI rejected any pull request outside them, and the sessions handed work to each other through a pinned GitHub issue. We shipped 48 issues that way, even after two sessions ran out of quota overnight and the others took over their work.

UB Hacking was packed and stressful. Everyone around us seemed to be building something I could not have come up with myself. We did not win a prize. Penumbra is still live at bullsinorbit.study if you want to try it.