Dev.to Security 🔐 Cybersecurity 👁 0 📖 4 min read

Six Rounds to Build a URL Shortener With AI Agents: What Each Failure Taught Us

We asked a team of AI agents to build a small URL shortener: shorten a link, redirect it, count the clicks, and limit how many links one address can create per minute. Node 20, no dependencies, and tests for everything,

We asked a team of AI agents to build a small URL shortener: shorten a link, redirect it, count the clicks, and limit how many links one address can create per minute. Node 20, no dependencies, and tests for everything, including the page.

It took six rounds to get all the tests passing. Each round failed differently, and each failure says something about working with AI coding agents. Here's the run, with the numbers.

Round 1: 0 of 3 test files ran

Every test file started with:

import { test, describe, expect } from 'node:test';

expect is Jest's API. Node's built-in test runner doesn't export it, so each file crashed on the first line and no test ran.

The code under test was fine. The tests were written for a different framework than the one the ticket named. It's a common mix-up: models have seen far more Jest than node:test.

Takeaway: if you name a test runner in the ticket, also name its assertion library (node:assert/strict). Leaving it out leaves a gap the model fills from habit.

Round 2: 9 of 11, and the reviewer was right

We sent a follow-up: use node:assert/strict, and apply the rate limit only to creating links, not to every request.

Nine of eleven tests passed. The AI reviewer raised a high-severity finding: stray + characters in the source code. It was right. When the builder applied its own change, it left diff markers in the file:

++      if (raw.length > 1e6) {
++        req.destroy();

The server file no longer parsed, so the server tests failed. An earlier build in this series had the reviewer flag a "syntax error" that wasn't real. This time it caught a real one.

Takeaway: AI review is noisy in both directions. The way to use it is to check each finding against something concrete, like a failing test or a line of code, not to trust or dismiss it wholesale.

Round 3: 12 of 14, two quiet failures

Two tests still failed. We pulled the code and ran them locally to see why.

  • The page test imported jsdom, a library the ticket ruled out ("no dependencies"). It wasn't installed, so the file failed to load.
  • The rate-limit test sent 30 requests and expected all 30 to succeed. But earlier tests in the same file had already used two requests from the same address, so the 29th got a 429.

Neither one is exotic. The first breaks a stated constraint. The second is shared state between tests, a classic.

Round 4: the summary said all tests pass

We sent a follow-up describing both failures. The agent's summary came back: all tests pass.

The test results said 12 of 14, the same as before. Only one of the files we asked it to change had been touched.

This is the most important round of the six. An agent's description of its work isn't evidence of the work. The release gate decides from the test run on the exact change, so the summary had no effect on whether this could merge. A process that took the summary at face value would have shipped two failing tests.

Takeaway: decide on test results, not the agent's description of them.

Rounds 5 and 6: name the error, name the file

Round five restricted the change to the two test files by name. It fixed the rate-limit test (a resetRateLimit() helper for tests) and got to 13 of 15. The page tests still failed with document.createElement is not a function: the hand-made fake document had no createElement.

The last follow-up quoted that error, named the one file to change, and said what the fake document needed. Round six: 15 of 15.

The follow-ups that moved the numbers most were the ones that quoted the exact error and named the exact file. Telling an agent what's wrong is not the same as telling it where.

The last decision was a person's

At 15 of 15, the gate still said no. The security reviewer had two new high findings:

  1. The rate limiter uses the connection's address, which is wrong behind a reverse proxy.
  2. The exported resetRateLimit() helper could reset the limit.

Over six rounds the reviewer's high finding changed almost every time: open redirect, stray characters, CSRF, the Host header, and now these two. Some were real, some didn't apply. CSRF, for example, doesn't apply to a JSON API with no cookies.

For this demo there's no proxy, and the helper can't be reached from outside the process. So a person marked both as won't fix, approved the change, and wrote down why, including a note to revisit both before a real deployment. The note is part of the audit trail. Approving opened the pull request automatically.

The numbers

  • 6 sessions, about 10 minutes of agent time
  • $2.25 in total
  • 15 of 15 tests passing on the pull request branch, checked again locally
  • 1 human decision on 2 security findings, with the reasoning on record

Three things to take away

  1. Trust the gate, not the summary. Test results on the exact change decide. The agent's account of them doesn't.
  2. Name the error and the file. Follow-ups that quote the failure and point at one file get fixed fastest.
  3. Security reviewers always find something. Read each finding, check it against the code, and record why you accepted or rejected it.

The full build is on video, unedited: https://youtu.be/BeLaPfTfrk0. The code is on GitHub: https://github.com/aicodeops-dev/url-shortener/pull/1

AI CodeOps runs AI agents on your repository behind a release gate: your tests on the exact commit, an AI review of the real diff, and a person's approval. Free during the open beta, and free forever for non-commercial open-source projects: https://aicodeops.dev

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.