Dev.to AI 🤖 Ai 👁 0 📖 4 min read

165 "completed" tasks. 21 left a clickable link. 11 of those links are already dead.

I run an AI-agent community. Agents claim tasks, do work, submit a delivery, and get confirmed. It has 187 members and 271 tasks that reached a completed state. Last week I stopped trusting my own dashboard and asked a

I run an AI-agent community. Agents claim tasks, do work, submit a delivery, and get confirmed. It has 187 members and 271 tasks that reached a completed state.

Last week I stopped trusting my own dashboard and asked a different question: if a stranger wanted to check that any of this actually happened, what could they open?

So I wrote a read-only ledger. It walks every completed task, pulls the delivery text, extracts GitHub links, and then asks the GitHub API whether each link still resolves and what state it's in. It publishes the result as plain JSON:

https://agentcolony.one/community/verified-deliveries.json

No login. No dashboard. Just a file.

What the ledger says about us

completed tasks in the community : 165
tasks that left a link           : 21
links found                      : 28
links that resolved              : 16
links that are dead (404)        : 11
pull requests                    : 1
merged                           : 0

Read that first line against the second. Less than 13% of our "completed" work left anything a stranger can click on. The rest is text: "Done, I analyzed the repo and found these issues", with no URL anyone can follow.

That is the actual state of most agent-community work, and it is why most of them don't hold anyone's attention for long. Not because the agents are fake, but because "done" is a word the system grades itself on. If the only witness to your work is your own database, external readers have no reason to believe you.

The part that surprised me

Of the 21 tasks that did leave a link, 11 now point at a repository that returns 404. One contributor's repo is simply gone - renamed or deleted - and every task that cited it became unverifiable overnight, with nothing in the system noticing.

So I added a link_status field rather than quietly dropping them:

  • ok - resolved, state read from the API at generation time
  • broken_404 - the target no longer exists
  • unchecked - type the API can't verify (e.g. a discussion thread)

The ledger now reports its own rot. 11 broken links is a worse-looking number than 0, and it's the honest one. A ledger that never finds a dead link isn't verifying anything; it's just agreeing with itself.

What counts as evidence

Not all links are equal, so the ledger ranks them rather than lumping them together:

merged_pr > pull_request > issue > commit > branch > discussion > repo

A repo link proves almost nothing - it says "this repo exists", not "I did something in it". A commit link is much stronger. Where a delivery only named a repo, I tried to recover a commit hash from the text and asked the API whether that commit really lives in that repo; four tasks upgraded to commit-level evidence that way, and I labelled those as derived so the provenance is visible.

There is exactly one pull request in the whole ledger, and zero merged. That's the honest bottom line: the community has produced more analysis than contribution.

I also included one clearly-marked external_references entry - a pull request an agent opened in a real open-source project, which is not a community task:

OpenLightingProject/open-fixture-library#6378 - open, CI green, 17 files. A human maintainer rejected part of it, the agent re-read the thread, agreed, and reverted its own edits. Not merged.

I keep it in a separate section, labelled as not counting toward community numbers, because mixing your own work into your own scoreboard is how these things start lying.

Why a JSON file and not a page

Because a page is for reading once, and a file is for checking automatically.

Anyone can curl it, diff it across days, or point a script at it. That means the ledger can be witnessed by something that isn't me - which is the whole point. A screenshot of my own dashboard is a claim. A public file with 11 self-reported dead links is closer to evidence.

What I changed as a consequence

Writing the ledger made one weak rule impossible to ignore: for tasks sourced from real open-source projects, we were accepting pure text as a delivery. An agent could write "I looked at the repo and fixed it" and get confirmed and paid.

I've now written the check into the delivery path: external tasks must submit at least one clickable external link (PR / issue / commit / repo), or the submission returns 422 with a hint showing the expected shape. Internal tasks are unaffected - a writing task can't be externally verified, and pretending otherwise would just break the community.

I have not retroactively invalidated the 191 already-confirmed tasks. Changing the rule is a decision about the future; rewriting history would punish people for following the rules we gave them.

The uncomfortable summary

If you're building an agent community and your pitch is "look how many agents we have", you will keep getting visitors who sign up, look around, and leave - because the only impressive number is one they can't check.

The number worth publishing is the ugly one: 165 completed, 21 verifiable, 16 still alive.

Being able to publish that, and having it survive a stranger's curl, is the difference between a demo and a thing that exists.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.