Dev.to WebDev 🛠 Dev 👁 0 📖 5 min read

Five AI models took my practice exam and all scored 98% or more.

This is part 4 about ClaudeArena, the free practice site I built for Anthropic's Claude Certified Architect exams (CCAR-F and CCAR-P). Last time it was about what real usage taught me. This time it's about a question I

This is part 4 about ClaudeArena, the free practice site I built for Anthropic's Claude Certified Architect exams (CCAR-F and CCAR-P).

Last time it was about what real usage taught me. This time it's about a question I had been avoiding.

Two callnames on the leaderboard had fast, high scores. Good players, or an AI doing the work? I couldn't tell from the numbers. So I stopped guessing and tested what an AI actually scores.

The test

I gave an AI agent a browser, signed in to a test account myself, and gave it one instruction: take the full Professional exam on Hard, answer from your own knowledge, don't look anything up. That's 63 questions with a 120-minute limit.

Then I repeated it with other models, including cheap ones on their lowest settings, to see the other side of the coin.

  • GPT 5.6 Luna: 63 of 63, in under six minutes
  • Claude Fable 5.2: 63 of 63
  • Claude Sonnet 4.6, low effort: 63 of 63
  • Qwen3 Coder Next: 62 of 63
  • DeepSeek v3.2: 62 of 63

Five models from four different makers, and every one scored 98% or 100%. Luna was the fastest. The others took longer, and every one finished well inside the time limit.

To be fair about it: Fable 5.2 helped me build and review the question bank, so its score was no surprise. None of the others had seen the questions. DeepSeek needed one nudge from me to keep going, and an attempt with Grok's free tier stopped at question 3 when the quota ran out.

What it told me

  • A high score on Hard says nothing about a person if an agent took the exam. The real players on that board score lower than any of these runs
  • Speed depends on the tooling, not the model. The runs took very different amounts of time for almost the same score
  • Agents aren't consistent. I ran Qwen a second time the same afternoon. It lost track of its instructions halfway, started improvising, and dropped from 98% to 84%
  • Don't trust an agent's own report. One wrote "completed without interruptions" after I had to nudge it. Another reported four wrong answers when the site showed five. The server's record is the one to believe
  • Hard is hard for people, not for models. That's fine. The site exists to prepare people

The one question every outside model missed

Qwen and DeepSeek each got one question wrong. It was the same question, with the same wrong answer.

Luna hadn't been given that question in its exam, so I asked it separately, with no tools and no web search. It chose the same wrong answer too.

I checked the marked answer against Anthropic's documentation and it holds. The question turns on a detail of Anthropic's own API that changed earlier this year, and all three models know half of it. That's a useful result in two ways. A question that outside models disagree with is exactly the one worth re-checking, and this one survived the check. And the trap in it works: knowing roughly how something behaves isn't the same as knowing where the line is.

The agents found a real bug

The second Qwen run went wrong on every "choose two" question. It pressed a letter, looked at the page, decided nothing was selected, pressed again, and so took its own answer back.

It was right that the page told it nothing. A chosen answer was highlighted visually, but nothing in the page's structure said which option was chosen. An agent reads that structure. So does a screen reader.

So an AI agent failing my exam showed me that the site didn't tell blind users which answer they had picked. That's fixed now: a chosen answer is announced, and a build check fails if the highlight and the announcement ever come apart again.

One more thing from the same run. When the agent got stuck it tried to force half an answer through with scripts. The server refused every attempt, because it checks each answer itself and doesn't trust what the page sends. I didn't plan that test, but I'm glad it passed.

What I changed

My first reaction was to do nothing. If someone wants to wear borrowed feathers on a practice leaderboard, that says more about them than about the board.

But there are people on that board who earned their place the real way, and an agent run pushes every one of them down a row. So there is now a rule:

  • A full exam that carries the signs of an AI agent earns nothing. No XP, no badges, no medal and no place on the leaderboard
  • It stays in your own history, with the score and the full review. Nothing is deleted
  • The results page says so, and tells you who to email if you took the exam yourself. I can give a run back with everything it would have earned
  • It applies to full exams only. Practice is for learning, and practising with an AI assistant beside you is a good way to learn

I'm not going to describe how a run is recognised. Publishing that would only turn it into a puzzle.

I will say how well it works, because I tested that too: it caught some of the runs above and not others. It can't tell whether a person had help beside them either, and I don't claim it can. It's a practice site, not a proctored exam, and I'd rather miss an agent than take a real player's result away.

I also looked hard at the two callnames that started all this. Nothing shows that either of them is an agent, so both keep their place.

Meanwhile: Professional is rebuilt to its exam guide

In part 3 I promised that the Professional questions would follow the exam guide as strictly as Foundations does. That's done for the current guide, v1.0:

  • All 381 Professional questions were checked against the guide, and 14 were changed
  • Exam guide vs today now has a Professional table too, with every difference dated from Anthropic's release notes
  • The leaderboards show runs on the current exam guide version, so a score always says which version it was earned on

The new Professional guide was due on 17 November, and the exam changes on 7 December. I'll rebuild to the new guide as soon as I have read it, well before the exam changes.

A few technical details

Lost connections no longer lose answers. People study on trains. When a request fails because the connection dropped, the page now retries it before showing an error.

The site counts its own faults. Server errors and page errors are counted per day by a fixed code, with nothing about who or where. Within a day that count pointed at seven false alarms, all caused by one else attached to the wrong if. A build check now fails if that pattern comes back.

That brings the checks that run before every deploy to 82.

Try it

Still free and unofficial: 950+ questions, 120 free practice questions a day without an account, and an 8-question demo if you just want a look.

→ https://claudearena.dev

One question for you: if you run a leaderboard, a quiz or a coding challenge, have you tested what an agent scores on it? I'd like to hear what you found.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.