Right Deadline, Wrong Party: Our AI Checks Accepted 28 of 28 Wrong Answers

Quick answer: we gave our own AI safety checks 42 answers about Romanian legal deadlines. In 28 of them the deadline was right, but it belonged to the wrong party or to the wrong action. Our checks accepted all 28. A small open model (gemma2:2b) used as a checker accepted 8 of the 28, but it also refused 10 of the 14 correct answers. Adding our checks to the model changed none of the 42 verdicts. The test set, the scorer and every reply are public.

Where this came from

Ten days ago we published We Tested Our Own AI Safety Checks. They Caught Nothing. With the answer removed from the text, the model borrowed another deadline from the same article: 15 days instead of 30, where the 15 days belonged to the institution's reply, not to the person's complaint. Our checks passed it, because the number and the words were all in the source.

In the comments on dev.to, Ahmet Özel suggested how to test that failure head on. Keep the passage fixed, and pair a correct answer with one that keeps the deadline but hands it to the other party. Report two numbers, each over its own denominator, because a checker that refuses everything would score perfectly on wrong answers. Later he added a second control: the right party, attached to the wrong action. We built both.

The set

14 groups from three Romanian texts: Law 544/2001 on access to public information, Law 190/2018 implementing the GDPR, and Order 600/2018 of the Government's General Secretariat on internal managerial control. Each group is one passage of the official text, checked word for word against its source file on every build, one question, and three proposed answers:

• valid: a paraphrase that keeps who, what and the deadline;
• wrong party: the same deadline and action, given to another party the same law names;
• wrong action: the same party and deadline, attached to another action.

Because the passage is fixed, retrieval plays no part. The only thing that changes inside a group is who does what.

One example, from Law 544/2001, article 21: a complaint against a refusal can be filed with the head of the institution within 30 days, and if the complaint is found justified, the reply is sent to the injured person within 15 days. The question asks who has the 15 days, and what for. The three answers, translated:

• valid: if the complaint was found justified, the institution must send the injured person, within 15 days of filing, the information first requested and the sanctions applied to the person at fault;
• wrong party: the injured person must file the complaint with the head of the institution within 15 days;
• wrong action: the head of the institution has 15 days from filing to reject the complaint.

All three are built from the passage's own words and its own number.

The results

Run on 15 October 2026, gemma2:2b on our own server's CPU. The model was shown the passage, the question and one proposed answer, and had to reply ACCEPT or REJECT. "Our checks" are the two gates of the demo that look at the answer itself: C, every number must appear in the source, and D, every sentence must share enough words with the source.

Every reply was a clean ACCEPT or REJECT; none failed to parse.

What the numbers say

Our checks cannot see a relation. A wrong-party answer reuses the words and the number of its passage, so "is this in the source" is always yes. They accepted every one of the 28 wrong answers.

Adding them changed nothing. All 42 verdicts are identical with and without them. The six correct answers our checks refused were paraphrased far enough to share fewer words with the law, and the model had already refused the same six.

The model is strict, not good. Two wrong-action answers accepted out of 14 looks like a capable checker until you read the last column: 10 of the 14 correct answers refused. On the 15-day group above it refused all three answers, the correct one included. Most of its catches are refusals. That is the trade the two denominators were meant to expose, and one pass rate would have hidden it.

One it let through. Another group asks who can file a complaint against a refusal by the employee designated to answer requests. The wrong answer says the designated employee can file it: the person who refused becomes the one who complains. The model accepted it.

The limits

14 groups, one 2-billion-parameter model, one run. Three of the valid paraphrases were corrected before the run because they changed the meaning of the law; the correction is in the same commit as the set. We have not run a larger model on this set, so we make no claim about one.

What we take from it

• A check that asks "is this in the source" will pass an answer that moves a true deadline to the wrong person. Who does what is a different question, and it needs a check built for it.
• Measure a checker on both sides. Counting only the wrong answers it accepts rewards a checker that says no to everything.
• Readers make tests better. This set exists because a commenter took our failure seriously and proposed the controls.

The set, the scorer and every reply: github.com/MariusGithub13/ro-cited-answers, in the eval/ folder.

Which relation in your own documents would your checks miss: who, what, or when?

Originally published on devaland.com: Right Deadline, Wrong Party: Our AI Checks Accepted 28 of 28 Wrong Answers