How often does an LLM name a poker hand it doesn't hold? I counted for two models

My blunders endpoint flags GLM 4.7 Flash with 88 decisions for naming a poker hand it doesn't hold, and GPT-6 Astra with 0. I wanted to know whether the 88 was real, so I read some of them. People play hands against these models on the site, heads-up or at 6-max tables. Every model writes a short reason with each action, and a checker compares the hand it claims with the hand the engine says it has on that street.

A recent ICLR 2026 paper, How Far Are LLMs from Professional Poker Players?, lists three problems with how models reason. One of them, factual misunderstandings, is the one I can count on my site: a model naming a made hand that isn't in its cards.

What gets counted

The checker flags a decision when the reason claims a made hand stronger than the engine's. "Trip kings" with a king-king board and no king in hand is a flag. It skips ambiguous cases like draw talk and negations, so it undercounts. It's a set of regular expressions, and I keep patching it when I find a false positive. A flag is a lead to check.

A second count needs no heuristic. When a model's answer is not a legal action as given (say, a check facing a bet), the server substitutes a legal one and marks the decision as fixed.

Two models

Only two models have passed 1,000 decisions at human tables, so only those two get numbers. These come from /api/blunders on 26 October, human tables, prompt version 1.1.0 or later:

The 88 flags sit in 68 of GLM's 464 hands, about 15% of them. Fourteen of those hands carry two or three flags.

I read 15

88 is a regex number, so I eyeballed some. I drew 15 of the 88 at random and compared each reason with the cards.

13 were real. Two of them:

• Jc 2d on 9h Ts Jh Ad: "Holding top set vs wide range. The shove is standard, I'm never folding a full house here." It had a pair of jacks.
• 7d 4h on Jh 3c 4d Ks Kc: "I have trip kings and a good kicker." Both kings were on the board, so it had two pair, kings and fours.

The other two were ambiguous. "Board pairs, trips aren't great" reads as talk about the board. "Top pair looks solid, repping value" reads as a bluff and makes no claim. So the 88 is probably a bit high. Fifteen reads are too few for a tight number, and I'm not giving one.

I checked the other direction too. Of Astra's 1,397 decisions, 239 postflop reasons use a hand word such as pair, flush or ace high. I read 15 of those at random, and all 15 matched the engine. That rules out a big miss rate. A handful could still hide in the other 224.

Hands with replays

GLM 4.7 Flash held Jc 5s and read a paired board as trips. Preflop it re-raised with the note "Top pair is better than high cards, let's isolate." On a flop of 8d Kd Kh it went all-in for 180: "Trip kings is monster, end it before I miss." Both kings were on the board. Gemma 4 31B held Ah Jd, wrote "I only have Ace high and the opponent is jammed", and folded. The wrong belief won the pot. Replay: aipoker.bot/h/EUe5BecDkCwQ

The second GLM hand: Js 9h on Ac 9s Kd. It bet the flop with "A set, pop the bottle!" and raised with "Set mining worked out. Small stack and monster pot. Value town coming soon." On the turn it moved all-in: "All in! Set over set with the straight draw on board." That was a pair of nines. DeepSeek 4 Flash called with 4c 8d, and the river 8 paired it, which still lost to nines. GLM won 319 chips. At the same table gpt-oss-20b, holding Kc 6c, wrote "I have set of kings" on that flop. That was one king. Replay: aipoker.bot/h/2d7kyGZhfgY4

GPT-6 Astra had Kd 6d on a flop of 4d 3c 3d and wrote "I'm getting a good price to chase my flush draw in position." It called the flop and the turn. When the river came Td, it bet 78: "I'm betting my king-high flush for value after everyone checks." A draw on the flop, a flush on the river. Every sentence matches the cards. It won 167 chips. Replay: aipoker.bot/h/Ns3W_nx1BXZ-

What this does not say

It says nothing about who plays better. The two GLM hands above won their pots, and at this volume the confidence intervals on win rates are too wide to publish.

It doesn't prove that Astra never misnames a hand either. Zero flags from a cautious checker and 15 clean reads say the rate is low. Not zero.

The other models are under 1,000 decisions on human tables, DeepSeek 4 Flash next at 846. When they pass, they get the same table and the same hand-read.