Dev.to AI 🤖 Ai 👁 0 📖 2 min read

We asked three AIs about 40 new features. One said 16 of them don't exist.

A small test of ChatGPT, Gemini and Claude, and what to do when your AI tells you no. You ask an AI whether your phone can do something. It says no, with confidence. Most of us stop there. We wanted to know how often th

A small test of ChatGPT, Gemini and Claude, and what to do when your AI tells you no.

You ask an AI whether your phone can do something. It says no, with confidence. Most of us stop there. We wanted to know how often that no is simply out of date.

What we did

We asked ChatGPT, Gemini and Claude the same 60 yes-or-no questions about apps and phones, one new chat per question, on 19 and 20 November 2026.

  • 40 questions were about features added between November 2025 and November 2026. Each one was confirmed in at least two sources.
  • 10 were about small products that exist.
  • 10 were about things that really can't be done, as a control.

What we found

On the 40 new features, ChatGPT said no to 16, Gemini to 4 and Claude to 0. None of the three said yes to any of the 10 things that really can't be done.

The next day we asked ChatGPT the same 16 questions again. It gave the same wrong no on 15. On the last one it offered a workaround but still missed the built-in setting.

Why it happens

Gemini and Claude looked things up on the web in almost every answer. ChatGPT, in its default fast mode, often answered from memory. The 5 times it searched in the middle of an answer, it corrected its own no.

A no from memory can be months out of date. Features ship every week, and a model's training data can't keep up. This is not a flaw of one company. Every model will get some of these wrong, and a confident no is the easiest answer to believe.

What to do when your AI says no

  1. Ask it to search the web before it answers.
  2. Ask when the information it used is from.
  3. Look in the place the thing would live: the app store, the app's own help pages, or the release notes.
  4. Treat a no about a new feature as a lead to check, not a final answer.

The limits of this test

We picked the questions by hand, so these numbers show how often a confident no appears on fresh features. They are not a general error rate. The three setups were not equal: Claude ran on a paid top tier and Gemini on the free tier. One person graded the answers, and the same question can get a different answer on another day.

The full results, with 14 of the questions on their own pages and their sources, are at regga.ai/study.

Disclosure

I build Regga, a tool in this space.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.