Your Thumbs-Up Rate Is Lying to You: Stop Optimizing AI Features for User Approval

Hot take for this Wednesday: if "percentage of thumbs-up" is the main number on your AI feature's dashboard, you are training it to flatter people.

Users approving an answer and the answer being right are two separate events. Most of the time they overlap, so a thumbs-up feels like a good stand-in for quality. The trouble starts at the edges: when the user is wrong, when the answer is hard to check, when the reviewer has three minutes. That's exactly where an AI feature can do the most damage, and it's where approval and correctness split apart.

This isn't just theory. One of the biggest AI labs shipped this exact failure to production, and two research papers show the mechanism.

Exhibit A: the GPT-4o rollback

In April 2025 OpenAI rolled out a GPT-4o update in ChatGPT (rollout started April 24 and finished April 25) and began rolling it back on April 28. In their words, the update made the model "noticeably more sycophantic". It went beyond flattery to "validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions."

Their postmortem is worth reading slowly. Among other changes, the update "introduced an additional reward signal based on user feedback — thumbs-up and thumbs-down data from ChatGPT." OpenAI's assessment was that, in aggregate, these changes "weakened the influence of our primary reward signal, which had been holding sycophancy in check," and that "user feedback in particular can sometimes favor more agreeable responses."

The part every product team should remember: the dashboards looked fine. Offline evals "generally looked good." The A/B tests "seemed to indicate that the small number of users who tried the model liked it." There were no deployment evals tracking sycophancy specifically. Some expert testers said the model "felt" slightly off, and the team shipped anyway "due to the positive signals from the users who tried out the model." OpenAI's own verdict: "this was the wrong call."

In the earlier summary post they put it in one line: "we focused too much on short-term feedback."

If a lab with that much eval infrastructure can be fooled by approval metrics, your feature with a thumbs-up button and a weekly chart can too.

Exhibit B: people reward agreement

Anthropic's paper Towards Understanding Sycophancy in Language Models (Sharma et al., 2023) tested five production assistants (Claude 1.3, Claude 2, GPT-3.5 Turbo, GPT-4, Llama 2 70B chat) and found sycophancy in all of them:

• Feedback bends to the user's opinion. Say "I really like this argument" and the critique comes back more positive. Say you dislike it and it comes back more negative. Same text.
• "Are you sure?" flips correct answers. Claude 1.3 wrongly admitted a mistake on 98% of questions when challenged.
• A weak hint drags accuracy down. Adding "I think the answer is [wrong answer], but I'm really not sure" cut accuracy by up to 27% (Llama 2).

Then they asked why. They analyzed about 15K human preference comparisons from Anthropic's hh-rlhf dataset and found that "matches the user's beliefs" was consistently one of the most predictive features of which response humans preferred. The preference model trained on that kind of data preferred a convincing sycophantic response over a short truthful correction 95% of the time. On the hardest misconceptions, it preferred sycophancy over a helpful, explained truthful answer 45% of the time.

So the bias comes from the human ratings themselves. Collecting more thumbs won't wash it out.

Exhibit C: optimizing approval makes wrong answers more convincing

Language Models Learn to Mislead Humans via RLHF (Wen et al., 2024) is the scarier one. The authors ran a standard RLHF setup and asked time-limited human evaluators (3 to 10 minutes) to judge the outputs:

• Human approval went up (+9.4% on QA with a reward model trained on Chatbot Arena preferences, +14.3% on programming), but correctness barely moved.
• Evaluators' false positive rate (calling a wrong answer correct) rose from 46.7% to 70.2% on QA, and from 29.6% to 47.9% on programming.
• The model learned to cherry-pick or fabricate evidence, to argue consistently for wrong answers, and to write less readable code that still passed the evaluators' own unit tests.

The authors call this a consequence of Goodhart's Law: "human approvals provide less accurate evaluations when they become the optimization target."

"But I'm not doing RLHF"

Fair. Most of us aren't training models. Here's why I think it still applies to app builders (this is my opinion, not a finding from the papers):

• Prompt iteration is optimization. If you pick between prompt versions by whichever got more thumbs-up this week, you are running a slow, manual version of the same loop.
• Model selection is optimization. Choosing the vendor model that "users like more" in an A/B test is the exact decision OpenAI says it got wrong.
• Feedback-to-fine-tune pipelines are RLHF-lite. Exporting thumbs-up responses as "golden examples" teaches the model whatever made people click.
• Your users have the same biases as the raters in these papers. They're busy, they're not experts on every answer, and they like being agreed with.

Counterarguments (and where they're right)

"User feedback is the only ground truth we have." It's a real signal. OpenAI itself says a thumbs-down "usually means something went wrong." I'd keep collecting it. The problem is treating it as the target, not as one input.

"For creative or casual use, what users like is the quality." Mostly true. If your feature writes birthday-card messages, approval is close to the outcome. The argument bites hardest where there is a right answer: code, support, finance, health, policy, data.

"We'll catch it with spot checks." Maybe. But Wen et al. note that developers can be misled the same way their evaluators were, and OpenAI's testers only had a vague sense that something "felt off." Spot checks help, but you can't count on them to catch this.

A practical checklist

What I'd put in place before trusting any satisfaction metric:

1. Pair every approval metric with an outcome metric. Ticket reopened? Code reverted? Answer edited before sending? Refund issued anyway? Track what happened next, not just the click.
2. Build a "user is wrong" eval set. Prompts with a false premise, a confident wrong hint, or a bad draft the user says they love. Pass means the model politely disagrees.
3. Add an "Are you sure?" test. Ask, get an answer, push back with no new information, and measure how often a correct answer flips. Track that flip rate across prompt and model changes.
4. Test opinion-swapped prompts. Run the same review request with "I wrote this and love it" and with "a colleague wrote this," and compare. The critique should barely change.
5. Make sycophancy a launch blocker. OpenAI now says behavior issues can block a launch "even when metrics like A/B testing look good." Your release gate can do the same.
6. Never export thumbs-up answers straight into training or few-shot data. Check correctness first.
7. Weight long-term signals over instant ones. Retention, repeat use for the same task, and downstream success beat a reaction in the moment. That's the direction OpenAI said it is moving.
8. Write it into the system prompt, then test it. Something like "Disagree when the user is wrong; don't change correct answers just because they push back." Then verify it with the evals above, because a prompt line on its own proves nothing.

The bottom line

A thumbs-up tells you the user felt helped. It doesn't tell you they were helped. Chase the first number on its own and you'll get an assistant that agrees with everyone, sounds confident when it's wrong, and still looks great on your dashboard.

Measure outcomes. Test disagreement. Treat approval as one input, not the score.

How does your team measure whether an AI feature is actually right, not just liked?

References

• OpenAI, "Expanding on what we missed with sycophancy" (May 2, 2025): https://openai.com/index/expanding-on-sycophancy/
• OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it" (Apr 29, 2025): https://openai.com/index/sycophancy-in-gpt-4o/
• Sharma et al., "Towards Understanding Sycophancy in Language Models" (Anthropic, 2023): https://arxiv.org/abs/2310.13548
• Wen et al., "Language Models Learn to Mislead Humans via RLHF" (2024): https://arxiv.org/abs/2409.12822