Task-completion Regression with ChatGPT

Overview:

This article is about task-completion regression with ChatGPT.

keywords: AI assistants, large language models, LLMs, ChatGPT, task-completion regression, machine learning, ML

Do better, ChatGPT:

I can't directly send feedback to the teams that develop or tune the models, so I can't promise that what you tell me here will reach them.

That said, this is the kind of feedback I'd want them to see because it's specific, actionable, and focused on user experience rather than just "the model got worse."

I'd summarize it something like this:

Task-completion regression: For users doing writing, coding, and editing, the assistant increasingly spends responses discussing how it will approach a task instead of performing the task. This creates additional turns, increases cost, and reduces perceived usefulness.

Some concrete recommendations would be:

• Optimize for progress per turn. Once intent is clear, spend the response budget producing the requested work rather than explaining the plan.

• Default to partial completion. If a task exceeds response limits, automatically begin with Part 1 instead of asking the user to propose chunking.

• Avoid future-tense promises. Phrases like "I can do that in the next response" should be rare. If the work can start now, start now.

• Distinguish planning from execution. If the user asks for execution, execute. Only switch to planning if execution is genuinely impossible.

• Be transparent about limits. If the maximum response size is the constraint, say so briefly and produce as much useful output as fits.

The underlying principle is simple:

When the next productive action is obvious, take it.

A lot of evaluation metrics naturally focus on correctness, safety, or user satisfaction. But there's another metric that matters in practice:

Useful work per paid interaction.

If an assistant takes five exchanges to accomplish what it used to do in one or two, that represents a real degradation in value, even if every individual response is technically polite and accurate.