Most of Your Examples Were Labelled Normal and So Were Most of the Urgent Tickets

The triage prompt
had a short definition
of urgent and normal,
and six examples.

Real tickets,
copied from the queue,
with their labels.

Five were normal,
because most tickets are.

The last two
were among them,
because that is the order
they were pasted in.

For a week
the model sorted the queue.

Then someone noticed
a payroll outage
had sat for a morning
marked normal.

So had a locked out branch.

So had a fraud report.

None of them
were hard to read.

The examples had taught
something nobody wrote down.

In 2021
Zhao and colleagues
measured this in GPT-3
and named three habits.

Majority label bias.

The model leans
toward the label
that appears most often
in the examples.

Recency bias.

It leans
toward the labels
near the end of the prompt.

Common token bias.

It leans
toward answers
that were common words
in its training.

Changing only
which examples went in,
and in what order,
moved accuracy on some tasks
from close to chance
to close to the best results
of the time.

The questions
had not changed at all.

Newer models
are steadier than that one.

Steadier is not immune.

So treat the examples
as part of the classifier,
because they are.

Balance them.

As many urgent as normal,
even though the queue is not.

Shuffle the order,
and do not let
the same label
close the list every time.

Then test it the boring way.

A hundred tickets
with labels people agree on,
scored against the prompt.

Run the same set again
with the examples reordered.

If the scores move,
the order is deciding,
not the tickets.

Watch the label counts as well.

If the model calls
ninety five percent of tickets normal
and the real share is eighty,
it has learned the prompt,
not the job.

And when you change an example,
run the set again.

Swapping one ticket
for a better one
is a change to the model's instructions,
even if nobody calls it that.

Your examples
were never a sample of the queue.

They were a lesson,
and the model learned
every part of it.

– Serguey Asael Shinder