LitPivot makes a quiet research bug visible: the metric can outlive the idea

Your research idea changed. Your evaluation spreadsheet still looks excellent.

Those facts can coexist for an uncomfortable reason: the spreadsheet may still be answering the old question.

I thought about this while reading Ai2’s LitPivot write-up on Hugging Face. The system connects an early research proposal with selected literature and separates the proposal into its problem, contribution and evaluation. Changing one part can leave another needing revision. That dependency is more interesting to me than another tool offering to polish a paragraph. [1]

Here is a small example of the mismatch. Both rankings below have perfect topic precision in their first three results. One finds none of the contradictory evidence in the collection. The other finds all of it.

Synthetic counterexample, not LitPivot output. A, B and C support a claim; D, E and F contradict it. Every document is relevant to the topic.

The question moved; the score didn’t

Imagine starting with the question, “Can I retrieve documents about this topic?” Topic relevance is a reasonable first thing to measure. Precision at three counts how many of the first three results are relevant, divided by three.

Now narrow the proposed contribution: “Can I help a researcher find evidence that challenges a particular claim?” You have added a requirement about the evidence’s relationship to the claim. A document can be relevant to the topic while offering no contradiction.

In the constructed collection, all six documents are relevant. Put A, B and C first and topic precision at three is 3 / 3. Put D, E and F first and the same metric is still 3 / 3.

For the disconfirming-evidence subtask, I can also count contradiction recall at three: the number of contradictory documents retrieved in those positions, divided by the three contradictory documents available. That gives 0 / 3 for the first ranking and 3 / 3 for the second.

The old metric hasn’t become mathematically wrong. It has become incomplete for the revised claim. The spreadsheet is innocent; the job description changed without telling it. 🧮

Make the blind spot executable

Save this as metric_demo.py and run python3 metric_demo.py. It uses only Python’s standard library. The rankings and document labels are prescribed teaching inputs; no retrieval model or language model generates them. 🧪

from fractions import Fraction
from itertools import permutations

# Synthetic documents, all relevant to the same topic.
relevant = set("ABCDEF")
contradictory = set("DEF")
rankings = {"A-first": "ABCDEF", "D-first": "DEFABC"}

def score(ranking, k=3):
    top = set(ranking[:k])
    return (Fraction(len(top & relevant), k),
            Fraction(len(top & contradictory), len(contradictory)))

for name, ranking in rankings.items():
    precision, recall = score(ranking)
    print(f"{name}: topic precision@3={float(precision):.2f}, "
          f"contradiction recall@3={float(recall):.2f}")

all_scores = [score(r) for r in permutations("ABCDEF")]
assert len(all_scores) == 720
assert {p for p, _ in all_scores} == {Fraction(1)}
assert {r for _, r in all_scores} == {Fraction(0), Fraction(1, 3),
                                   Fraction(2, 3), Fraction(1)}
histogram = {str(v): sum(r == v for _, r in all_scores)
             for v in sorted({r for _, r in all_scores})}
assert histogram == {"0": 36, "1/3": 324, "2/3": 324, "1": 36}
print("all permutations:", len(all_scores))
print("contradiction-recall counts:", histogram)

The output is:

A-first: topic precision@3=1.00, contradiction recall@3=0.00
D-first: topic precision@3=1.00, contradiction recall@3=1.00
all permutations: 720
contradiction-recall counts: {'0': 36, '1/3': 324, '2/3': 324, '1': 36}

I enumerate every ordering to check that this isn’t just a conveniently chosen pair. There are 6! = 720 permutations. Topic precision at three remains one throughout them, while contradiction recall takes four different values.

There is also a direct combinatorial check. To retrieve exactly j contradictory documents among the first three, choose j from the contradictory set and 3-j from the supporting set, then order both the selected and remaining documents. The count is:

choose(3, j) × choose(3, 3-j) × 3! × 3!

For j = 0, 1, 2, 3, that produces 36, 324, 324 and 36. It agrees with the enumeration. ✅

This count describes possible orderings of a tiny constructed collection. It is not a probability estimate for a real retriever, a LitPivot result, or evidence that one system outperforms another.

What LitPivot actually contributes to this discussion

LitPivot retrieves literature through Asta Paper Finder, groups it around the proposal facet being edited, and links suggested revisions to supporting excerpts. Researchers choose the papers and can accept, edit or reject the revisions. The system can flag connected passages when a change leaves them out of alignment. [1, 2]

In the reported lab study, 17 researchers used LitPivot and a baseline editing/chat setup with the same paper collection. Blind domain-expert ratings concerned literature grounding; novelty and usefulness improvements were participant reports. Those are different evidence types. Better grounding is valuable, but it is not an external demonstration that an idea is unprecedented or will work. [1]

That is why I would keep three questions separate when revising a proposal: what problem does the cited work establish, what does the new contribution add, and what observation would show that addition helped?

When the contribution changes, trace that change through the labels, dataset construction, splits, baselines and failure cases. In my example, adding a contradiction-retrieval claim requires stance labels. It also requires a definition of contradiction: different entities, conditions or time periods can make apparently conflicting statements compatible.

⚠️ Optimizing only for contradictory results would introduce another problem. A literature review also needs supporting work, qualifications and null findings. The additional metric checks one explicit subtask; it does not certify a balanced review or the truth of the underlying claim.

The useful habit is to ask which old evaluation would remain unchanged even if the new contribution failed completely. That is often the first metric worth replacing or supplementing. 💡

References

1. Ai2’s LitPivot explanation on Hugging Face.
2. LitPivot research paper.

Asset credit: original illustration of the prescribed synthetic rankings, checked by exact enumeration.

AI disclosure

AI was used to draft the article.