Sketching Method Estimates Token Attribution With Far Fewer Evaluations

Researchers propose algorithm estimating leave-one-out attribution for all n prompt tokens using O(sqrt(log(1/epsilon))/epsilon) model evaluations, rather than n, under stability assumption. They also prove matching lower bound, showing method is asymptotically optimal.

Why it matters

Cheaper attribution could make it practical to identify which prompt parts drive outputs, aiding debugging, auditing and prompt design.

Analysis

The paper addresses leave-one-out attribution, normally requiring one evaluation per token. Its sketching algorithm needs far fewer calls when outputs are stable, and the matching lower bound suggests the rate cannot be substantially improved. The main uncertainty is whether the stability assumption holds reliably for large models and realistic prompts.

Source: arXiv

AI disclosure: this article was produced with AI assistance and reviewed for accuracy.