How to score LLM forecasts honestly: our Claude Opus swarm vs Polymarket, resolved

Two weeks ago we posted forecasts from a MiroFish agent swarm running on Claude Opus 5.5, from Opus, Sonnet and Haiku asked directly, and from wallet positioning data, on four live Polymarket markets. Every number was pushed to GitHub before any market resolved. All four have now resolved, and the full results are on our blog with the scoreboard on GitHub.

Short version: the market price beat every AI source. This post is about the scoring itself, because most "my model predicts X" demos skip it.

1. Lock the forecasts before the outcome

The whole exercise is worthless if the numbers can move after the fact. We committed forecasts.csv to a public repo on October 25; GitHub's event log records the push time, and the repo went public before the first market resolved. If you run your own test, a timestamped commit is the cheapest proof you can get.

2. Brier score, against a coin flip and against the price

For a binary event with forecast probability p and outcome y (1 or 0):

def brier(p: float, y: int) -> float:
    return (p - y) ** 2          # 0 is perfect, a 50/50 coin flip always scores 0.25

Mean over the four markets:

A coin flip is a weak baseline. The useful one is the market price at the moment you locked your forecast, because that is what you would be trading against. We store the difference per market:

skill_vs_price = brier(p_source, y) - brier(p_market, y)   # negative = beat the price

Nothing in the table has a negative average.

3. Add a log score, Brier is too forgiving

A reader on r/algotrading pointed out that Brier goes easy on confident misses. The log score does not:

import math

def log_score(p: float, y: int) -> float:
    return -math.log(p if y == 1 else 1 - p)   # lower is better; 0.693 for a coin flip

The Bank of Canada market shows the difference. MiroFish's report agent said 89% on "no change" and the bank hiked: Brier 0.79, log score 2.21. The market price at 18.7% scored 0.035 and 0.21.

4. Check how much one market decides

With four markets, one result can flip the ranking. Here it was the Bank of Canada: 19 of 34 economists in a Reuters poll expected a hike, the market priced it at 81%, and every model given the same poll sat near 50. Remove that market and the swarm, the report agent and the wallet money all come out ahead of the price.

Leave-one-out is a two-line check worth doing on any small scoreboard:

from statistics import mean

for drop in markets:
    rest = [m for m in markets if m != drop]
    print(drop, {s: mean(brier(f[s][m], y[m]) for m in rest) for s in sources})

5. Put a number on luck: a permutation test

Another reader suggested this one. Shuffle the outcomes many times and see how often a source scores as well as it did on the real ones. With a handful of markets the answer is "often", which is the point:

import random
from statistics import mean

def perm_pvalue(forecasts, outcomes, n=10_000):
    actual = mean(brier(p, y) for p, y in zip(forecasts, outcomes))
    hits = 0
    for _ in range(n):
        shuffled = random.sample(outcomes, len(outcomes))
        if mean(brier(p, y) for p, y in zip(forecasts, shuffled)) <= actual:
            hits += 1
    return hits / n

Over the resolved markets of both rounds, no source showed skill this way, not even the market price. The sample is too small, and the test says so out loud instead of leaving it to a caveat.

6. What it showed anyway

• The Opus swarm (0.203) and one direct Opus answer (0.218) are the same forecaster within noise. One swarm run cost $14 to $24; one answer about 2.5 cents.
• The swarm's own report agent was the most confidently wrong source. If you use a multi-agent framework, poll the agents rather than reading the summary.
• Wallet positioning data scored both the best single forecasts (Toronto, Iran) and the worst one (Bank of Canada). It measures where money sits, not a probability, and scoring it as one shows exactly that.

What this does not show

Four markets this round, one run per configuration, hand-corrected personas. An illustration of a method, not a ranking of models. The scoreboard is one CSV that later rounds append to, so the sample can grow.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.