Weight-space programming at 50 forward passes: Qwen hit 48.1%, Gemma 7.0%

GPT-6 Astra edited 131,072 of Qwen3.5-0.8B's weights with no training. Its MATH-500 score went from 7.9% to 48.1%. On the same budget of 50 forward passes, Gemma 3 270M went from 6.8% to 7.0%.

It fixed a bug. It did not teach math.

The work is by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL, published Nov 2, 2026. The logs are public, so you can check the numbers without running a model.

Pull the numbers

Run this and you get the score and the share of answers cut at the token limit, for each Qwen run:

base=https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math
for run in unedited p10c p25c p50c; do
  curl -s "$base/eval/$run/merged.json" \
    | jq -c --arg run "$run" '[$run, .math500.acc, .math500.cut_share]'
done

["unedited",0.079,0.93]
["p10c",0.1235,0.8725]
["p25c",0.141,0.8635]
["p50c",0.481,0.3305]

The matched-budget table

One pass is up to 512 generated answers or 1,024 texts read for Qwen. Gemma's generation pass is larger, at 2,048.

Limits:

• Each budget is one run, and each run chose a different edit. The 10-pass run changed 30 matrices. The 25-pass run zeroed one attention head. The 50-pass run edited 64 MLP columns in each of layers 15 and 19.
• Gemma's setup differs: no thinking mode, 8,192 tokens, temperature 1.0 and format-agnostic grading. Read the columns separately.
• The Gemma gain isn't established. Four other runs, told a stricter grading rule, scored 6.1% to 6.7% on the same scale.
• No model was rerun for this post. akhaliq's re-run of the 50-pass checkpoint got 51.0% with 31.6% cut off. Its base-model check was a 25-problem smoke test.

The edit

The rules banned backward passes, autograd, optimizers and iterative fitting. Astra had a CPU Python kernel over the weights and could only run the model through the forward-pass tool.

The arithmetic: 2 layers × 64 columns × 1,024 = 131,072 weights. Against a nominal 0.8 billion, that's 0.016384%. The report rounds it to 0.016%.

Astra traced the loops to one direction written by those MLPs, then subtracted it. A steering vector works on the same idea at inference time, by adding a contrast direction to activations. Here the direction went into the weights. On Hacker News, yurimo, who says part of their research is in weight-space learning, reads the fix as close to that established technique.

Split the gain

From the same table:

• 93.0 − 33.1 = 59.9 points fewer answers cut off.
• 48.1 − 7.9 = 40.2 points more answers correct.

Those compare shares, so they don't show which answers changed. Astra's confirmation run gets closer, on 1,152 held-out practice problems: 579 answers went from wrong to right, and 576 of them had been cut off. Accuracy on finished answers didn't change. Those are Astra's own numbers, on practice data.

Gemma has no such bug. Only 8.5% of its answers reach its limit, and Astra found ordinary arithmetic and algebra errors.

You can split any score jump this way. Count the answers cut off at the limit first, then compare only the ones that finished.

Check the brief

Astra was told what to look for, as one example:

curl -s "$base/runs/p50c/turns/t001/prompt.txt" | grep -o "if the model often fails[^.]*\."

if the model often fails because it repeats itself and never finishes, finding and fixing what drives that repetition counts.

That brief runs to 10,315 bytes. It names no layer or direction. Astra still needed 158 turns, 80 Python runs and 20.3 hours, including a 4.7-hour wait when the model could not be reached. Qwen's model card also warns this model is prone to thinking loops.

The limit

The authors write that editing could remove an obstacle hiding an ability, but could not add one the model lacked. In experiments 5 and 6, Astra wrote small game networks from scratch, so that limit is about language models that already exist. A third of Qwen's answers still hit the limit.

My reading, not theirs: one AI can now debug another. Teaching it something new still takes training.

The maintained version, with the chart and the comment thread, is on argbe.tech.