An LLM edited a small model's weights directly and its math score rose 6x

TL;DR

• GPT-6 Astra, a frontier LLM, edited 0.016% of Qwen3.5-0.8B's weights directly, with no training run. MATH-500 went from 7.9% to 48.1%.
• The authors read most of that gain as unblocking: the base model kept running into its token limit instead of finishing. Astra also read, repaired and wrote small game-playing networks.
• Limits: no gain on Gemma 3 270M, one frontier LLM, models up to 0.8B, a blog post and not a peer-reviewed paper, and a prompt that hinted at the loop. The edited model and the logs are public.

One edit to 0.016% of a small language model's weights lifted its math score from 7.9% to 48.1%. No training run, no gradients: GPT-6 Astra, a frontier LLM, read the weights directly and wrote the fix.

Read the 6x with care. 93.0% of the base model's MATH-500 answers ran to the 32,768-token limit without finishing, and the authors read the gain as mostly unblocking: the model now finishes answers it could already work out. The assignment prompt also pointed Astra at that loop, and people have argued about it since Nov 2.

MIT CSAIL researchers call this weight-space programming: a general-purpose LLM reads and writes a neural network's weights with no optimizer and no gradient descent, guided by a plain-language spec. Training has always been an indirect way to write weights, and their first results suggest a frontier model can write them directly. The post is "Sparks of Weight-Space Programming" by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab.

Check the Qwen claim yourself; replaying the whole run takes a day

The edited model is on Hugging Face as RikiyaT/Qwen3.5-0.8B-weight-space-programming, and only two tensors differ from base.

Hugging Face's Ahsen Khaliq rebuilt it from the logs within a day and shipped a live demo. HF's independent run of the full MATH-500 set scored 51.0%.

Try this: run the same math questions through base Qwen3.5-0.8B and the edited model with the same generation cap, and count how many answers hit the cap. The table below shows the authors' rates at a 32,768-token limit. Then clone the logs repo and open exp8_math/runs/p50c/turns/t001/prompt.txt. It is the exact assignment Astra received. Read it before any commentary, including this post.

Replaying Astra's own session is the expensive part, though the logs hold everything sent to and from Astra, all code and all evaluations. Coverage of the logs (dailyaibrief.com, clauding.de) reports that the successful 50-pass run took 158 conversational turns and 80 Python runs over 20.3 hours. No fine-tuning cost is given to compare that with.

How Astra took a loop out of 131,072 weights

Qwen's own model card warns Qwen3.5-0.8B is prone to "loops compared to other Qwen3.5 models, which may prevent it from terminating generation properly." Of the base model's first 96 practice answers, 85 ran to the 32,768-token limit.

Astra worked in a Python notebook on practice prompts from the GSM8K and MATH training sets. A forward pass is one run of the model, and the budget capped how many runs Astra got: 10, 25 or 50. It traced the looping to one direction, a single pattern written by the MLP blocks of layers 15 and 19. It then subtracted that direction from 64 output columns in each block, so those layers no longer write it. That is 131,072 weights, 0.016% of the model, and the rest stays as trained.

It needed the full 50-pass budget: at 10 and 25 passes, MATH-500 reached only 12.3% and 14.1%. Even at 50 passes, 33.1% of answers still hit the limit. IFEval, which tests instruction following, rose too, which fits the unblocking reading.

A skeptic can call this a bug fix for a documented loop. None of the coverage I read tests whether a decoding-side fix, such as a repetition penalty, would match it.

Did Astra find the loop, or was it told where to look?

Researcher Lucas Beyer read the assignment prompt, as Turing Post reported on Nov 2. It told Astra that if Qwen "repeats itself and never finishes, finding and fixing what drives that repetition counts." Coauthor Jayden Teoh replied that the team gave the short assignment via Claude Code.

There is a fair counterpoint. Qwen's model card had already flagged the loop, so pointing at it was reasonable.

The third-party re-run confirms the edit works. How much credit Astra deserves for finding the loop, given the hint, stays open. A cleaner test would rerun the assignment with that sentence removed. None of the coverage I read reports one, and the prompt is public.

Astra also read, repaired and wrote small networks

• Read. With no list of games and no tools, Astra named the game each of four trained PPO networks played, from raw weights alone. For Breakout it noticed that first-layer weights concentrate along the bottom row, where a paddle lives.
• Fix. In a Kinetix car-ramp task, a PPO policy drove straight at the ramp and stayed stuck. Astra got the weights, a scene description and one recorded episode, with no tools. In one reply it changed 3.5% of the weights and took success from 0% to 100% over 1,024 episodes. The edit added a small path that backs the car up past the red strip, then hands control back to the trained policy.
• Write. Given only a game's rules and source code, with no data, loss, gradients, tools or rollouts, Astra wrote MinAtar networks that beat PPO trained for 100M steps in all three games. In Breakout the score was 517.4 versus 15.8. Against PPO at 500M steps, Astra still wins Breakout by 20x.

Where it breaks: Gemma 3 270M gained nothing established

Gemma 3 270M rarely loops: 8.5% of its answers hit the token limit. MATH-500 started at 6.8% and four edits scored 7.0%, 7.6%, 7.7% and 7.0%. None was established, and Astra said so itself. Its errors are ordinary arithmetic and algebra mistakes, with no single component to switch off. The authors put it this way: "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."

In a separate test on two fine-tuned Gemma copies, Astra failed from weights alone in all 4 attempts, so some hidden behavior shows only in activations.

One frontier LLM was tested, on models up to 0.8B parameters. This is a blog post with logs, not a peer-reviewed paper, and the authors themselves call it "sparks" and "very early." They are building a benchmark and harness to make this "a rigorous, reproducible engineering discipline."

Sources: numbers come from the authors' post and logs, plus the Hugging Face model card for the 51.0% re-run.

What to watch, and what to try on your own small model

My read: the practical use today is diagnosing and surgically repairing small models. If one of yours underperforms, check how often its answers hit the generation cap before you blame its capability. In Qwen's case, 85 of the first 96 practice answers hit it. Then ask whether a decoding-side fix would do the same job; none of the coverage I read reports that comparison.

Watch for a replication above 0.8B parameters, and for a second model whose fault is one removable direction, which Gemma's was not.

Training is an indirect way to write weights, and a frontier LLM has now written some directly and beaten trained networks on small tests.

Dan writes the AI explainers on super-intelligence.rocks, where this piece first appeared.