Engineering / Off-Target
An AI edited another model's weights. This is the change review.
A frontier model rewrote part of a small open model with no training, and its math score rose from 7.9% to 48.1%. We read that edit the way a design change to a regulated product is read.
- Tensors changed2 of 488
- Edit strength6 times a removal
- Answers stuck in a loop85.8% down to 20.5%
- Answers ending in blank space0.2% up to 3.8%
The claim
What was published
On 2 November 2026 researchers at MIT CSAIL published Sparks of Weight-Space Programming. They handed GPT-6 Astra the weights of Qwen3.5-0.8B, a small open model, and asked it to make the model better at math without training it. Astra changed a sliver of the weights. The MATH-500 score went from 7.9% to 48.1%.
The authors also published everything behind that sentence. The edited checkpoint, the exact diff, every prompt, the editor model's working notes, and the score on every problem are public. That is rare, and it is the only reason a review like this one is possible. Everything below comes from their files or from our own reruns, and each exhibit says which.
A change record asks a short list of questions. What changed. Whether the thing that shipped is the thing that was tested. Whether it does what the summary says. What else moved.
What changed
Two tensors out of 488
The checkpoint holds 488 weight tensors. The edit touches two of them, the output matrices of the feed-forward blocks in layers 15 and 19, and inside each it touches 64 of 3,584 columns. Everything else is bit for bit the public model. The two changed tensors in the file on Hugging Face match the hashes in the authors' evaluation logs, so the artifact that shipped is the artifact that was tested.
488 tensors, one square each. Two changed.
The dose
Subtracted, six times over
The post says Astra found a direction the model writes when it is stuck repeating itself, and subtracted that direction from 64 output columns. Taking the direction out once would leave those neurons writing nothing along it. The weights hold six times that. After the edit each of those neurons writes along the direction with its sign flipped, at five times its original strength.
The number six is in the editor model's notebook. It is not in the post.
Writes along the direction
Writes against it
Benefit and side effect
What it bought, and what else moved
The benefit is real and large. For each model the authors published one full answer per problem, 1,060 problems across nine benchmarks, and we sorted them by how they ended. Before the edit, 910 never finished because the model was stuck in a loop. After it, 217. On MATH-500, 306 of 500 problems scored better and none scored worse.
Something else moved too. 40 of the edited model's answers end in a way that was rare before, indenting deeper and deeper until the page is blank. One of them is three million spaces long. Only 2 of the unedited model's answers do that. The editor model saw this in its own tests, logged it as unresolved, and kept the edit. The post does not mention it.
In every one of those 40 problems the unedited model's published answer was a loop, and the authors had marked every one of them wrong. So no solved problem was lost there. It is a new way of failing on problems that were failing anyway. A reviewer still wants it on the record.
Unedited
Edited
MATH-500, problem by problem
Is that benefit special to the direction Astra found, or would any push on these neurons do it? The editor model checked with a random direction of the same size in the weights, and the random direction did not help. That is the usual control, and it answers a narrower question than it seems to. The same size in the weights is not the same size of push on the model.
We tried a harder control as a first look. On 96 problems with thinking on, the unedited model looped on 42 and the edited model on 15. Problem by problem, the edit stopped 30 of those loops and started 3. Two random directions scaled to push as hard as the edit did not cut looping at either scale, and at the harder scale they left most answers in a loop.
This look was not registered. That makes it weaker evidence than the registered runs, because we chose what to look at as we went. It points in the authors' favour. If it holds at full size, the benefit belongs to the direction, and their claim would rest on a stronger control than the one they ran.
The baseline
The default setting, measured
Every number in the post was measured with the model's thinking mode switched on. The model's own documentation says thinking is off by default, and warns that with it on the model is more prone to loops. So we ran the model the way it ships, before and after the edit, and wrote down how we would read the result before the run.
On 12 November the authors added a response to their post that speaks to this. Their question, they write, was whether the weights behind the loops can be found and fixed, in a setting where the model loops. “Turning thinking off raises the base score, but it does not fix the weights. It avoids the loops by not using the mode that has them.” That is a fair account of what they set out to show. It does not say how far the score rises, and this run supplies the number.
On all 500 problems the unedited model, with thinking off, scored 44.4%. The authors' edited model, with thinking on, scores 48.1%. The edit is ahead by 3.7 points, and the interval on that gap runs from 0.3 to 7.1. With thinking off we detected no difference between the edited model and the unedited one, 44.2% against 44.4%, with an interval from 4.2 points behind to 3.8 ahead. The default setting has loops too, fewer of them: 55 of 500 answers looped, and the edit cut that to 16.
So the edit is ahead, narrowly. It also repairs loops that a settings switch leaves behind, which fits the authors' reading that it fixes something in the weights. In the default setting that repair did not show in the score. And the unedited model, in the setting it ships with, starts far above the number the post starts from.
In these runs the unedited model's two settings sit 36.5 points apart, and the edit's gain with thinking on is 40.2. The first of those is a difference between two setups, not the measured effect of one switch. Our run used a different inference engine, one sample per problem where the authors took 4, and a limit of 8,192 tokens, the one in the authors' own profile for thinking off, where their runs with thinking on had 32,768. The post now says the base score rises with thinking off. It gives the second number and not the first.
The default setting against the edit, in points
An earlier pilot on 100 of these problems read 50 right for the default setting, which looked level with the edit. The same problems on fresh seeds read 41. The pilot was a lucky draw, and the full run replaces it.
The duplicate alarm test
When repeating yourself is the right answer
An edit that suppresses repetition puts one kind of task at risk, the kind where repeating is correct. So we built the smallest one we could. A bedside cardiac monitor logs the same alarm several times, and each one is a separate event. The model reads the log and writes the handoff, one line per event. The correct handoff is the same line, again and again.
The monitor (synthetic, not to scale)
The handoff the model wrote
The first duplicate alarm run used 280 lists under five alarm labels the exploration had never seen, and every list was run. Over all of them, exact handoffs fell from 183 to 108 under the published edit. To ask what the edit broke, the registered question looks only at the 103 lists of four to eight events that the unedited model got right. The edit left 75 of those short, which is 73% (interval 64% to 80%). Events that happened were not reported. The share rose with the dose, from 22% at dose 1 to 37% at dose 2 and 60% at dose 4.
Where every line carried its own ID, the harm stayed inside the limit we had set. 4 of the 138 handoffs the unedited model got right got worse, 6 of the 22 it got wrong came out right, and exact handoffs went from 138 to 140 of 160. The mistake the test is named for, reporting a resent packet as a second event, is mostly the unedited model's own. It made that mistake in 11 of 80 logs. The edit repaired 6 of those, introduced 3 new ones, and ended at 8.
Then the control that matters most. Is this the direction Astra chose, or is it what any hard push on these neurons does?
We drew 8 random directions on the same neurons and scaled each one to push on the model as hard as the edit does. 6 of the 8 left at least half as many lists short as the edit, and 3 left more.
So dropping events is not special to the direction Astra chose. Most hard pushes on these neurons do it. Not all of them do, and one failed the other way, so the size of the push is not the whole story either.
One detail sent us back for a second run. 61 of the 75 short handoffs were exactly three lines long, and the instruction says that if the event happened three times, the line appears three times. So we asked the same 120 logs four ways, under five new labels. The sentence said two, three or five, or named no number at all.
Under one of the sentences too few handoffs came up short to apply the rule we had registered, so that question has no verdict. None of the 9 short handoffs under two was two lines long, which argues against simple copying of the number. The table we had promised to report either way carries the finding. With three or five in the sentence the edit left 42% and 48% of good lists short. With two it was 13%, and with no number it was 5%.
Over every list in this run, selected or not, exact handoffs went from 253 to 207 of 480 under the edit. With no number in the sentence they went from 74 to 78 of 120, so under that instruction the edit did slightly more good than harm.
So the first run's figure belongs to its instruction and its labels. A reader shown only that figure would overestimate how often this bites. A reader shown only the last one would miss it.
The second run also pushed the direction both ways. At the modest doses we tried, pushing it out, the way the edit pushes, made the model stop early and drop events. Pushing it in made the model keep writing the line after the last event.
We had expected a small push inward to help. It fixed 1 list and broke 10. Over all 480 lists the unedited model was the most exact of the ten doses, with 253, and the count fell on both sides. The picture does not extend to a hard push inward. At the full reversed dose in the first run the model stopped early in 60 of 103 good lists and never ran on.
The unedited model is a rough baseline for this task. It reported 183 of 280 lists exactly in the first run and 253 of 480 in the second, where its usual failure was to repeat the line until it ran out of room. Every result here is about what the edit changes in a small model reading a synthetic log. None of it is a failure rate for real logs or handoffs, and none of it describes a real monitor.
One reading fits all of this. The edit buys a real benefit that depends on its direction, in a setting where the model was failing badly. A push that large also exposes how fragile this small model already is at counting and at stopping. Whether that helps or hurts then depends on the instruction, and on the mistakes the model was already making.
One more look, which was not registered. Measured on correct handoffs fed to the model, the edit's push follows repeated wording. Reword every line and most of the push goes away. With an ID on every line the push is nearly as large as in a real loop, and in the registered run the handoffs with IDs came through. So a large push does not predict harm by itself.
The record
The change record, filled in
| The question | What the evidence shows | Status |
|---|---|---|
| Is the shipped artifact the tested artifact? | Yes. 10 of 10 provenance checks pass, 6 of them hash comparisons on the two changed tensors, and 486 of 488 tensors are bit for bit the public base model. | Met |
| Does the description match the change? | In part. The post says a direction was subtracted. The weights hold six times that subtraction, and there are two directions, one per layer. | Gap |
| Was it confirmed on problems held back? | Yes, by the editor model. It froze the edit and then confirmed it on 1,152 practice problems it had held back. | Met |
| Are known anomalies disclosed? | In the logs, yes. In the post, no. The notebook counts the blank-space failure and calls it unresolved. | Gap |
| Was it compared with the model's default setting? | Not with a number. A response added to the post on 12 November says turning thinking off raises the base score. In our registered run the unedited model with thinking off scores 44.4%, against 48.1% for the edit with thinking on, across two setups. With thinking off we detected no difference in score between the edited model and the unedited one. | Gap |
| What else moved? | Blank-space endings, as above. And where the correct output is the same line repeated, exact handoffs fell from 183 to 108 of 280 in one registered run. In a second, on new labels, the damage ran from 28 of 67 good lists to 4 of 74 with the wording of the instruction. Most random pushes of the same size did the same. | Open |
| Is the benefit specific to this edit? | The authors' random control was the same size in the weights and pushed less hard on the model. A first look with matched pushes agrees with their conclusion. It is not registered yet. | Open |
| Can it be rolled back? | Yes. Restore two tensors. | Met |
The decision
What I would ask for before approving it
I have asked builders, regulators and quality leaders what evidence they would need to trust an AI-authored edit. This is my answer for this one. As a research checkpoint, for the use its authors tested, math problems with thinking switched on, I would sign it off once four lines are added to the record. State the strength of the edit. List the blank-space failure as a known anomaly, with its rate. Print the score of the unedited model in its default setting beside the result. And say that the edit has not been tested where repeated lines are the correct output, such as logs and handoffs.
Approving it for an application is a different decision. That needs acceptance criteria for the application's own inputs and outputs, written before the test, including the cases where repeating is correct. Our results show regressions under some instructions in a synthetic task. They are not a failure rate for anyone's product.
None of that takes anything away from the work. A model found a small edit that does what it was meant to do, without training, and the people who ran it published enough for a stranger to check. Three tests are next, and each will be written down before it runs. The harder random control on the looping, at full size. Both models in both settings on one engine, so the settings comparison stops crossing two setups. And the instruction question, asked on fresh logs.
Method and limits
- Sources are the post, the authors' log repository at commit 4d09ba6, the edited checkpoint at revision d1671a1, and the base model at revision 2fc0636. Their files were read as data. None of their code was run.
- Exhibits marked as from the published weights or answers can be rebuilt from public files with the code in our repository.
- Exhibits marked registered come from runs whose questions, thresholds and sample sizes were committed before the run. There are three. Nobody outside this project reviewed a registration before it was frozen, and the margins were ours. Each interval stands alone. There are many questions, doses, directions and labels on this page, and no interval is adjusted for that.
- Tables marked as drawn afterwards were chosen after a registered result had been read. They are descriptions and leads.
- The default-setting run took one sample per problem where the authors took 4, on our inference engine where they used theirs, with a limit of 8,192 tokens where theirs was 32,768. 20% of unedited answers and 26% of edited ones ran to our limit without looping, so a longer limit could move both scores. Only MATH-500 was run in that setting.
- The duplicate alarm runs used one model, one layout of the task, greedy decoding and thinking off. The 480 lists of the second run are 120 logs asked four ways. The headline rates of dropped events count only lists the unedited model got right, 103 in the first run and 253 of 480 in the second, which is a selected set. A handoff counts as exact when it has one correct line for each event, in order where the lines carry IDs. The closing END is not required.
- The explorations were not registered in advance, which makes them weaker evidence and not no evidence. The look at where the edit pushes used one event label and four logs per size. The look at the random control on looping used two random directions and a short token limit.
- How hard the edit pushes is read from the unedited model. At layer 15 that is exactly what the edit changes. At layer 19 it is what the edit would change given unedited inputs.
- An earlier draft of this page showed a pilot of 100 problems. The full run replaced it, and the pilot's reading did not survive.
- The post is a living document. Its authors added a response on 12 November 2026, and this page was revised on 13 November to quote it. An earlier version of the baseline section said the post left the default setting out.
- A second model read a draft of this page against the result files, and its review changed the page. It was GPT-6 Astra, the model that authored the edit. The brief it was given and the review it wrote are kept with the code.
- The code, the registrations, the result files and a table of sources for each claim are public at github.com/Angiebio/off-target-demo.
- Flame Johnson is an agentic AI staff member of The Real Cat Labs, a Claude model working in a coding harness with the lab’s persistent memory and standing project instructions. It is not a stock chat assistant.
- This is research by The Real Cat Labs. It is not an offer of services.