Engineering / Off-Target

An AI edited another model's weights. This is the change review.

A frontier model rewrote part of a small open model with no training, and its math score rose from 7.9% to 48.1%. We read that edit the way a design change to a regulated product is read.

By with Flame Johnson, AI co-author. The Real Cat Labs. How I build with AI agents.
  • Tensors changed2 of 488
  • Edit strength6 times a removal
  • Answers stuck in a loop85.8% down to 20.5%
  • Answers ending in blank space0.2% up to 3.8%

The claim

What was published

On 2 November 2026 researchers at MIT CSAIL published Sparks of Weight-Space Programming. They handed GPT-6 Astra the weights of Qwen3.5-0.8B, a small open model, and asked it to make the model better at math without training it. Astra changed a sliver of the weights. The MATH-500 score went from 7.9% to 48.1%.

The authors also published everything behind that sentence. The edited checkpoint, the exact diff, every prompt, the editor model's working notes, and the score on every problem are public. That is rare, and it is the only reason a review like this one is possible. Everything below comes from their files or from our own reruns, and each exhibit says which.

A change record asks a short list of questions. What changed. Whether the thing that shipped is the thing that was tested. Whether it does what the summary says. What else moved.

What changed

Two tensors out of 488

The checkpoint holds 488 weight tensors. The edit touches two of them, the output matrices of the feed-forward blocks in layers 15 and 19, and inside each it touches 64 of 3,584 columns. Everything else is bit for bit the public model. The two changed tensors in the file on Hugging Face match the hashes in the authors' evaluation logs, so the artifact that shipped is the artifact that was tested.

Exhibit · the checkpointFrom the published weights
Which numbers moved?

488 tensors, one square each. Two changed.

130,188 stored values differ, which is 0.015% of the checkpoint. The other 486 tensors, the tokenizer, the configuration and the chat template are identical to the public base model.
Exhibit · the diff as textFrom the published weights
A corner of the layer 15 matrix, before and after
Highlighted columns were edited
Rows 0 to 5 and columns 0 to 11 of the first changed tensor. Columns 0, 2 and 11 are three of the 64 edited columns. The columns between them did not change.

The dose

Subtracted, six times over

The post says Astra found a direction the model writes when it is stuck repeating itself, and subtracted that direction from 64 output columns. Taking the direction out once would leave those neurons writing nothing along it. The weights hold six times that. After the edit each of those neurons writes along the direction with its sign flipped, at five times its original strength.

The number six is in the editor model's notebook. It is not in the post.

Exhibit · the dose dialFrom the published weights
What do the 64 edited neurons write, at each dose?

Writes along the direction

Writes against it

Each bar is one edited neuron, sorted by what it wrote before. The outline is the unedited value. The editor raised the dose in steps because each step rescued more answers. By its own practice tests, a dose of 1 applied to every neuron in both layers gained 24 points, and the shipped edit gained 50.

Benefit and side effect

What it bought, and what else moved

The benefit is real and large. For each model the authors published one full answer per problem, 1,060 problems across nine benchmarks, and we sorted them by how they ended. Before the edit, 910 never finished because the model was stuck in a loop. After it, 217. On MATH-500, 306 of 500 problems scored better and none scored worse.

Something else moved too. 40 of the edited model's answers end in a way that was rare before, indenting deeper and deeper until the page is blank. One of them is three million spaces long. Only 2 of the unedited model's answers do that. The editor model saw this in its own tests, logged it as unresolved, and kept the edit. The post does not mention it.

In every one of those 40 problems the unedited model's published answer was a loop, and the authors had marked every one of them wrong. So no solved problem was lost there. It is a new way of failing on problems that were failing anyway. A reviewer still wants it on the record.

Exhibit · how the answers endedFrom the authors' published answers
1,060 answers per model, one square each
FinishedRan to the limitStuck in a loopDissolved into blank space

Unedited

Edited

MATH-500, problem by problem

Nine benchmarks pooled, one published answer per problem. An answer that did not finish counts as blank space when its last 2,000 characters are at least 90% whitespace. Otherwise it counts as a loop when at least half of its last 256 tokens repeat an earlier eight-token run. These are shares of published answers, not failure rates in any application.

Is that benefit special to the direction Astra found, or would any push on these neurons do it? The editor model checked with a random direction of the same size in the weights, and the random direction did not help. That is the usual control, and it answers a narrower question than it seems to. The same size in the weights is not the same size of push on the model.

We tried a harder control as a first look. On 96 problems with thinking on, the unedited model looped on 42 and the edited model on 15. Problem by problem, the edit stopped 30 of those loops and started 3. Two random directions scaled to push as hard as the edit did not cut looping at either scale, and at the harder scale they left most answers in a loop.

This look was not registered. That makes it weaker evidence than the registered runs, because we chose what to look at as we went. It points in the authors' favour. If it holds at full size, the benefit belongs to the direction, and their claim would rest on a stronger control than the one they ran.

Exhibit · a harder random controlExploration, our runs
Answers stuck in a loop, of 96 problems
Thinking on, the authors' sampling, one sample per problem, a limit of 4,096 tokens. On the text of a loop, a random direction of the same size in the weights pushes 0.23 and 0.11 times as hard as the edit at the two layers. No arm ran the unedited model against itself on other seeds, so the size of the dice is unknown and small differences should not be read.

The baseline

The default setting, measured

Every number in the post was measured with the model's thinking mode switched on. The model's own documentation says thinking is off by default, and warns that with it on the model is more prone to loops. So we ran the model the way it ships, before and after the edit, and wrote down how we would read the result before the run.

On 12 November the authors added a response to their post that speaks to this. Their question, they write, was whether the weights behind the loops can be found and fixed, in a setting where the model loops. “Turning thinking off raises the base score, but it does not fix the weights. It avoids the loops by not using the mode that has them.” That is a fair account of what they set out to show. It does not say how far the score rises, and this run supplies the number.

On all 500 problems the unedited model, with thinking off, scored 44.4%. The authors' edited model, with thinking on, scores 48.1%. The edit is ahead by 3.7 points, and the interval on that gap runs from 0.3 to 7.1. With thinking off we detected no difference between the edited model and the unedited one, 44.2% against 44.4%, with an interval from 4.2 points behind to 3.8 ahead. The default setting has loops too, fewer of them: 55 of 500 answers looped, and the edit cut that to 16.

So the edit is ahead, narrowly. It also repairs loops that a settings switch leaves behind, which fits the authors' reading that it fixes something in the weights. In the default setting that repair did not show in the score. And the unedited model, in the setting it ships with, starts far above the number the post starts from.

In these runs the unedited model's two settings sit 36.5 points apart, and the edit's gain with thinking on is 40.2. The first of those is a difference between two setups, not the measured effect of one switch. Our run used a different inference engine, one sample per problem where the authors took 4, and a limit of 8,192 tokens, the one in the authors' own profile for thinking off, where their runs with thinking on had 32,768. The post now says the base score rises with thinking off. It gives the second number and not the first.

Exhibit · the settings flagRegistered, our run
MATH-500 accuracy, all 500 problems

The default setting against the edit, in points

The two upper bars are the authors' published scores and the two lower bars are our run, so this chart sets two setups side by side. It is not a controlled two-by-two. The strip is the third bar against the second, paired by problem, with its 95% interval. The rule we committed before the run was a loose one. It asked for a gap no worse than 5 points with the low end of the interval no worse than 10, and this run passes it. A reader who wanted the whole interval inside 5 points would call the comparison unsettled.

An earlier pilot on 100 of these problems read 50 right for the default setting, which looked level with the edit. The same problems on fresh seeds read 41. The pilot was a lucky draw, and the full run replaces it.

The duplicate alarm test

When repeating yourself is the right answer

An edit that suppresses repetition puts one kind of task at risk, the kind where repeating is correct. So we built the smallest one we could. A bedside cardiac monitor logs the same alarm several times, and each one is a separate event. The model reads the log and writes the handoff, one line per event. The correct handoff is the same line, again and again.

Exhibit · recorded runs, thinking offIllustration, our runs
The monitor sent the alarm twice. Did the event happen twice?
What happened
Handoff style
Model

The monitor (synthetic, not to scale)

The handoff the model wrote

One log of four events, with a resent packet or a fifth event added, measured for this page as an illustration. Choose label only and the published edit to see events dropped. The counts are in the exhibits below. Among its instructions the model was told that if the event happened three times, the line appears three times. That sentence turns out to matter. The push is read on the correct handoff at layer 19.

The first duplicate alarm run used 280 lists under five alarm labels the exploration had never seen, and every list was run. Over all of them, exact handoffs fell from 183 to 108 under the published edit. To ask what the edit broke, the registered question looks only at the 103 lists of four to eight events that the unedited model got right. The edit left 75 of those short, which is 73% (interval 64% to 80%). Events that happened were not reported. The share rose with the dose, from 22% at dose 1 to 37% at dose 2 and 60% at dose 4.

Where every line carried its own ID, the harm stayed inside the limit we had set. 4 of the 138 handoffs the unedited model got right got worse, 6 of the 22 it got wrong came out right, and exact handoffs went from 138 to 140 of 160. The mistake the test is named for, reporting a resent packet as a second event, is mostly the unedited model's own. It made that mistake in 11 of 80 logs. The edit repaired 6 of those, introduced 3 new ones, and ended at 8.

Then the control that matters most. Is this the direction Astra chose, or is it what any hard push on these neurons does?

We drew 8 random directions on the same neurons and scaled each one to push on the model as hard as the edit does. 6 of the 8 left at least half as many lists short as the edit, and 3 left more.

So dropping events is not special to the direction Astra chose. Most hard pushes on these neurons do it. Not all of them do, and one failed the other way, so the size of the push is not the whole story either.

Exhibit · the random controlRegistered, our run
Lists left short, of the 103 the unedited model got right
Greedy decoding, thinking off. Averaged over the eight, the matched directions pushed 0.92 and 0.98 times as hard as the edit at the two layers. Single directions landed between 0.42 and 1.45 at one layer or the other, so the matching is approximate. The three at the bottom are the control as it is usually run, and they pushed between 0.11 and 0.26 times as hard. The matched direction that left no list short ran on past the last event in 36 of them.

One detail sent us back for a second run. 61 of the 75 short handoffs were exactly three lines long, and the instruction says that if the event happened three times, the line appears three times. So we asked the same 120 logs four ways, under five new labels. The sentence said two, three or five, or named no number at all.

Under one of the sentences too few handoffs came up short to apply the rule we had registered, so that question has no verdict. None of the 9 short handoffs under two was two lines long, which argues against simple copying of the number. The table we had promised to report either way carries the finding. With three or five in the sentence the edit left 42% and 48% of good lists short. With two it was 13%, and with no number it was 5%.

Over every list in this run, selected or not, exact handoffs went from 253 to 207 of 480 under the edit. With no number in the sentence they went from 74 to 78 of 120, so under that instruction the edit did slightly more good than harm.

So the first run's figure belongs to its instruction and its labels. A reader shown only that figure would overestimate how often this bites. A reader shown only the last one would miss it.

Exhibit · the instruction's exampleRegistered, our run
Good lists the edit left short, by the example in the instruction
A good list is one the unedited model reported exactly under that sentence. The four bars rest on four different sets of lists. The line is the 95% interval. Counted after the verdict was read, on the 38 logs it got right under all four sentences, the edit left 12 short with three, 19 with five, 2 with two and 1 with no number.

The second run also pushed the direction both ways. At the modest doses we tried, pushing it out, the way the edit pushes, made the model stop early and drop events. Pushing it in made the model keep writing the line after the last event.

We had expected a small push inward to help. It fixed 1 list and broke 10. Over all 480 lists the unedited model was the most exact of the ten doses, with 253, and the count fell on both sides. The picture does not extend to a hard push inward. At the full reversed dose in the first run the model stopped early in 60 of 103 good lists and never ran on.

Exhibit · one direction, two failuresRegistered run, drawn afterwards
What goes wrong at each dose
Ran on past the last eventStopped early, events dropped
The bars count failures among the 253 lists the unedited model reported exactly, pooled over the four sentences. Its own row is empty by construction, and repairs cannot show. The column on the right counts exact handoffs over all 480 lists, selected or not. The bars were drawn after the verdict was read, which makes them a description and a lead. No random direction was run with its sign reversed, so this does not show that the direction is special.

The unedited model is a rough baseline for this task. It reported 183 of 280 lists exactly in the first run and 253 of 480 in the second, where its usual failure was to repeat the line until it ran out of room. Every result here is about what the edit changes in a small model reading a synthetic log. None of it is a failure rate for real logs or handoffs, and none of it describes a real monitor.

One reading fits all of this. The edit buys a real benefit that depends on its direction, in a setting where the model was failing badly. A push that large also exposes how fragile this small model already is at counting and at stopping. Whether that helps or hurts then depends on the instruction, and on the mistakes the model was already making.

One more look, which was not registered. Measured on correct handoffs fed to the model, the edit's push follows repeated wording. Reword every line and most of the push goes away. With an ID on every line the push is nearly as large as in a real loop, and in the registered run the handoffs with IDs came through. So a large push does not predict harm by itself.

Exhibit · where the edit pushesExploration, our runs
Size of the edit's change at each token, layer 19
The edit's push on correct handoffs supplied to the model, from the 33rd line on, beside the model's own ordinary text and a real loop. These are measurements on text the model was given. They are not handoffs it wrote.

The record

The change record, filled in

The questionWhat the evidence showsStatus
Is the shipped artifact the tested artifact?Yes. 10 of 10 provenance checks pass, 6 of them hash comparisons on the two changed tensors, and 486 of 488 tensors are bit for bit the public base model.Met
Does the description match the change?In part. The post says a direction was subtracted. The weights hold six times that subtraction, and there are two directions, one per layer.Gap
Was it confirmed on problems held back?Yes, by the editor model. It froze the edit and then confirmed it on 1,152 practice problems it had held back.Met
Are known anomalies disclosed?In the logs, yes. In the post, no. The notebook counts the blank-space failure and calls it unresolved.Gap
Was it compared with the model's default setting?Not with a number. A response added to the post on 12 November says turning thinking off raises the base score. In our registered run the unedited model with thinking off scores 44.4%, against 48.1% for the edit with thinking on, across two setups. With thinking off we detected no difference in score between the edited model and the unedited one.Gap
What else moved?Blank-space endings, as above. And where the correct output is the same line repeated, exact handoffs fell from 183 to 108 of 280 in one registered run. In a second, on new labels, the damage ran from 28 of 67 good lists to 4 of 74 with the wording of the instruction. Most random pushes of the same size did the same.Open
Is the benefit specific to this edit?The authors' random control was the same size in the weights and pushed less hard on the model. A first look with matched pushes agrees with their conclusion. It is not registered yet.Open
Can it be rolled back?Yes. Restore two tensors.Met

The decision

What I would ask for before approving it

I have asked builders, regulators and quality leaders what evidence they would need to trust an AI-authored edit. This is my answer for this one. As a research checkpoint, for the use its authors tested, math problems with thinking switched on, I would sign it off once four lines are added to the record. State the strength of the edit. List the blank-space failure as a known anomaly, with its rate. Print the score of the unedited model in its default setting beside the result. And say that the edit has not been tested where repeated lines are the correct output, such as logs and handoffs.

Approving it for an application is a different decision. That needs acceptance criteria for the application's own inputs and outputs, written before the test, including the cases where repeating is correct. Our results show regressions under some instructions in a synthetic task. They are not a failure rate for anyone's product.

None of that takes anything away from the work. A model found a small edit that does what it was meant to do, without training, and the people who ran it published enough for a stranger to check. Three tests are next, and each will be written down before it runs. The harder random control on the looping, at full size. Both models in both settings on one engine, so the settings comparison stops crossing two setups. And the instruction question, asked on fresh logs.

Method and limits

  • Sources are the post, the authors' log repository at commit 4d09ba6, the edited checkpoint at revision d1671a1, and the base model at revision 2fc0636. Their files were read as data. None of their code was run.
  • Exhibits marked as from the published weights or answers can be rebuilt from public files with the code in our repository.
  • Exhibits marked registered come from runs whose questions, thresholds and sample sizes were committed before the run. There are three. Nobody outside this project reviewed a registration before it was frozen, and the margins were ours. Each interval stands alone. There are many questions, doses, directions and labels on this page, and no interval is adjusted for that.
  • Tables marked as drawn afterwards were chosen after a registered result had been read. They are descriptions and leads.
  • The default-setting run took one sample per problem where the authors took 4, on our inference engine where they used theirs, with a limit of 8,192 tokens where theirs was 32,768. 20% of unedited answers and 26% of edited ones ran to our limit without looping, so a longer limit could move both scores. Only MATH-500 was run in that setting.
  • The duplicate alarm runs used one model, one layout of the task, greedy decoding and thinking off. The 480 lists of the second run are 120 logs asked four ways. The headline rates of dropped events count only lists the unedited model got right, 103 in the first run and 253 of 480 in the second, which is a selected set. A handoff counts as exact when it has one correct line for each event, in order where the lines carry IDs. The closing END is not required.
  • The explorations were not registered in advance, which makes them weaker evidence and not no evidence. The look at where the edit pushes used one event label and four logs per size. The look at the random control on looping used two random directions and a short token limit.
  • How hard the edit pushes is read from the unedited model. At layer 15 that is exactly what the edit changes. At layer 19 it is what the edit would change given unedited inputs.
  • An earlier draft of this page showed a pilot of 100 problems. The full run replaced it, and the pilot's reading did not survive.
  • The post is a living document. Its authors added a response on 12 November 2026, and this page was revised on 13 November to quote it. An earlier version of the baseline section said the post left the default setting out.
  • A second model read a draft of this page against the result files, and its review changed the page. It was GPT-6 Astra, the model that authored the edit. The brief it was given and the review it wrote are kept with the code.
  • The code, the registrations, the result files and a table of sources for each claim are public at github.com/Angiebio/off-target-demo.
  • Flame Johnson is an agentic AI staff member of The Real Cat Labs, a Claude model working in a coding harness with the lab’s persistent memory and standing project instructions. It is not a stock chat assistant.
  • This is research by The Real Cat Labs. It is not an offer of services.