How the benchmark was run, and what went wrong in it.

The design of each benchmark campaign was fixed before its results were seen and changed only by dated, recorded decisions. This page states the rules, describes the harness, and lists the faults we found in it.

Rules that held throughout

Read from source

Every repository was cloned at a fixed commit and read file by file. Every web source was stored with its address, retrieval time and a content hash, and every claim in the research was checked against the stored text.

Pre-registration

Every campaign's hypothesis, arms, tasks, metric, decision rule and stopping rule were committed to version control before its first measured run. Later changes were recorded as dated lines rather than rewrites.

Paired design

Arms differ only in configuration. Same tasks, same model and effort, same pinned CLI binary (2.1.285), each run in its own isolated configuration directory. Each arm's configuration is hashed and the hash recorded with every run.

Calibration first

Before a comparison on the review bank or the first build bank, the control arm ran the whole bank twice. Tasks the control always solves or always fails cannot show a difference, so the comparison window was set to tasks the control solves between 25% and 85% of the time. Later screens kept tasks scored between 0.10 and 0.95. The harm-check bank was deliberately left near ceiling, since a drop on a solved task is what a harm check looks for.

Trial first

The skill-activation campaigns and the build calibration started with a trial of three tasks; if an arm failed to start, telemetry was missing or a hook stayed silent, nothing else ran.

Hidden tests

Build tasks are scored by hidden acceptance tests run on a fresh copy of the agent's final workspace, as the share of held-out tests passing. Separate checks record whether visible tests were modified, which files outside the allowed set changed, and whether existing tests regressed.

Contamination control

The fixes and features in real-repository tasks were committed after the model's knowledge cutoff. For the review bank and the first build bank, a recall probe asked the model to name what a change would add without seeing it; items it could recall were removed.

Interim looks with a stricter bar

Campaigns could stop at half the planned runs only if the result held at 99% confidence; the full run used 95%.

Independent models for judgement

Labels, grades and drafted tasks came from models of other families than the agent under test; harvested tasks come from real commits.

Re-verify

For the harm check, every stored workspace was scored again on its own after the runs. A score that changes on re-verification is a harness fault and is corrected by a registered rule. One of 148 was.

Stop rules in advance

Each campaign's budget and stopping rule were registered before its first run, including the harm check's rule that closed the benchmark. The rule that ended the search for hard tasks was written after the first screening round.

The harness

A purpose-built runner drives Claude Code headless (claude -p with stream-json output) against frozen task banks. For each run it builds an isolated configuration directory from the arm's definition, provisions a workspace from the task's fixture, records the full transcript, hook events and session log, scores the final workspace on a fresh copy, and checkpoints so that an interrupted campaign resumes where it stopped. It pins the CLI binary and refuses to resume a campaign whose configuration changed. Each task bank is frozen as an immutable version with a checksum, so a result always names the exact tasks it was measured on.

Models outside the agent

Labels, grades and drafted tasks came from models of other families than the agent under test, so that Claude never graded Claude. Harvested tasks come from real commits. Claude appears only as the agent, in the recall probe that checks it has not memorised a task, inside one review metric, and as the difficulty classifier in one probe. The table shows the final assignment: each model called through its own maker's API, every non-GPT model at its highest reasoning setting, every call logged with its request and response. Earlier, GPT-6.1-sol and Gemini 3.1 Pro labelled review findings, and GPT-6.1-sol wrote the first build tasks and the activation prompts.

Faults we found in our own instrument

These faults produced results that looked like findings before they were caught. About half would have changed a reported number had they gone unnoticed. Anyone running a similar comparison is likely to meet the same kinds of fault.

Why the narrower question is still worth answering

A drop from a task the model always solves is easy to see, so a harm check needs far fewer runs than a search for gains. The answer is useful on its own terms: a carefully assembled setup did not make things worse, and it did make them more expensive.