107 configurations specified.
Every candidate change found in the research was written up as an arm before anything ran: a hypothesis, a control, a treatment, a metric, a predicted effect and a falsifier. They were ranked by what a resolved result would settle, the chance of resolving it, and its cost. The repository measurements (R0 to R2 on the runs page) install whole repositories rather than arms from this list; their results are on the repositories page.
- Measured: 2: the forced skill-evaluation hook on its own, in two versions. The full setup, which bundles many of the others, was measured as a whole and is not in this list.
- Specified, not run: 73. Most needed a task bank on which the control fails often enough to show a gain, and that bank could not be built.
- Dropped: 27, with the review campaign, when the review bank proved too easy and the question moved to building software.
- Set aside or removed: 5.
Stages: AMB ambient context; RIG rigging the harness; CON contract; PLN plan; IMP implement; REV review; RTN routines; BEN benchmark instruments.
| ID | Arm | Stage | Task bank | State |
|---|---|---|---|---|
| 7 | Forced skill evaluation (UserPromptSubmit hook) vs native discovery | AMB | TB-A v1 (Spence's 41 positive + 5 negative prompts, harvested; Svelte skills as the … | measured |
| 7b | The forcing hook in the build's wording, on the build's own skills (TB-A v2) | AMB | TB-A v2 (frozen 2026-10-10: 63 positives on the build's seven model-invocable skills, 14 … | measured |
| 1 | A1 − A0: what the user-scope rationale CLAUDE.md buys | AMB | TB-B | specified, not run |
| 2 | Scaffolding necessity by model tier (bitter-lessoned vs artisanal) | AMB | TB-B; phuryn's battery shape (floor tasks + planted-key tasks) for the floor items | specified, not run |
| 3a | Current obra/superpowers as a system (the premise falsifier) | AMB | TB-B (C2 also runs TB-R 100 × 1 as a secondary) | specified, not run |
| 4 | Before/after shrink exemplars as CLAUDE.md-audit calibration | AMB | not stated | specified, not run |
| 5a | Project glossary of terms of art | AMB | TB-B items with domain terms | specified, not run |
| 5b | The preference block: the four missing daily-lane instructions | AMB | TB-B (multi-part slices) | specified, not run |
| 5c | Perishable/permanent split with dates | AMB | TB-B | specified, not run |
| 6 | Block-at-submit hook (pass-file on git commit) vs the Stop-hook bound | RIG | TB-B | specified, not run |
| 8 | Bram-style worklist gate (PreToolUse: an edit must map to an approved item) | AMB | field trial (a week of our real use), not a TB | specified, not run |
| 9 | jj snapshot-per-edit hook vs git worktree + Stop hook (optional) | RIG | none defined (no bank has bad-edit recovery items) | specified, not run |
| 10 | Contract trio (what would have to be true / falsifiers / base rate) as a human-run checklist | CON | field trial: our own tasks, logged | specified, not run |
| 11 | Pre-mortem skill at Plan | PLN | TB-P | specified, not run |
| 12 | knowledge-work-plugins templates (system-design, write-spec, tech-debt) and a thin Shape-Up design skill | PLN | TB-P | specified, not run |
| 13 | idea-generation "Key Risks: what would make this wrong / next steps" template at Contract | CON | TB-S | specified, not run |
| 14 | Lightweight ADRs with a Chesterton's-fence field vs without | PLN | TB-M (items whose session 2 is tempted to remove a load-bearing constraint) | specified, not run |
| 15a | compound-engineering ce-plan (full) vs native planning | PLN | TB-P 30 × 2; TB-B 20 × 1 downstream | specified, not run |
| 15b | ce-plan with model elevation off | PLN | TB-P | specified, not run |
| 16a | Session opener: "First run the tests" | IMP | TB-B | specified, not run |
| 16b | Session opener / in-loop instruction: red/green TDD | IMP | TB-B (TS and Python slices with Stryker / mutmut) | specified, not run |
| 17a | Test authorship: isolated test-writer → implementer vs single agent + fresh-context reviewer | IMP | TB-B | specified, not run |
| 17b | tdd-guard hard gate (optional) | IMP | TB-B | specified, not run |
| 18 | Approved Scenarios for behaviour-heavy slices | IMP | TB-B behaviour-heavy slices | specified, not run |
| 19 | Conformance suite / differential oracle for port and refactor tasks | IMP | TB-B port/refactor items (none planned yet) | specified, not run |
| 20 | gstack /qa browser routine vs the official webapp-testing skill | IMP | TB-B web slices with a seeded UI regression | specified, not run |
| 22 | Elements of Style as a reference for documentation generation | IMP | documentation tasks (none in the five banks) | specified, not run |
| 25c | Fail-closed Stop-gate review vs deliberate review invocation | REV | TB-B (implementation sessions with seeded defects reachable at stop) | specified, not run |
| 25e | Oracle-style escalation on stuck tasks | REV | TB-B items where the agent stalls | specified, not run |
| 27 | Evidence ledger + review freshness (gstack Step 16) vs the Stop hook + /code-review | REV | TB-B | specified, not run |
| 28 | Compound / lessons loop (ce-compound → docs/solutions/) vs dated lessons notes | RTN | field trial: 30 days of our work | specified, not run |
| 29 | Log-mining routine + Cherny-style maintenance routines vs none | RTN | field trial | specified, not run |
| 30 | Ralph loop (as shipped, with an iteration cap) vs /goal + Stop hook for autonomous batch runs | RTN | TB-B long-horizon items (multi-slice backlogs) | specified, not run |
| 31 | Browser-artifact evidence (shot-scraper video or Playwright run) for web features | IMP | TB-B web slices with a seeded UI regression | specified, not run |
| 32 | Trajectory-shape sensor: verify-steps per edit (nilenso) | BEN | every campaign | specified, not run |
| 33 | Rules as encyclopedia articles vs rules as commands | AMB | TB-B | specified, not run |
| 37 | type-design-analyzer's 4-dimension rubric as a Haiku grader on type designs | BEN | hand-labelled type designs (none yet) | specified, not run |
| 40 | Mutation testing as a merge gate (Stryker on TS) | IMP | TB-B TS slices | specified, not run |
| 43 | Coverage-drop: hard gate vs a rationale line | IMP | TB-B items with a spec-over-tests temptation | specified, not run |
| 44 | Approved-fixtures review cost | IMP | field trial: our own reviews | specified, not run |
| 45 | Binding / mock-leak sweep | IMP | Python slices | specified, not run |
| 46 | Maintenance-proxy gate (Shore) | RTN | field: our repos | specified, not run |
| 47 | ADR Y-statement index (auto-drafted at Plan) vs auto-memory only | PLN | TB-M | specified, not run |
| 48 | Index-then-retrieve (3–5 Y-statements) vs the full ADR corpus always on | PLN | TB-M items with ≥ 10 prior ADRs | specified, not run |
| 49 | GCC-style versioned scratchpad vs the ADR index | PLN | TB-M | specified, not run |
| 50 | Visible-vs-held-out test-pass gap as a reward-hacking sensor | BEN | TB-B | specified, not run |
| 52 | Delegation: plan big, execute small (coordinator + cheap workers) vs rigor-matched solo | IMP | TB-B | specified, not run |
| 55 | Lint-feedback phrasing: coaching guide vs bare linter message vs "improve this code" | RIG | LiinaSuoniemi/prompt-vs-metric-eval cases (18 functions; clone, harvested) | specified, not run |
| 56 | Haiku self-preference rerun | BEN | 24 code-shaped prompts (the card's spec) | specified, not run |
| 57b | HumanLayer commands after trimming | PLN | not stated | specified, not run |
| 58 | anti-slop lint preset on vs off (TypeScript lane) | RIG | TB-B TypeScript slices | specified, not run |
| 59 | Phrasing: positive recipe vs prohibition vs no guidance | AMB | the spec's composition fixture (superpowers repo) | specified, not run |
| 60 | Rationalization-table ablation (TDD skill) | AMB | TB-B, or superpowers-evals' own tasks if obtained | specified, not run |
| 61 | Round-trip-loss check (syntaqlite) | BEN | port/refactor items | specified, not run |
| 62 | episodic-memory's "50–100× context saving" | RTN | our session archive | specified, not run |
| 63 | Verification hook set: PostToolUse lint/typecheck + Stop-hook test gate | RIG | TB-B | specified, not run |
| 64 | Hooks + Playwright browser check on web slices | RIG | TB-B web subset | specified, not run |
| 65 | A-sham for superpowers (load vs content) | AMB | TB-B | specified, not run |
| 66 | Verification-nag lines in CLAUDE.md | AMB | TB-B | specified, not run |
| 73 | Native planning + an Explore research subagent | PLN | TB-P | specified, not run |
| 74 | beads vs the ADR index | PLN | TB-M | specified, not run |
| 75 | Bundle confirmation: A1′ + every adopted helper | BEN | TB-B 40 × 2; TB-R 60 × 2 | specified, not run |
| 76 | Cadence: continuous build then evaluator vs evaluator per slice | IMP | TB-B multi-slice items | specified, not run |
| 77 | Held-out scenario evals with satisfaction scores alongside boolean tests | REV | TB-B | specified, not run |
| 78 | Vertical-slice instruction | IMP | TB-B multi-layer slices | specified, not run |
| 79 | Modifier handles (YAGNI, subtractive, make-the-change-easy, WET) | IMP | TB-B | specified, not run |
| 80 | /goal grader: Haiku rubric vs Opus judgment | IMP | TB-B items with an unmet acceptance criterion at first stop | specified, not run |
| 81 | Hook placement: full-suite PostToolUse gate vs Stop-hook only | RIG | TB-B | specified, not run |
| 82 | Circuit-breaker calibration | RIG | TB-B long-horizon items | specified, not run |
| 83 | JSONL-vs-markdown overwrite heuristic | AMB | TB-M two-session items with a state file | specified, not run |
| 84 | Stock vs --bare (the bundle question) | BEN | standard set v1 (12 tasks), run on the Opus 5.5 lane via --task-set-from claude-sonnet-5 | specified, not run |
| 89 | Opus 5.5 on the verbose prompt (CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT=0) | AMB | TB-B | specified, not run |
| 91 | Daily effort: Opus 5.5 at high (this machine's setting) vs medium (D7) | IMP | TB-B | specified, not run |
| 92 | Subagent routing: CLAUDE_CODE_SUBAGENT_MODEL=sonnet vs unset | AMB | not stated (TB-B is where delegation happens) | specified, not run |
| 93 | Claude Mods prompt.section: trimming the verbose lane | RIG | not stated | specified, not run |
| 23 | The build's combined reviewer vs a free-form "review this PR" | REV | TB-R | dropped |
| 24a | Reviewer + question-bank items (EL Example 1; B6 §7c) | REV | TB-R | dropped |
| 24b | Reviewer + inversion / via-negativa stance | REV | TB-R silent-fallback items | dropped |
| 24c | Reviewer + competitive nudge ("whoever finds more real issues") | REV | TB-R | dropped |
| 25a | Thin codex review seam (shimo4228) vs same-family fresh-context review | REV | TB-R | dropped |
| 25b | codex-plugin-cc /codex:adversarial-review vs the shimo4228 seam | REV | TB-R | dropped |
| 25d | pal-mcp consensus + clink vs the seam | REV | TB-R | dropped |
| 26 | Cross-provider peer agreement vs a single cross-model call | REV | TB-R | dropped |
| 34 | /code-review --fix vs /simplify | REV | TB-R (incl. clean diffs) | dropped |
| 35 | Single generalist reviewer vs the five-evidence panel at the ≥ 80 gate | REV | TB-R | dropped |
| 36 | pr-test-analyzer's behavioural-coverage rubric vs our test-gap heuristic | REV | not stated | dropped |
| 38 | Grader-agreement calibration pre-run | BEN | 100–120 labeled traces per dimension, ~50/50 (validate-evaluator:25), split 15/42/43, … | dropped |
| 42 | Cross-model transfer of harness self-optimization (Superpowers-6 style) | RTN | TB-R | dropped |
| 51 | Fresh-context single-pass reviewer vs same-session review | REV | TB-R items replayed as session2 (implement then review) | dropped |
| 53 | Thin-template reviewer (knowledge-work-plugins code-review/SKILL.md, 118 lines, 0 imperatives) | REV | TB-R | dropped |
| 54 | Anchored vs continuous confidence scores | BEN | TB-R | dropped |
| 67 | Official code-review (same-family fresh context, compute-matched) vs no review pass | REV | TB-R | dropped |
| 68 | Persona: generic role ("You are a senior backend engineer reviewing for correctness") | REV | TB-R, 1 run | dropped |
| 69 | Persona: real name only ("Review as Peter Steinberger would") | REV | TB-R, 1 run | dropped |
| 70 | Persona: real name + explicit heuristics | REV | TB-R, 1 run | dropped |
| 71 | Persona: explicit heuristics only, rationale style, no name | REV | TB-R, 1 run | dropped |
| 72 | Persona: heuristics-only through the cross-model seam | REV | TB-R, 1 run | dropped |
| 85 | Find-everything lane: Sonnet 5.5 at max vs Opus 5.5 at max (and Fable 5.1 at max) | REV | TB-R (120 defects / 60 diffs, 15 clean) | dropped |
| 86 | Find-everything effort ladder: Opus 5.5 at max (and xhigh) vs medium | REV | TB-R | dropped |
| 87 | Sonnet 5.5 at max with Anthropic's no-reviewer-subagents paragraph | REV | TB-R | dropped |
| 88 | /code-review --max-findings all vs the default limit | REV | TB-R | dropped |
| 90 | Haiku grader on the lean vs the verbose prompt | BEN | a hand-labelled set | dropped |
| 3b | MOASP one-shot probe | AMB | TB-B, one run per item (exploratory) | set aside |
| 41 | Mandatory TDD system prompt on Fable 5.1 (Kent Beck) vs none | IMP | TB-B | set aside |
| 21 | AC-first overnight pipeline (opslane/verify) | IMP | — | removed |
| 39 | Refactoring-to-reduce-tokens (per-step token deltas) | PLN | our own codebases (a protocol, not a TB) | removed |
| 57a | mattpocock architecture skills (codebase-design + improve-codebase-architecture) vs nothing | PLN | our own codebases (no TB) | removed |