Decision-making skills

I built Decision-making skills as an agent skill that reads what’s hard about a decision and routes to the one procedure suited to it, de, a Python CLI harness that runs the experiments testing whether the skill helps, and a site that publishes the evidence for both. The skill installs as a directory copied into a coding agent’s skills folder. The model calls run from my own machine; the 22-step gate runs there and in GitHub Actions alike.

The decision

The project pre-registers before it publishes. A dated prediction goes into notebook/ before any result exists, and de check’s provenance step reads the commit graph itself: a run whose README names no prediction, or whose prediction was committed after the run it explains, does not publish. A second, hash-locked refusal guards the future confirm-tier run: it checks that a skill’s text and its analysis script still match what was pre-registered, and it carries its own full branch-coverage floor. Until a confirm run calls it, pyproject.toml declares that module by name as unwired, so the build itself keeps the label honest instead of letting an unexercised gate sit quietly and look tested.

Architecture

A placebo arm anchors the evaluation: a file matched to the real procedures on length and structure, built to sound helpful without implementing any of their logic, ships in the same directory as the real ones and stands in for the null hypothesis. A separate cot arm gives the model a plain “think step by step” instruction, isolating the skill’s own routing logic from the general benefit of asking a model to reason before it answers. Three arenas separate iteration from evidence: dev and screen may change the skill freely and never emit a verdict; confirm targets a private holdout and does emit one, and code enforces that split rather than discipline.

The same discipline governs what gets published about it. The sharpest finding the harness has produced so far is about its own instrument: the corpus behind the earlier results was largely solvable without a model. That scopes to answer-key version 2. The rebuilt corpus is far harder to shortcut, at 0.7054 against a 0.6667 baseline, and version 5 has since widened it to 330 items so that all six procedures have positives. The two keys are kept apart, and SCORECARD.md carries both figures with the arms measured against each.

The skill itself is one entry point that reads what’s hard about a decision, such as too much unranked context, advice that might not fit the person, or consequences that fork downstream, and routes to one of six procedures built for that shape rather than running every procedure on every question.

What the gate is aimed at

The gate is pointed at my own honesty more than at the code. de check binds every number the project publishes to a quoted sentence in the file that produced it, refuses a run unless git ancestry shows its prediction predates its data, and refuses a guard that has tests but no caller. It works: eleven of my own measurements have been caught broken, and all eleven are published. One result, N9, is void and published anyway, carrying the diagnostic that voided it, because 70 of its 516 model calls came back unparseable against a parse rate floor registered before the run started. The instrument turned on the panel behind it too, and three judges with a kappa of 0.862 compute to 1.10 effective raters.

That is also the finding I would keep if I could keep one. The project set out to measure whether a routing skill improves a decision, and what it has measured best so far is itself.

What is planned

docs/RESEARCH_PROGRAMME.md lays out sixteen tracks in eight parts, and it opens by naming the four load-bearing parts of the question it is trying to answer: a written skill, that measurably improves decisions, in an agentic system, that accumulates context over turns and delegates to sub-agents. It then says plainly that the repository has tested none of the four together. Every model call so far is a single claude -p with no tools and no session. Accumulation has been rendered and never lived. Delegation has not been attempted.

The programme’s own venue map names four instrument families across two axes, whether sub-agents run and whether turns accumulate: V1 is a single call with no sub-agents, the cell every result on this page lives in; V2 lets turns accumulate; V3 fans sub-agents out once and aggregates once; V4 is turns that accumulate and sub-agents that delegate, the system the question above is actually about.

The instrument caught itself a second time on 2026-08-27, on origin/evolution-study at commit 5daf654, unmerged and unsettled. Two prompt optimizers, GEPA and SkillOpt, run on a matched budget against the same seed skill, both produced winning skills that had transcribed the answer key rather than improved decision quality: SkillOpt’s winner appended an “Examples from the training data” section naming exact training values and verdicts across eight of the ten templates, and GEPA’s winner collapsed into a specialist for one template alone. That matched-budget comparison’s own accidental repeats, calls the resumed search happened to ask twice, gave 49 pairs scored twice with 7 disagreements: a 14.3% per-item flip rate, Wilson 95% [7.1%, 26.7%], which puts 1.2 items of noise on a 21-item score against winners three items above their seed, so the score gains sit inside the instrument. A separate, earlier pilot run, the same target model but a different reflector, had already caught the same failure mode on a different template: GEPA’s winner there memorised a distractor rule written for a security-patch template. In the matched-budget run itself, GEPA’s winner instead memorised a flight-rebooking template and left the rest alone. The remedy is recorded on the same branch, commit 499246c: a split that holds out scenario templates rather than instances or seeds, because a held-out instance of a memorised template still lets the memorised rule answer it.

Two lanes run in parallel and neither waits on the other. The product lane ships skills people install, and the research lane validates them. Ordering it the other way round produces a paper with a skill attached, and this is meant to be both. Inside the research lane the parts are sequenced so the cheapest disconfirming evidence arrives first.

The programme also writes down what would kill it, before a null could be mistaken for a fourth dead corpus. The kill condition is that the process measure moves and the outcome measure stays put, which is the shape of every result the repository has produced so far, and which has already happened to someone else at scale: a 2026 cluster-randomised trial across 103 clinical officers and 9,691 patients raised note quality on every domain and left treatment failure where it was. There is a competing reading for half of it, found on 2026-08-13. A flat result looks the same as a ceiling, and nobody had measured the ceiling. Track N measures it, which makes the earlier “nothing about a description changes discrimination” one of two live readings rather than a finding.

The verdict vocabulary is the part I would point at. A skill can come out SHIP, PROVISIONAL, NULL, HARMFUL, UNTESTED or WITHDRAWN, and a procedure I stop using for fourteen consecutive days is marked WITHDRAWN on that fact alone. Fourteen days was chosen to survive a holiday and not to survive disinterest, and it was written down before anything was near it, so it could not be picked to spare one. Evidence that can only come out positive is not evidence.

SCORECARD.md and docs/STATUS.md carry the running record, with corrections appended rather than edited.