How this code got checked

Test-driven development used to be the practice everybody agreed with and nobody did, because it doubled the typing. That objection is gone: tests cost very little to produce now, and a model will write thousands of cases in an afternoon. Volume alone proves little. What makes a suite worth trusting is what it was built to catch and the gate it is wired into: a threshold enforced in configuration, a hook that blocks the commit, a pipeline that fails the build.

That's the thesis this page tests against: TDD doesn't prevent bugs from happening. It can code a bug right through. The version of that failure I watch for most is narrower. A model will code a bug straight through a test it wrote itself, because the test encodes what the model already believed. It does not know how the system is supposed to behave, or how two pieces are meant to fit together. Those two gaps are where nearly everything goes wrong.

So the interesting question stopped being how much is tested and became what is checking the thing that is checking. What follows is the answer I arrived at, in the order I arrived at it.

The coverage threshold is 95 per cent per file, on lines, functions, branches and statements, and it is enforced in configuration rather than in a habit. Per file matters more than the number does: a project-wide average lets a well-tested module pay for an untested one, and the untested one is always the one that breaks.

One project pushed this further with a pre-commit hook that generates a temporary test configuration scoped to exactly the files staged in that commit, so old code can neither hide a new gap nor block a small fix. A second script then fails if the hook's list of gated packages and the check script's list have drifted apart. The gate checks its own configuration.

Four repositories carry hosted CI: PHIGuard's marketing-seo.yml runs on every push, every pull request and a weekly schedule; Decision-making skills runs its whole 22-step de check gate in check.yml on every pull request and every push to main, which it can do because the gate makes no model calls and is deterministic; CapVeri has a six-job pipeline held in reserve, its triggers commented out in ci.yml with a note on how to switch it back on; GrantPipe builds its release artifact there. Everywhere else, the gate is a named, scripted, exit-code-enforced check instead. pnpm verify in ventora-platform is nine steps (schema check, metrics check, secret scan, script tests, lint, typecheck, per-file coverage, and two smoke suites), and the release scripts require it to pass. verify:rollout in ventora-crm is fourteen steps, and deploy is defined as depending on it. run-affected-checks in grantpipe, kaiplan, gavelhouse and ventora-platform resolves which packages a commit touched and runs each one's build, typecheck, coverage, ruff, mypy and pytest. Kaiplan runs two layers of husky hooks: pre-commit, then a pre-push that runs lint, typecheck and coverage again. CapVeri runs thepre-commit framework, ten hooks including a real build:dev. The three .NET repositories run dotnet format --verify-no-changes, dotnet build anddotnet test from .githooks/pre-commit, layered in parallel, and a non-zero exit blocks the commit. Coverage is enforced in configuration:--cov-fail-under=95 in two pyproject.toml files, and a 95 per cent per-file threshold across six kaiplan vitest configs and nineteen in ventora-platform.

It ran on local machines rather than GitHub's runners, because paying for GitHub Actions minutes across fifteen projects running at once cost real money, and that was the whole reason.

CI is local. Treat pnpm verify plus the relevant Python package checks as the required CI gate before handoff. (ventora-platform CLAUDE.md:127)

The gate ran on one operating system and one runtime version each time, narrower than a hosted matrix would have checked.

Decision-making skills carries three guards past that step count alone. A provenance step reads the commit graph itself and refuses to publish a run whose README names a prediction committed after the run it explains, so a result cannot be dated into looking pre-registered after the fact. A three-judge panel adjudicates the project's own answer key blind: each judge runs in an isolated subprocess, sees only raw turn text, and never the maintainer's label or the other judges' verdicts, and disagreement above a pre-registered threshold moves the label instead of the first judgment standing. The same instrument has now caught itself twice: first when the corpus behind every result published up to that point turned out to be 89 per cent solvable by counting words, and again on an unmerged branch, where two prompt optimizers both produced winning skills that had transcribed the project's own answer key rather than improved decision quality.

After the money maths in CapVeri moved from Python to TypeScript, the Python stayed: kept, maintained, tested continuously against the engine that shipped, though never deployed again.

The reason is narrow and, I think, correct. Tests written alongside a rewrite encode the author's understanding of the requirements. If that understanding is wrong, the tests agree with the bug, the suite is green, and the bills are wrong. A reference implementation written earlier, from the same source documents, breaks that loop: when two independent implementations disagree on a generated input, either one of them is wrong or the specification is ambiguous, and all three of those are worth knowing.

The written mandate cuts both ways, and that part matters as much as the idea: matching the oracle is evidence, and diverging from it is a signal worth checking. Neither one gets to be the authority.

The unpublished CAM audit engine is graded against twenty-four synthetic lease-and-statement pairs. Twenty of them supply the same reconciliation statement in more than one file format, so a planted defect runs more than one ingestion path into the system.

Two of the twenty-four plant no defect at all. Their expected-findings set is empty. They are there so the harness measures false positives as well as recall, because a detector that flags everything scores perfectly on a corpus where everything is a defect.

The generated report is the part I would point at first. It lists four rules that produced zero findings across three runs, calls them reproducible misses, and prints that in the same table that reports a pass. Printing a miss next to a pass, in the same table, is what makes the rest of the table worth reading.

An agent opens the product in a real browser, uses it, takes screenshots and reads them back. It is the least sophisticated thing described on this page and it finds more than anything else here.

The reason is the shape of what models get wrong. They write the parts well and then wire them together badly, or half-finish the join and move on. Integration and appearance are the two things a unit test is structurally unable to see, and they are exactly where the work is still weakest. Text still overflows its container. Things are still misaligned. That has not stopped being true.

Where a model has to read a document and pull figures out of it, the governing rule keeps arithmetic out of its hands: it extracts and classifies, and typed code calculates. That boundary survived a full rewrite from Python to TypeScript, which is the only evidence that it was ever real.

Around it sit six independent guards: a critic that reviews findings for plausibility, a judge from a different model family asked to argue against the extraction, a verifier that recomputes the arithmetic behind a stated result, a check that every asserted number traces to a place in the source document, a cross-validation of independent reads of the same input, and a guard fencing the one calculation that appears in almost every finding.

The verifier's own comment in the source concedes its limit: a hallucinated extraction pushed through correct arithmetic produces an equally true-looking proof. Leaving that note in the code is the honest version of shipping the guard.

They were built in the order the failures arrived, and the pattern I draw from them is that the labour moved and the judgement stayed put. A model writes a suite faster than I can read one, and it writes a suite that agrees with a bug just as fast. Deciding what would have to be true before the output could be trusted, then building the thing that settles it, is the part that stayed mine.

That instinct came from the day job. I am certified at the ISTQB Advanced level, and for four years at Vanco my work has been making other engineers' automation trustworthy. Whether it is the most useful thing on this site is a judgement for whoever reads the work, which is why the work comes first and this section comes last.