Each result below is from my own code. Five were measured against a rule or target set before the run; two were found in review. Each says what was expected, what was found, and what changed. The code and the result files are public.

1. The small auditor's rejections were mostly wrong

The expectation was that a claim check plus a small entailment model (DeBERTa-v3-base) would catch unsupported sentences while the model writes. A rejected sentence is cut and rewritten (the loop), so the answer improves.

Measured, legal-rag-verifier, 6 to 8 October 2026: on a battery of planted errors, fixed and hashed before the run, it caught 80.0% (124 of 155) and wrongly rejected 7.9% of correct sentences. On live output from Qwen 2.5 7B on one NVIDIA L4, about UK statutes, 18 of its 46 rejections were real errors. On 40 software questions graded by hand, the loop's answers were good 29 and 30 times, against 32 for one plain answer from the same model.

Qwen 27B then replaced it as the judge, right on 90.1% of 151 sentences not used in tuning, against 58.3% for the small model. With it, answers from memory (no source in the prompt) holding a wrong sentence fell from 198 to 28 of 240 for the 7B. Details: generation-tier case study, §6.7.

2. Hiding a model behind a key was recovered 100%

The expectation was that a secret rotation and permutation of an open model's weights would let an untrusted cloud fine-tune and serve it without reading client data. The rule: fail if any attack recovers 10% or more of the secret tokens.

Measured, blind-counsel, October 2026, SmolLM2-135M, 400 secret passages: the operator recovered 100% of the secret content from public statistics of the weights, at every fine-tuning checkpoint and across 4 keys. Recovery fell below 10% only when the model's prediction error was 2.9 times the original, which makes it unusable.

The design was dropped. Client data stays on the device and the cloud serves only public law. Details: confidential AI case study, §2.

3. Task routing failed its pre-registered rule, twice

The expectation was that the laptop model picks the right task for every matter held out from development, or refuses it.

Measured, blind-counsel, 6 October 2026: 7 of 8. A pregnancy discrimination matter, client still employed, was sent to unfair dismissal instead of refused. Later checks stopped it and no advice was issued, but the verdict is FAIL. blind-adapters, 8 October 2026: 10 of 12. Two redundancy matters were sent to unfair dismissal.

Between the two studies, each task got a public scope note that the router reads, and a failed precondition such as "dismissed? No" became a hard refusal. That fixed the blind-counsel miss: 10 of 10 on the development set, and both out-of-scope held-out matters refused. It did not separate the two in-scope tasks, which is the second failure. Routing is the weakest gate in both studies.

4. The router missed its latency target

The target was a p99 under 2 ms for routing a query.

Measured, legal-rag-router 0.1.0, 30 September 2026, full UK index, 306,336 calls × 3 passes: p99 2.10 ms on Apple M4, 9.21 ms on an Intel i5-3570. The tail is refusals, which rank suggested titles.

The target stands as missed. Both figures are under the 25 ms gate in the statute collision case study.

5. The rollback latency target was not reachable

The verifier's specification set 30 ms for a rollback and retry.

Measured, 6 October 2026, Qwen 2.5 7B on one L4: a retry costs 117 ms (69 ms with 8-bit weights). With 16-bit weights, one forward pass of a 7B model on an L4 takes about 53 ms. With 8-bit weights the retry still takes 69 ms. 30 ms cannot be met on this hardware.

The measured 117 ms, and 69 ms with 8-bit weights, are the figures quoted in place of the target.

6. A calculator fell back instead of refusing

The expectation was that every statutory date calculation refuses when an input is not verified.

A review of the scored blind-counsel run of 6 October 2026 found that when notice was given but its date was not verified, the calculator used the termination date. In one matter this did not change the answer, but it was wrong.

The fix is on the main branch, where the calculator now refuses. Tag poc-v1 still reproduces the scored run.

7. My own case study named the wrong statutory instrument

The expectation was that the agentic thrashing case study is right about the law it audits.

On 17 October 2026 I found that it attributed the £68,400 unfair dismissal cap to SI 2011/3006. The cap was set by SI 2010/2926; SI 2011/3006 lists £68,400 only as the limit it replaced. The audit's sealed ground truth names no SI, so its scoring was unaffected.

The page is corrected, with a dated note. The error was caught while building a table of the limit in force on each date, whose build checks each Order's old limit against the previous Order. On the same test question the audited system had drifted on, that table returned £68,400 from SI 2010/2926 on 1,000 of 1,000 passes.

Where to check

• legal-rag-verifier: docs/VERDICT.md, docs/measurements.md
• legal-rag-router: docs/measurements.md, results/pit-cs2-2026-10-17.md
• blind-counsel and blind-adapters: REPORT.md

The same method applies to a system I did not build. The Retrieval Integrity Diagnostic runs a battery fixed, hashed and dated before any response exists against your legal RAG system, and reports counts. £500, up to 5 working days, credited in full against the next stage. Engagements.