legal-rag-verifier is an open-source proof of concept. It checks each sentence of an answer while the model is still writing it. A rejected sentence is cut, the model is given the statutory text it needs, and the sentence is written again. The text already checked stays in the server's memory, so a retry computes only the new words.

Every figure below is Measured, reference implementation, on English questions about UK statutes unless a line names another set. Each method was fixed before its run. The code, methods, statistical tests and result files are on GitHub. The specification it was built from is in the generation-tier case study, §6.7.

Summary

• Measured, 14 October 2026, 240 questions answered from memory, Qwen 27B as judge on one NVIDIA L40S: answers holding a wrong sentence fell from 132 without the loop to 23 with it for Qwen 27B checking its own answers (83% fewer), and from 198 to 28 for Qwen 2.5 7B (86% fewer).
• Measured, 14 October 2026, Qwen 2.5 7B, 95 wrong sentences it wrote from memory: the default retry was wrong again 71 times. A retry with the provision added as a message at the cut was wrong again 16 times.
• Measured, 15 October 2026, the same 240 questions with the source in the prompt: answers holding a wrong sentence fell from 25 to 13 for Qwen 2.5 7B.
• Measured, 16 October 2026, the same setup for Qwen 27B: it had 4 answers holding a wrong sentence with the loop and 4 without, at 2.0 s and 1.1 s per answer at the median. Two large hosted models had 3 (Gemini 3.8 Flash) and 8 (Claude Opus 4.6).
• Measured, 8 to 11 October 2026: the first checker, a small model that scores whether a sentence follows from a text, was usually wrong on live answers. On statutes, 28 of the 46 sentences it rejected were correct, by Claude Opus 5.5's labels. Qwen 27B replaced it as the judge and was right on 90.1% of 151 sentences graded by hand, against 58.3%.
• Measured, 6 October 2026, Qwen 2.5 7B on one NVIDIA L4: a retry costs 117 ms, against 270 ms to process the prompt again.

The cost of checking a finished answer

A hosted model API returns a finished answer. A checker placed after it can only accept or reject the whole answer. One wrong sentence means a full regeneration, and every word of the rejected answer is paid for and discarded. The case study calls this cost per question the Hallucination Tax and prices it in §4.

By design, a check per sentence discards one sentence instead of one answer, and the reader sees checked text sooner.

How the loop works

1. The model writes one sentence.
2. A judge, a second model or the same one, checks it against the provisions supplied with the question.
3. A sentence that passes is kept. A rejected one is cut back to its start and rewritten, at most 3 times. This is a rollback. After 3 failures the engine writes a fixed refusal sentence.

The loop runs on SGLang, an open-source server for language models. While reading text, a model computes an intermediate result for every word, called the KV cache. SGLang keeps that result after a request ends and reuses it for any later request that starts with the same text. A retry starts with the text already kept, so only the words after the cut are computed.

Before the rewrite, the loop adds the provision text as a new message in the conversation, placed at the cut. The model reads it before writing the sentence again. It does not appear in the answer.

How answers were graded

The grader differed by test. Where models graded, the sentences they left open came to me. From 14 October, a sentence word for word equal to one graded in an earlier test kept that grade.

The 46 retries of 7 October are counted by the first checker's verdict.

What the cache and the inserted text each do

The provisions were already in the instructions at the start of the conversation for every retry below.

Measured, 7 October 2026, Qwen 2.5 7B on one L4, 46 rejected sentences retried from the same point, checked by the first checker:

Emptying the cache changed no result. It raised the time of the first request from 105 ms to 337 ms. The cache makes the retry cheap, and the position of the provision changes what the model writes.

The first checker's verdicts were mostly wrong (next section), so the retries were compared again with the Qwen 27B judge.

Four retries were compared, each from the same point after a wrong sentence was cut:

1. The default retry restricts the first word. If the rule-based check found the right value in the source, the model may only write that value. Otherwise it may not start with the rejected sentence's first word.
2. A message at the cut adds the provision as a new user message after the text kept so far, with the question restated. It stays in the conversation for the rest of the answer.
3. Plain text at the cut feeds the same provision into the model as if it continued the answer, with no message around it. It is removed before the next sentence.
4. A KV cache insert computes the provision's KV cache on its own and attaches it at the cut, so the model holds the provision in memory without reading it in that position. It is removed before the next sentence.

A good retry is supported by the source, answers the question and reads on from the text before it. A retry wrong again is unsupported by the source or contradicts it. An empty retry wrote nothing. The rest were supported but off the question or did not read on.

Measured, 14 October 2026, Qwen 2.5 7B in Hugging Face transformers on one L40S, which gives direct access to the KV cache, 95 wrong sentences written from memory and rejected by the Qwen 27B judge, first retry only:

Plain text gave the most good retries, but in the live loop it ended answers early: 30% of Qwen 27B's answers from memory ended in the refusal, against 4% with the message (14 October 2026). The loop uses the message.

The first checker failed

The first checker was a rule-based check on figures, citations, Act titles and qualifiers, followed by DeBERTa-v3-base, a small model that scores whether a sentence follows from a text.

Measured, 6 October 2026: on a test set of 346 planted errors and correct sentences, fixed and hashed before the run, it caught 80.0% of the errors and rejected 7.9% of the correct sentences. 89% of its rejections were right.

Measured, 8 October 2026, Qwen 2.5 7B on one L4, live answers:

• On statutes, it rejected 46 sentences, and 28 of those were correct, by Claude Opus 5.5's labels.
• On 40 software questions with the documentation quoted as the source, graded entirely by hand, 1 of its 12 rejections was a real error. It missed 5 of the 8 wrong plain answers. The loop's final answers were good 29 and 30 times, against 32 for one plain answer.
• With the provision added at the cut, Qwen copied it. On the 46 statute retries, 28 pasted the provision or started mid-clause, and the checker accepted them.

It failed on statutory language. It misread double negatives, judged a correct rule against its exception, and passed sentences that dropped a qualifier. On the test set it caught 6 of 18 dropped qualifiers.

The LLM judge that replaced it

The replacement judge is Qwen 27B with its weights compressed to 4 bits. It is asked whether the source states each sentence, and a sentence the source does not state is rolled back.

Measured, graded by hand, 11 October 2026, 151 sentences never used to tune any judge:

A judge call took 0.56 s at the median, on the same server as the model writing the answer (12 October 2026). A 1B model trained on Qwen 27B's labels and two hosted decision APIs were also tested. None came within 3 points of Qwen 27B on both hand-graded tests, the rule set before the runs.

The loop with the judge

Measured, 240 questions per model, answers holding at least one wrong sentence. "From memory" means without the source in the prompt.

Measured, 12 October 2026, Qwen 27B on one L40S, 40 questions run live. With the source in the prompt, an answer took 1.9 s at the median. 39 of the 40 answers had no rollback, and none of the 49 sentences was wrong. From memory, an answer took 3.1 s. 3 of 50 sentences were wrong, all three the engine's refusal, against 36 of 65 without the loop.

Measured, 7 October 2026, Qwen 2.5 7B on one L4: the first checked sentence appeared after 3.8 s when sentences were checked one by one, against 8.1 s when the whole answer was checked first.

Against Gemini 3.8 Flash and Claude Opus 4.6

Measured, 16 October 2026, the 240 questions with the source in the prompt. Each model answered each question once, with randomness turned off.

The rule set before the run allowed a gap of 4 percentage points, chance included. Qwen 27B with the loop met it against both models. Qwen 2.5 7B with the loop did not.

The match comes from Qwen 27B with the source in the prompt. Without the loop it had the same 4 answers holding a wrong sentence, at 1.1 s instead of 2.0 s. It states less per answer than either hosted model.

Where the loop pays

The loop reduces wrong answers when the model answers from memory, or when the model is small. From memory, it cut answers holding a wrong sentence from 132 to 23 for Qwen 27B and from 198 to 28 for Qwen 2.5 7B. With the source in the prompt, it cut them from 25 to 13 for the 7B.

With the source in the prompt and Qwen 27B writing, the loop added 0.9 s per answer and removed no error. On those answers the judge rolled back more right sentences than it caught wrong ones. It was right on 88.9% of their sentences, against 94.4% with no judge (11 October 2026).

A system that already puts the source in the prompt gains accuracy from this loop with a small model and none with a large one. In both cases a sentence caught during writing costs one retry instead of a new answer.

Limits of this evidence

• Each setup ran once, on one server type, with randomness turned off.
• Against the hosted models, Claude Opus 5.5 graded alone, including the answers of Claude Opus 4.6, a model of the same family. It was softer on Claude Opus 4.6's sentences than on the others.
• The 1.9 s figure is from 40 questions. On 240 questions the same setup took 2.0 s.
• The runs of 14 to 16 October use the LLM judge as a script in the repository at commit c044b84. Release 0.2.0 does not include it.
• The original target of 30 ms per retry is missed. One pass of a 7B model on an L4 takes about 53 ms, and a retry 117 ms.
• With the first checker, the per-sentence loop and a full retry of the answer were not told apart on accuracy.

Reproduce

docs/VERDICT.md states the verdict. docs/measurements.md holds every figure, and docs/GRID.md every method, each committed before its run. docs/RESEARCH.md maps each test to its scripts and result files.

The same kind of test, fixed and dated before any answer exists, can be run against a legal RAG system I did not build. The Retrieval Integrity Diagnostic reports counts. £500, up to 5 working days, credited in full against the next stage. Engagements.