What an academic legal-RAG auditor gets right, what it gets expensive, and why your enterprise buyer's TPRM questionnaire is about to ask you about both
If you are a CTO at a legal-tech company that has just started fielding Third-Party Risk Management questionnaires from an AmLaw 100 firm or a bank's procurement team, you have probably noticed something uncomfortable: the questions have changed.
Two years ago the questionnaire asked where the data lives, who has access, and whether you hold SOC 2. Custody questions. Answerable questions. Now there is a second block of questions underneath, and it asks things like "describe the controls that prevent the system from generating unsupported legal assertions" and "what is your measured hallucination rate and how is it monitored in production" and "how do you demonstrate that a retrieved authority is currently in force."
Those are not custody questions. They are correctness questions, and they are much harder to answer, because the honest answer for most RAG pipelines is a shrug wrapped in a paragraph about prompt engineering.
The questions are getting sharper because the assurance side is reading the literature. And in July 2026 the literature acquired a very good, very quotable reference point: Akremi (2026), "Ontology-Driven Legal Rule Auditor for Secure, Trustworthy, and Governed RAG Systems," published in MDPI Computers (Vol. 15, Issue 8, Art. 471) out of Umm Al-Qura University.
It is a genuinely strong paper. It is also, read carefully, the clearest existing demonstration of why the standard approach to legal-RAG governance is economically unshippable — and it demonstrates this using its own headline numbers.
This post is a full breakdown of the paper: what it built, what it proved, where its architecture creates a production ceiling, and what an alternative control architecture looks like when you design — not yet build — against the same class of problem. I have no product to sell you at the end of it, and nothing past §7 is more than a design worked out on paper; I will be explicit about that distinction when we get there. The reason to read it is that the assurance teams gatekeeping your enterprise deals are converging on the ideas in this paper, and it is better to have opinions about them before someone asks.
1. What the paper actually built
Let me describe it fairly, because it deserves that.
Most legal-RAG "governance" in the wild is a system prompt that says do not make things up and a citation regex. Akremi's system treats governance as an architectural layer with formal semantics, and the design is coherent end to end.
The ontology is a control, not documentation. An RDF/OWL ontology models legal instruments, provisions, jurisdictions, topics, and lifecycle states. Crucially it is not a passive schema — it is queried at request time to constrain what the retriever is allowed to see. Properties like belongsToJurisdiction are enforcement primitives.
Retrieval is gated before it happens. An NLU module infers jurisdiction and topic from the query, and the ontology gate filters the candidate chunk set before vector similarity is computed. If your query is about Saudi PDPL, the HIPAA chunks are not in the race.
Lifecycle validity is reasoned, not assumed. SWRL rules classify provisions as Active, Superseded, or Deprecated, with :isSupersededBy links between them. A superseded provision is structurally excluded from the evidence set. This is the part of the paper I like most, and I will come back to it, because it is the thing the paper does that I did not.
Two auditors bracket the model. An Input Gate screens for prompt injection and PII/PHI before generation. An Output Validator runs pySHACL shapes against the generated response — including a CitationInContextShape that checks every citation in the output actually exists in the approved evidence set.
Everything is hashed. Each interaction produces a cryptographic hash record, mappable to W3C PROV-O. This is exactly the artefact a TPRM reviewer wants and almost never gets.
The stack is deliberately modest: Llama 3.1 8B quantised to Q4_K_M served via Ollama, Ontotext GraphDB for the ontology, Qdrant for vectors, BAAI/bge-m3 embeddings, pySHACL for validation. It runs on a single RTX 4070 with 8GB of VRAM. The evaluation corpus is four real instruments — Saudi PDPL, EU GDPR, US HIPAA, US HITECH.
The safety results are excellent:
| Metric |
Result |
| Unsafe refusal rate |
100% |
| Stale-law leakage |
0% |
| p95 latency |
~14.1 s |
| Abstention rate |
51.4% |
Read that last row again.
2. The number nobody is quoting
The system blocked 51.4% of all responses.
Not 51.4% of adversarial responses. Not 51.4% of responses that were actually wrong. 51.4% of everything the system was asked to do, it declined to answer.
To the paper's considerable credit, it does not hide this. Section 7.4 is candid about the mechanism: SHACL is a structural validator, and it discards entire responses over things like a synonym choice — the validator sees "ensures" where the shape expected "warrants" and fails the whole output. The paper also notes that the repair loop was "not separately instrumented or evaluated," which is a polite way of saying that regenerating a failed response introduces cache contamination and roughly doubles compute, so they did not test it.
For an academic contribution this is fine. The paper's claim is that ontology-governed retrieval plus declarative validation can make legal RAG safe, and it demonstrates that convincingly. Nobody is shipping this to lawyers.
But you are shipping to lawyers. So run the number.
A system that refuses half of all queries is not a product. It is a product with a coin flip in front of it. Every one of those refusals consumed a full generation pass — the tokens were produced, the GPU time was spent, the latency was incurred — and then the output was destroyed. You paid full price for zero user value, 51.4% of the time.
And here is the part that matters for your procurement conversation: your enterprise buyer will not read the abstention rate as a cost problem. They will read it as a safety benchmark. "The literature achieves 100% unsafe refusal and 0% stale-law leakage" is now a number in someone's evaluation rubric. You are going to be asked to match it. If the only architecture you know for matching it is post-hoc validation, you will match it by refusing half your queries, and your product will die of it.
So the useful question is not how do I hit those safety numbers. It is: is the abstention rate a property of governance, or a property of this particular enforcement architecture?
It is the second one. Here is why.
3. Root cause: validation granularity is mismatched with contamination granularity
The paper's control flow is: generate the whole response, then judge the whole response.
graph LR
subgraph AKREMI ["Akremi (2026): post-hoc validation"]
direction LR
A1[Query] --> A2[Ontology gate]
A2 --> A3[Retrieve filtered chunks]
A3 --> A4[LLM generates full response]
A4 --> A5[SHACL validator]
A5 -->|Pass| A6[Release]
A5 -->|Fail| A7["Block — 51.4% of responses"]
end
Contamination, however, does not occur at the granularity of a response. It occurs at the granularity of a claim — usually one sentence, sometimes one clause. A 900-token answer with one unsupported sentence in it is not 900 tokens of garbage. It is roughly 40 tokens of garbage and 860 tokens of correct, useful, expensive work.
Post-hoc validation cannot express that distinction. Its only two verbs are release and destroy. So a single bad sentence costs you the entire response, and because the validator is structural it also costs you responses that were never wrong in the first place — just phrased unexpectedly.
This is the mechanism behind the 51.4%. It is not that the model was wrong half the time. It is that the enforcement boundary sits at the wrong granularity, and the blast radius of any single failure is the whole output.
Move the enforcement boundary and the number moves with it.
4. Root cause: your inference backend determines your control surface
There is a decision buried in the paper's methods section that constrains everything downstream, and it is easy to skim past: the system serves Llama 3.1 8B via Ollama.
Ollama does not expose KV cache state to the application layer.
That single fact forecloses an entire category of control. If you cannot reach into the generation state during inference, then you cannot:
- inspect what the model has committed to mid-generation
- selectively discard the attention state associated with a bad span
- resume generation from a known-clean point
Which means your only available remediation is generate everything, judge everything, destroy everything. The post-hoc architecture is not really a design choice in this paper. It is the only architecture available given the serving layer.
This generalises, and it is the single most portable lesson in the whole analysis: the compliance controls you are able to implement are bounded by the inference hosting you chose, and that choice is usually made for reasons that have nothing to do with compliance. If you are on a managed API endpoint — any of them — the generation state is opaque to you, and every control you can build is necessarily post-hoc. You will discover this at the worst possible moment, which is when an enterprise buyer asks you to demonstrate inference-time control and you realise the answer requires a migration.
Worth checking now rather than in a procurement cycle.
5. Root cause: SHACL validates structure, not meaning
The paper's CitationInContextShape checks that every citation appearing in the output exists in the approved evidence set. That is a real and valuable control — it kills the classic fabricated-citation failure mode, the one that has produced a documented trail of court sanctions.
It does not touch the harder failure mode.
Suppose the model cites Article 99 for one proposition and Article 94 for another. Both articles are genuinely in the approved evidence set. Both citations resolve. But the prose connecting them asserts a legal relationship that neither article supports — a conditional that is not there, a scope that is broader than the text, an obligation attached to the wrong party.
SHACL passes it. Every citation exists. Every shape is satisfied. The output is structurally immaculate and legally wrong.
The paper is honest about this too — Section 7.4 states that the validator "looks at the presence and metadata of the citations rather than the deep semantic logic of the English connecting them."
This is the gap that should worry you most, because it is the failure mode that survives every control in the standard playbook. Citation-existence checks pass it. Retrieval-precision metrics pass it. Groundedness scores computed at the document level pass it. It only fails if something is comparing this specific claim against this specific span of evidence and asking whether the first is entailed by the second.
6. Root cause: the ontology gate cannot see identifier collisions
This one is a retrieval-tier problem and it is specific to civil-law corpora, so it may not apply to you — but if you operate in Spain, Italy, France, Germany, or most of Latin America, it applies to you very much.
The paper's pre-retrieval gate filters on jurisdiction and topic. That is genuinely effective at the boundaries it can see: it will not let a HIPAA chunk contaminate a PDPL query.
Now consider a query about "Article 42" in a jurisdiction where the civil code, the commercial code, the labour code, and three procedural statutes each have an Article 42, all within the same topic space. The ontology gate has already done its job — correct jurisdiction, correct topic. It has no further discriminating metadata to apply. The disambiguation is handed to the embedding model.
Dense embeddings cannot do this. "Article 42" produces a near-identical vector regardless of which statute it belongs to, because the discriminating information is a discrete identifier, not a semantic property. Similarity degrades roughly as O(1/n), where n is the number of colliding articles — with four colliding articles you are choosing close to at random, and the model then generates fluent, confident, well-cited prose about the wrong statute. ColBERT-style late-interaction reranking does not rescue you, because the reranker is scoring the same undifferentiated token.
The failure is silent. There is no low-confidence signal, no refusal, no anomaly in the logs. The system retrieved something, it was topically plausible, and the answer reads perfectly.
This is the retrieval-tier equivalent of the Article 99/94 problem: a failure mode that passes every control designed to catch a different failure mode.
7. An alternative enforcement architecture
The following section models an alternative enforcement architecture, specifying the structural controls and formal properties derived from deterministic filtering and in-flight inference controls.
7.1 Moving the enforcement boundary inside generation
The first design targets the granularity mismatch directly. Instead of judging the response, judge each sentence as it is produced, so remediation is scoped to the failing sentence rather than the whole response.
graph LR
subgraph OURS ["In-flight control"]
direction LR
B1[Query] --> B2[Statute-binding router]
B2 --> B3[Retrieve bound chunks]
B3 --> B4[LLM generates with in-flight audit]
B4 --> B5[Per-sentence NLI]
B5 -->|Entailed| B6[Stream to user]
B5 -->|Contradiction| B7[Truncate KV cache, resume clean]
end
The mechanism:
- Self-hosted inference with cache access. vLLM/SGLang with RadixAttention rather than a managed endpoint. This is the enabling decision — everything else depends on being able to reach the KV cache.
- Sub-span extraction. For each generated sentence, ColBERTv2 token-level MaxSim scores identify the specific ~50-word span of evidence that sentence is claiming support from. Not the document. Not the chunk. The span.
- Sentence-level NLI. An 8B natural-language-inference auditor evaluates entailment between the sentence and that span, returning Entailment / Contradiction / Neutral. This is a semantic judgement — it is the control that catches the Article 99/94 problem, because it asks whether the claim follows from the evidence, not whether the citation exists.
- Selective rollback. On Contradiction, the RadixAttention tree is navigated back to the last verified branch point. The contaminated branch is garbage-collected — the model's attention state no longer contains the bad sentence, so it cannot condition subsequent tokens. A corrective context prefix resumes generation from clean state.
The last point is the one that's easy to skip past on paper but is actually the crux: it is not enough to delete a bad sentence from the output string. If the bad sentence is still in the KV cache, the model continues reasoning from it, and you get a response that is superficially corrected and structurally still poisoned. Truncation is the difference between hiding contamination and eliminating it.
None of this has been run. Here is what the construction implies, with each line labelled by how it's derived rather than presented as a result:
| Measure |
What the design implies |
Basis |
| Contaminated output reaching the consumer |
0%, by construction |
If truncation happens before the next token is sampled, the contaminated span cannot condition anything downstream. That's a property of the control flow, not something that needs a benchmark to establish — assuming the NLI auditor's verdict is correct, which is a separate, real question §9 does not claim to answer. |
| Wasted tokens per contradiction event |
Roughly the ratio of sentence length to response length — illustrative example: ~40 vs. ~800 tokens, ≈95% less waste than discarding the whole response |
Arithmetic on assumed lengths, not a benchmark. Real sentence and response lengths will vary by domain and query type. |
| Rollback vs. full regeneration |
An order of magnitude or more faster, directionally |
A cache-tree rollback is a pointer operation; full regeneration re-runs the forward pass from scratch. I have not profiled this on real hardware and am deliberately not putting a millisecond figure on it until I have. |
| Per-query cost vs. regenerate-on-failure |
Follows from the cost formula in §7.3 once you plug in a real r and real token costs |
Modeled, not measured. See §7.3. |
The waste-reduction line is the one that matters conceptually, and it holds regardless of the exact lengths: if contamination is one sentence and you discard one sentence instead of the whole response, you save roughly the ratio between them. The architecture does not need to be clever to win there. It needs to operate at the right granularity. Whether it actually achieves 0% contamination and sub-second rollback in practice is an empirical question I have not yet answered.
7.2 Deterministic disambiguation before the embedding layer
The second build targets identifier collision, and the design principle is that some problems should not be handed to a similarity function at all.
- Entity extraction parses article numbers and statute keywords from the query.
- Collision detection checks the extracted identifier against a pre-computed index of every article number that appears in more than one statute in the corpus.
- Hard metadata filtering constrains the vector search space to a single statute before any similarity is computed:
{statute_id = 'CdC', article_num = '42'}.
That is a boolean constraint with O(1) precision. It is not a better ranker. It is the removal of the ranking problem.
The claim here is stronger than the usual empirical "our system did better" claim, because it's structural rather than statistical: a boolean equality filter on {statute_id, article_num} either matches the queried statute or it does not. There is no similarity score to get unlucky on. So for any query where the router correctly extracts the article number and statute keyword — a much easier problem than full semantic retrieval, since these are typically well-formed, regex-extractable tokens — the wrong-statute failure mode is not reduced, it is structurally impossible, by construction.
What I cannot tell you without building it: what fraction of real-world queries the extraction step actually handles cleanly, how much smaller the search space gets on a real corpus, what the true collision rate looks like once you compute it against an actual civil-law dataset, or what this costs in infrastructure once it's running against production volume. Those are empirical questions, and I am not going to attach numbers to them that I have not measured.
7.3 The economics the literature does not model
The paper reports p95 latency and stops. There is no cost model anywhere in it — no per-query cost, no cost of blocked responses, no scaling behaviour, no comparison against an ungoverned baseline.
For a system that discards 51.4% of its generations, that omission is doing a lot of work.
The framing I use is a hallucination tax: the ongoing OPEX penalty a validation architecture imposes per query, as a first-class number rather than an implementation detail.
Ctax=r⋅Cgen+r⋅Tout⋅Pout
Where r is the failure rate that triggers remediation, Cgen is the cost of a full generation pass, Tout is output tokens, and Pout is price per output token. The structure of the formula is the argument: in a post-hoc architecture, the whole of Cgen is multiplied by r, because the whole response is destroyed. In an in-flight architecture, Cgen is replaced by the cost of the discarded span, which is one to two orders of magnitude smaller.
At r=0.514 — the paper's own number — the post-hoc tax is a permanent surcharge of roughly half your inference budget, forever, growing linearly with volume. That is not a tuning problem. It is a structural property of the enforcement architecture, and no amount of prompt engineering touches it.
Plug in a range of query volumes, from 10K/day to 1M/day, across model tiers, and the shape of the formula does not change: the gap between post-hoc and in-flight widens with scale, because post-hoc waste scales with volume × response length while in-flight waste scales with volume × span length. That is a property of the algebra, not a claim about anyone's actual traffic.
If you take one operational habit from this post, take this one: put the governance overhead in your unit economics model explicitly. Most teams model inference cost against successful responses and treat blocked responses as a rounding error. At a 51% abstention rate they are not a rounding error; they are the larger line item.
8. Where the paper beats the design above
I would not trust this analysis if it only ran one direction, so here is the other direction. Four things the paper does that the design in §7 does not — and this comparison holds regardless of whether §7 ever gets built, because it is a comparison of what each specification covers, not of implementation quality.
A formal, machine-readable governance specification. The RDF/OWL ontology with SWRL rules and SHACL shapes is a declarative artefact. You can hand it to an auditor. You can diff it across versions. You can reason about it independently of the code — and, notably, independently of whether it has been implemented, which is exactly the property this section of my own post cannot currently claim for itself. A router, an index, and an NLI auditor are procedural by nature, even on paper: they describe what the system does, not a specification an auditor can evaluate on its own terms. When a TPRM reviewer asks "show me your governance model," a SHACL file is a much better answer than a description of a pipeline. This is a real, structural advantage of the paper's approach, not just a maturity gap.
Lifecycle and temporal reasoning. The SWRL validity engine with :isSupersededBy supersession tracking is a real capability the design in §7 does not have an equivalent for. As designed, the statute-binding router would catch repealed provisions only incidentally — binding to the correct statute happens to exclude the wrong one, if that wrong one happens to be the repealed version. That is luck dressed as a control, not a mechanism, and it is a gap in the design itself, not something a build would fix. There is no hasEffectiveDate, no hasExpiryDate, no explicit supersession graph in what I have sketched. For a corpus where amendments are frequent, the paper's approach is the right one and mine is a gap I have not closed on paper, let alone in code.
Cross-jurisdictional comparison queries. "Compare HITECH and GDPR breach notification timelines" requires maintaining jurisdiction-separated evidence groups through generation so the model does not blend them. The paper handles this. The design in §7 does not — the router as sketched binds one statute at a time, and extending it to bind two in parallel while preserving separation is a real design problem I have not worked through.
Input-tier safety. The paper's Input Gate screens prompt injection and PII/PHI before anything else runs. The design in §7 has nothing at the input tier at all — it starts at retrieval and covers generation, and that is a straightforward, acknowledged hole. The paper's dual-auditor pattern is architecturally compatible with what's sketched in §7: an Input Gate ahead of the statute router would yield a clean three-tier stack — input safety → retrieval integrity → generation integrity — but that is itself an unbuilt, unverified extension on top of an unbuilt, unverified base.
The honest synthesis is that these look like complementary halves rather than competing designs, on paper. The paper solves what should be checked — it defines governance rules formally, in a way that survives audit, and it has actually been implemented and evaluated. The design in §7 is an argument about how enforcement might avoid destroying the product, worked out carefully but not yet tested against anything. Neither half, as it stands, is sufficient on its own: a formal governance specification with a 51% abstention rate does not ship, and an in-flight enforcement architecture that exists only as a diagram does not pass procurement either.
9. What neither of us solves
The genuinely open problems, stated as problems rather than as opportunities.
| # |
Unsolved |
Why it bites |
| 1 |
Multi-claim argumentative coherence |
Both SHACL and NLI validate claims individually. Neither validates the logical structure connecting them. A response where every sentence is independently entailed can still constitute an invalid legal argument. |
| 2 |
Sub-sentence streaming validation |
Sentence-boundary auditing introduces micro-pauses. Validating at sub-sentence granularity without user-perceptible latency is unsolved as far as I can tell. |
| 3 |
Automated amendment-graph construction |
The paper requires an administrator to manually assert :isSupersededBy links. Nobody automates the detection of which provisions a new amendment supersedes, and manual curation does not scale past a small corpus. |
| 4 |
Multilingual legal term disambiguation |
Terms that shift meaning across jurisdictions sharing a language — Spanish "letra" vs. "apartado" across Spanish and Latin American codes is the case I hit. No systematic solution exists in either body of work. |
| 5 |
Calibrated confidence for legal claims |
NLI confidence scores and SHACL pass/fail are not calibrated against any legal standard of certainty. Handing a lawyer "0.87" means nothing to them and arguably means nothing at all. |
| 6 |
Human-in-the-loop that actually satisfies EU AI Act Art. 14 |
Both bodies of work acknowledge the requirement. Neither demonstrates a complete implementation beyond flagging outputs as "Unverified." Nobody has shown me a production HITL design that a regulator would accept. |
Number 1 is the one I would most like someone to solve, and I do not currently have a good approach to it.
10. Turning this into questionnaire answers
Practical residue, because the abstract argument is only useful if it changes what you write in a form.
The TPRM correctness questions map onto a small number of architectural facts. If you can state these four things clearly, you are ahead of nearly everyone:
Where does your enforcement boundary sit? Pre-retrieval, post-retrieval, in-flight, or post-generation. This single answer determines your remediation cost, your abstention rate, and your latency profile. Most teams have never articulated it and discover their answer under questioning.
Is your validation structural or semantic? Citation-existence checking is structural. Claim-evidence entailment is semantic. Say which you do. If you only do the first, say that too — an accurate limited answer survives diligence far better than an inflated one, and a reviewer who catches you overstating one control will re-open all of them.
What is your control surface at inference time? This is really a question about your serving layer. Managed endpoint means opaque generation state means post-hoc only. Fine, if you know it and can say it. Not fine if you claim inference-time controls you cannot implement.
How do you handle identifier collision, if your corpus has any? Compute the collision count in your own corpus first — that is a short script, not a project. If it is non-zero and you are relying on dense similarity, you have a silent failure mode with no observability, and it is better to find it yourself.
Underneath all four is a single reframing that I think is the actual takeaway. Governance is not a policy layer bolted onto a pipeline. It is a property of where in the pipeline you put the enforcement boundary, and what state that boundary is allowed to touch. Akremi (2026) demonstrates this negatively and rigorously: excellent rules, wrong boundary, 51.4%.
The paper is worth reading in full. It is careful, honest about its own limitations in a way that is unfortunately rare, and it is going to be cited in assurance frameworks for the next several years. Better to have read it.
Methodology note: §§1–6 are a straight reading of the published paper (Akremi, 2026) and its own reported numbers — those figures are the paper's, not mine. §7 is a design, not a build: the architecture is worked out in enough detail to reason about its properties, but nothing in it has been implemented, run, or measured against a corpus, and every number in that section is labelled accordingly, either as a structural consequence of the design or as an explicitly-flagged illustrative estimate. Separately, I maintain an open-source deterministic evaluation harness, legal-audit-rag, for running pass/fail compliance checks against a RAG pipeline rather than LLM-as-judge scoring — it is not the source of any number in this post, and I am linking it here only because it is relevant to the general problem, not as evidence for §7's claims. Corrections to any of the above are welcome; I would rather be corrected than cited wrongly.
Source paper: Akremi, A. (2026). "Ontology-Driven Legal Rule Auditor for Secure, Trustworthy, and Governed RAG Systems." Computers, 15(8), 471.
]]>
A visual contemplation of the server architecture.