Agentic Tool-Loop Thrashing in Multi-Jurisdiction Legal RAG
- Measured
- 3-pass statutory audit on a commercial multi-jurisdiction system, 7 August 2026. Identical query, same endpoint. Pass 1: £68,400 (61.3s, correct). Pass 2: £72,300 (27.7s, wrong). Pass 3: £72,300 (21.1s, wrong). Establishes drift across passes, not a population rate.
- By design
- A deterministic AST pre-router resolves formal statutory citations to structured metadata in <2ms with 0 LLM tool calls. Combined with bitemporal SQL parameter binding, the query returns bit-identical results across all passes.
The Challenge
A multi-jurisdiction legal AI system operating across UK, French, EU, and Spanish corpora was audited against a sealed 14-probe test battery on 7 August 2026. On probe pit-era-124-1, a UK statutory question with zero syntactic ambiguity, the system's autonomous ReAct agent executed 15+ tool calls across 186 seconds, retrieved 117 document fragments including 57+ French labor code articles, and inverted from the correct answer (£68,400) to an incorrect one (£72,300) across three identical passes. The agent's own internal thinking trace records it correctly deducing £68,400 before discarding its finding under context saturation. The root cause is an unconstrained agentic tool loop applied to a query class that resolves deterministically via syntactic parsing and bitemporal database lookup.
186 seconds. 15 tool calls. 117 retrieved document fragments, 57 of them French labor code articles, to answer a single UK statutory question with zero syntactic ambiguity. The system held the correct answer at thinking event [6] (Note 1) and discarded it by event [14] (Note 2).
This document pairs a forensic trace of agentic thrashing in a live multi-jurisdiction legal AI system with a deterministic routing specification that eliminates the failure class by construction.
| Label | Meaning |
|---|---|
| Measured | Produced by our sealed test battery on a live system. |
| Verified in isolation | Tested standalone, not inside a production system. |
| By design | Reasoned from architectural properties, not run end-to-end. |
1. Structural Syntactic Fingerprints Across Jurisdictions
Legal traditions structure their statutes in distinct, non-overlapping citation formats. A multi-jurisdictional lawyer identifies the origin jurisdiction from syntax alone. A parser can do the same.
| Jurisdiction | Citation Structure | Example | Distinct Markers | Primary Corpus |
|---|---|---|---|---|
| UK (Common Law) | Section [X] of the [Title] Act [Year] |
Section 124 of the Employment Rights Act 1996 | Section, short title casing, 4-digit enactment year |
legislation.gov.uk |
| France (Civil Law) | Article [L/R/D].[Number] du [Code] |
Article L. 1235-3 du Code du travail | Article, hierarchical code prefixes (L/R/D), codified subject |
legifrance.gouv.fr |
| EU (Supranational) | Article [X] of [Directive/Reg] (EU) [Year]/[Number] |
Article 6 of Directive 2000/78/EC | Instrument types, year/sequential number | eur-lex.europa.eu |
| Spain (Civil Law) | Artículo [X] del [Ley / Real Decreto] |
Artículo 56 del Estatuto de los Trabajadores | Artículo, statutory instrument categories |
boe.es |
"Section 124 of the Employment Rights Act 1996" has zero overlap with "Article L. 1235-3 du Code du travail" or "Artículo 56 del Estatuto de los Trabajadores". The syntactic markers (Section vs Article vs Artículo, Act [Year] vs Code du [subject] vs del [instrument]) are orthogonal.
A formal citation is a zero-entropy primary key. When a query contains one, triggering multi-tool exploration is an architectural defect.
2. The Probe
Submitted query
"As at 1 January 2012, what was the maximum compensatory award for unfair dismissal under section 124 of the Employment Rights Act 1996?"
Ground truth
The applicable cap on 1 January 2012 was £68,400, set by the Employment Rights (Increase of Limits) Order 2011 (SI 2011/3006). The subsequent increase to £72,300 took effect on 1 February 2012, and the increase to £74,200 took effect on 1 February 2013.
This probe was part of a 14-query sealed test battery. Probes and ground truth were cryptographically committed prior to execution.
- Probes:
sha256:08c3b9b5fd9b5274c152738ed23c2d8faa3cb451273ba6bf34a9c5ca3a2d5964 - Ground truth:
sha256:0b249b5b4752d5ae9b860463d476731a20b05da279a3ea124f4ba705186b4316 - Sealed:
2026-08-07T08:51:57Z - Harness:
legal-rag-auditv0.1.0, commit0ca28a1fba7f3ca409047f60851f875b5ad2b7bb
3. Forensic Trace. Five Phases of Agentic Thrashing
The following is a step-by-step dissection of the thinking events and tool execution logs from the system's response to probe pit-era-124-1.
sequenceDiagram
autonumber
actor User
participant Agent as Autonomous ReAct Agent
participant FRCap as French/EU Legal DB
participant UKCap as UK Legislation DB
participant Web as Ungrounded Web Search
User->>Agent: "As at 1 Jan 2012, max compensatory award s.124 ERA 1996"
Note over Agent: Phase 1. Prior memory bias: "I recall it was £72,300..."
Agent->>FRCap: Broad query dispatch without metadata pre-filter
Agent->>UKCap: Broad query dispatch
FRCap-->>Agent: Returns 57+ French Labor Code articles and case law
UKCap-->>Agent: Returns scattered statutory fragments
Note over Agent: Phase 2. Confirmation bias query loop injects "£72,300"
Agent->>UKCap: Query: 'site legislation.gov.uk ... unfair dismissal £72,300'
Agent->>UKCap: Direct tool call: uksi/2011/3006 (contains £68,400)
UKCap-->>Agent: Returns SI 2011/3006 text (Event 6: deduces £68,400 is correct)
Note over Agent: Phase 3. Context saturated by 117 fragments, doubts SI 2011/3006
Agent->>Web: Searches 17+ Gov UK PDFs, impact assessments and consultations
Web-->>Agent: Returns 2012-2016 consultation papers and uplift notices
Note over Agent: Phase 4. Temporal inversion (Event 14: flips to £72,300)
Agent-->>User: "The maximum award was £72,300..." (wrong by £3,900)
3.1 Phase 1. Prior Memory Bias
The agent did not begin by querying a structured statutory table. It opened with an unanchored memory retrieval:
"I recall it was £72,300 during 2011–12, with the increase to £74,200 taking effect in February 2013, so I'll verify the relevant legislation and date."
This unverified anchor biased downstream retrieval.
3.2 Phase 2. Unfiltered Multi-Corpus Contamination
The system lacked a hard {jurisdiction: "GB"} metadata pre-filter. The agent broadcast queries across all active enterprise namespaces, retrieving 117 document fragments across 6 search calls.
Retained fragments included:
- 57+ French legal sources:
Arrêté du 23 novembre 1987 relatif à la sécurité des navires,Loi n° 2014-891 du 8 août 2014 de finances rectificative,Article L1235-3 du Code du travail,Cour de cassation, Chambre sociale, 8 septembre 2021, n° 19-20.538, and others. - Foreign common law:
Ervin Dean v Bahamas Power & Light (Bahamas).
Over 57 irrelevant foreign legal spans entered the context window for a UK statutory question.
3.3 Phase 3. Confirmation Bias Query Loop
Having biased itself toward £72,300, the agent dispatched search queries explicitly seeking confirmation:
site legislation.gov.uk uksi 2010 2011 Employment Rights Increase Limits unfair dismissal compensatory award £72,300 effective date"Employment Rights (Increase of Limits) Order 2011" section 124 2012 1 February 2012 72,300"section 124" "£72,300" unfair dismissal 1 February 2012 compensatory award
The target figure was embedded directly into the search constraints.
3.4 Phase 4. Near-Hit at Event [6]
At step 8 of the trace, the system called the statutory instrument (uksi/2011/3006) containing the true limit of £68,400.
In thinking event [6] (Note 1), the agent deduced the correct statutory limit:
"I need to verify the historical limit applying on 1 January 2012. The limit was £68,400 from 1 February 2011, with an increase to £72,300 taking effect on 1 February 2012, so the applicable maximum on the stated date was £68,400."
The deduction was accurate and verified.
However, the context window was already saturated with 117 fragments from French codes and ungrounded search hits. The agent experienced retrieval dissonance: rather than binding the £68,400 figure from SI 2011/3006, it questioned its own deduction and dispatched further web queries.
3.5 Phase 5. Inversion Collapse at Event [14]
The subsequent search wave ingested 17 external PDF documents, including impact assessments and consultation documents (13-575-unfair-dismissal-compensatory-awards-impact.pdf, 12-1038-unfair-dismissal-compensatory-awards-impact.pdf).
Confronted with conflicting references to multiple annual uplifts, the synthesis prompt collapsed. In thinking event [14] (Note 2), the agent reversed its position:
"I need to identify the statutory limit in force on 1 January 2012 under section 124 of the Employment Rights Act 1996. The applicable maximum compensatory award was £72,300, so I'll give that figure directly."
The system emitted:
"As at 1 January 2012, the maximum compensatory award for unfair dismissal under section 124 of the Employment Rights Act 1996 was £72,300."
This was wrong by £3,900. The applicable figure was £68,400 under SI 2011/3006.
4. Execution Telemetry
Three-pass battery results for probe pit-era-124-1. Each pass submitted the identical query to the same system endpoint.
| Pass | Total Latency | Thinking Duration | Thinking Events | Output Figure | Correct |
|---|---|---|---|---|---|
| Pass 1 | 61,275 ms | 57,829 ms | 21 | £68,400 | ✓ |
| Pass 2 | 27,661 ms | 23,567 ms | 8 | £72,300 | ✗ |
| Pass 3 | 21,053 ms | 17,951 ms | 8 | £72,300 | ✗ |
In the single-pass forensic trace where the retrieval inversion occurred (Section 3), the agent recorded 38,436 ms of thinking duration across 16 events out of 40,164 ms total latency.
The unconstrained web-client trace with web search active ran for 186,000 ms (3 minutes 6 seconds) across 15+ tool queries and 4 search waves.
Zero of 3 passes returned identical responses, scored as a response divergence finding.
Live on-screen verification of this 3-pass drift is documented in Video 1 (11:17–11:59). The contrast between Product A's abstention failure on fictional law versus System C's retrieval dissonance on held ground truth is analyzed in Video 2 (17:46–21:00).
5. Performance and Cost Comparison
| Dimension | Autonomous Agentic Tool Loop | Deterministic AST + Bitemporal DB | Engineering Delta |
|---|---|---|---|
| Execution Latency | 186,000 ms (unconstrained trace); API passes: 61.3s, 27.7s, 21.1s | <2 ms | 93,000× faster |
| Tool Calls | 15+ queries across 4 waves (117 fragments) | 0 LLM tool calls (1 direct DB hit) | Complete elimination |
| Context Noise | 57+ French articles/cases, 17 Gov UK PDFs | 1 exact row | 100% signal ratio |
| Token Load | ~28,500 tokens | ~120 tokens | 99.5% reduction |
| Multi-Pass Stability | Drifted: £68,400 → £72,300 → £72,300 | Bit-identical: £68,400 across all passes | Zero variance |
| Factual Output | £72,300 (wrong by £3,900) | £68,400 (correct, SI 2011/3006) | 100% precision |
6. Evidence Ladder
| # | Claim | Register | Source |
|---|---|---|---|
| 1 | Multi-jurisdiction citations carry orthogonal syntax markers across UK, FR, EU, and ES systems | By design |
Formal legislative drafting guidelines (Office of the Parliamentary Counsel UK, Légifrance Guide de Légistique, Manual de Técnica Legislativa BOE). |
| 2 | Autonomous agentic tool calling exhibits multi-pass factual divergence on point-in-time statutory queries | Measured |
3-pass audit on live system, 7 August 2026. Sealed sha256:08c3b9... and sha256:0b249b.... Pass 1 £68,400 (61.3s) vs Passes 2–3 £72,300 (27.7s / 21.1s). Demonstrated in Video 1 (11:17–11:59). |
| 3 | Unconstrained agent loops execute 15+ redundant tool calls taking >180s on zero-entropy statutory queries | Measured |
Live execution trace: 186s session, 117 fragments retrieved across 6 search calls on a single UK query. |
| 4 | Missing metadata pre-filters cause cross-jurisdiction contamination in mixed vector stores | Measured |
200-query collision run over ~1,400 Spanish civil statutes returning 100% incorrect statute codes under default HNSW search. |
| 5 | Deterministic AST citation extraction bypasses agent tool calling and eliminates retrieval latency | Verified in isolation |
Python regex/AST parser resolving UK/FR/ES citation patterns to structured metadata in <2ms across 500 test strings with 0 LLM tool calls. |
| 6 | Bitemporal database indexing guarantees bit-identical statutory version resolution for historical matter dates | By design |
SQL temporal constraints (WHERE valid_from <= '2012-01-01' AND valid_to > '2012-01-01') eliminate probabilistic temporal reasoning. |
7. Engineering Remediation. Four Deterministic Controls
flowchart TD
A["Raw User Legal Query"] --> B["Control 1. Syntactic AST Pre-Router<br/>Matches: 'Section 124' + 'Employment Rights Act 1996'<br/>Extracts: jurisdiction 'GB', type 'primary_act'"]
B --> C["Control 2. Hard Namespace & Metadata Pre-Filter<br/>Discards FR, ES, EU databases<br/>Locks scope to UK primary legislation"]
C --> D["Control 3. Bitemporal Parameter Binding<br/>Normalises 'as at 1 January 2012' to 2012-01-01<br/>SQL: valid_from <= '2012-01-01' < valid_to"]
D --> E["Control 4. Single-Hop Citation Binding<br/>Max tool calls = 0 or 1<br/>ReAct looping disabled, returns £68,400"]
Control 1. Deterministic AST Pre-Router
Bypass the LLM agent routing layer for formal citations. A regex parser detects unambiguous statutory patterns and extracts structured metadata:
import re
UK_STATUTE_PATTERN = re.compile(
r"(?:[Ss]ection|[Ss]\.?)\s*(?P<section>\d+[A-Za-z]?)\s+(?:of\s+the\s+)?(?P<act>[A-Z][A-Za-z\s]+Act\s+\d{4})"
)
def route_legal_query(query_str: str):
match = UK_STATUTE_PATTERN.search(query_str)
if match:
return {
"route": "DETERMINISTIC_STATUTE_LOOKUP",
"jurisdiction": "GB",
"act": match.group("act").strip(),
"section": match.group("section").strip(),
"allow_agent_loop": False
}
return {"route": "SEMANTIC_RAG", "allow_agent_loop": True}
Tested standalone across 500 test strings, resolving UK citation patterns to structured metadata in under 2ms.
Control 2. Hard Metadata Pre-Filter
Before any vector search or database query executes, inject mandatory metadata constraints:
{
"jurisdiction": "GB",
"corpus": "uk_legislation",
"instrument_type": "primary",
"short_title": "Employment Rights Act 1996"
}
This makes cross-corpus contamination (such as French labor codes or Bahamian case law) structurally impossible. A boolean index predicate cannot return a chunk from an unbound namespace.
Control 3. Bitemporal Point-in-Time Binding
Legal provisions change through secondary legislation. Annual Increase of Limits Orders modify monetary caps without amending the parent statute.
Do not allow the LLM to calculate temporal boundaries from unpinned chunk text. Bind the temporal anchor at the database schema layer:
SELECT statutory_cap, citation_si
FROM uk_statutory_limits
WHERE provision_id = 'ERA_1996_S124'
AND valid_from <= '2012-01-01'
AND valid_to > '2012-01-01';
-- Returns: £68,400 | SI 2011/3006
A temporal range constraint cannot return a future amendment.
Control 4. Tool-Budget Circuit Breaker
If a query resolves to a deterministic statutory key, enforce max_iterations = 1 on the agent orchestrator and disable multi-hop web exploration.
Bind citations to the retrieved database record. Do not allow the LLM to emit generative citation strings.
8. Honest Bounds
The AST pre-router requires formal citation structure. It does not solve:
-
Colloquial queries. Phrasing such as "What is the cap on unfair dismissal?" contains no structural fingerprint. These require constrained semantic classification with jurisdiction resolution, which is a separate classification problem.
-
Shared Commonwealth statute names. Statutes such as the UK Companies Act 2006 versus Singapore Companies Act 1967 share naming structures. Disambiguation requires secondary signals such as tenant jurisdiction context or restricted corpus scopes.
-
Directly applicable EU regulations. GDPR Article 82 applies concurrently in French and Spanish courts. The syntactic parser resolves the instrument to the EU corpus, but national implementing provisions require secondary lookup tables.
These boundary cases remain outside the scope of deterministic routing. The specification above addresses the query class where formal citations carry zero entropy, and where the measured system thrashed for 186 seconds regardless.
9. Reproducibility & Verification Data
The evaluation harness and test battery are open source.
| Artifact | Reference |
|---|---|
| Harness | legal-rag-audit v0.1.0, commit 0ca28a1fba7f3ca409047f60851f875b5ad2b7bb |
| Probe hashes | sha256:08c3b9b5fd9b5274c152738ed23c2d8faa3cb451273ba6bf34a9c5ca3a2d5964 |
| Ground truth hashes | sha256:0b249b5b4752d5ae9b860463d476731a20b05da279a3ea124f4ba705186b4316 |
| Seal timestamp | 2026-08-07T08:51:57Z |
| Private pre-disclosure | Raw audit bundle transmitted to System C engineering lead on 2 September 2026, prior to any public release |
Systems are anonymised as System A/B/C in public copy. Raw transcripts were pre-disclosed privately under the Publication Gate.
10. References and Notes
[1] Thinking Event [6]
Recorded in pit-era-124-1.single-pass.anonymised.json (order index 6, reasoning_summary, titled "Historical Compensation Limit"). Verbatim: "I need to verify the historical limit applying on 1 January 2012. The limit was £68,400 from 1 February 2011, with an increase to £72,300 taking effect on 1 February 2012, so the applicable maximum on the stated date was £68,400."
[2] Thinking Event [14]
Recorded in pit-era-124-1.single-pass.anonymised.json (order index 14, reasoning_summary, titled "Identifying the Award Limit"). Following context saturation from 117 retrieved fragments, the agent reversed its deduction: "I need to identify the statutory limit in force on 1 January 2012 under section 124 of the Employment Rights Act 1996. The applicable maximum compensatory award was £72,300, so I'll give that figure directly."
[3] Video Proof 1 (Live Multi-Pass Drift)
YouTube: Retrieval Disambiguation at 11:17–11:59 records the live 3-pass divergence on Section 124 ERA 1996 and initial diagnosis of agentic thrashing at 12:24–12:38.
[4] Video Proof 2 (Architectural Comparison)
YouTube: RAG Security Probes at 17:46–21:00 contrasts Product A's epistemic abstention collapse on non-existent law against System C's retrieval dissonance on held ground truth.
The Impact
Deterministic pre-routing eliminates agentic thrashing on formal statutory citations. Queries resolve in under 2ms with zero tool calls and zero factual drift across passes.
Run this measurement against your own system.
Retrieval Integrity Diagnostic: £500, fixed scope. Credited in full against the next stage.
What this costs