MEMON SYSTEMS

legal-rag-router: Deterministic Citation Routing for UK Legal RAG, Measured Against Gemini, Jev and Laya

legal-rag-router is an open-source Python library. It reads the legal citation in a query, checks it against an index of the UK statute book, and returns one decision before any vector search runs. The decision is a coordinate to retrieve from, a question for the user, or a refusal.

Version 0.1.0 covers UK legislation. It is on GitHub and PyPI, with a summary on /tools.

Summary

All figures are on 2,112 labelled queries, index snapshot 28 September 2026.

  • Measured, blind sealed run, 29 September 2026. 0 of 1,897 real citations bound to the wrong instrument. 0 of 68 invented Acts bound. 82 of 1,030 held-out real citations refused.
  • Measured, 1 October 2026. Gemini 3.8 Flash at the high thinking level, routing on its own, bound the wrong instrument for 102 of 1,897. Used as a citation parser in front of the router's index, it was the most accurate configuration measured, with 0 wrong bindings.
  • Measured, 30 September 2026. Router p50 0.14 ms, p99 2.10 ms on an Apple M4. The p99 target of 2 ms is missed.

The problem

Vector search returns the text nearest to the query. It does not return the provision the query cites. Asked about an Act that does not exist, it still returns something.

Two measurements on this site show the result.

  • Measured. Spanish statutory law, April 2026, black-box probes against a live public interface. 3 of 3 article-collision probes returned the wrong statute. Three probes show reproducibility, not a rate. Case study.
  • Measured. A commercial multi-jurisdiction system, 7 August 2026, one UK query run three times. The agent made 15 or more tool calls over 186 seconds, retrieved 117 fragments, 57 of them French labour code articles, and returned £68,400 once and £72,300 twice. £68,400 was correct. Case study.

In both cases the query named the law. The failure was in what retrieval was allowed to search.

The router addresses this class of failure for UK legislation only. It does not cover Spanish law, and it has not been run against either audited system.

What it does

One call returns one result. The result carries a status and a next action for the pipeline.

"What is the cap in section 124 of the Employment Rights Act 1996?"
  ROUTE_BOUNDED  uk/ukpga/1996/18/s124            retrieve only from here

"section 3 of the Digital Privacy Rights Act 2021"
  EPISTEMIC_ABSTENTION_INSTRUMENT_NOT_FOUND       refuse: no such Act
What it does — Status, Next action, When (Data Table)
Status Next action When
ROUTE_BOUNDED Retrieve, filtered to the coordinate The citation resolves
ROUTE_AMBIGUOUS Ask the user "the Companies Act" matches several Acts
EPISTEMIC_ABSTENTION_INSTRUMENT_NOT_FOUND Refuse The Act or SI is not in the statute book
EPISTEMIC_ABSTENTION_PROVISION_NOT_FOUND Refuse The Act exists, the section does not
ROUTE_OUT_OF_COVERAGE Declare out of coverage Scottish, Welsh, NI, EU or retained EU law
ROUTE_UNRESOLVED Discover, then bind The query cites nothing

A bound result produces a partition filter. The filter restricts vector search to the bound instrument and provision. For any other status the filter function raises an error, so there is no unbounded filter to fall back to.

A query that cites nothing goes to discovery. Discovery offers candidate provisions for the user to confirm. It never binds on its own.

By design, these properties follow from the architecture:

  • No model, no network call, no disk read per query.
  • The same input returns the same output.
  • Zero runtime dependencies. Standard library only.
  • Every index file is checked against a SHA-256 hash at load. A damaged index fails at load, not at query time.
  • The index holds coordinates, titles and lookup tables. It holds no provision text. Your store keeps the text.

Architecture

The router runs inside your process. Every query takes one pass through the steps below. A step that cannot complete safely returns ROUTE_UNRESOLVED. No step guesses.

flowchart TD
    Q["Query text"]

    subgraph ROUTER ["legal-rag-router: in process, no model, no network    "]
        PRE{"Precheck<br/>text, under 4 KB, UK in scope"}
        SIG{"Citation signal?<br/>normalise, prefilter"}
        SCAN["Scan<br/>titles, abbreviations, chapter and SI numbers,<br/>pinpoints, ranges, exclusions, date cues"]
        CAP{"Within size and<br/>complexity limits?"}
        LINK["Link provisions to instruments<br/>apply exclusions and prior-turn context"]
        RES["Resolve each citation against the index<br/>typo tiers: correct, ask or refuse"]
        IDX[("Index, snapshot 28 Sep 2026<br/>134,219 instruments, coordinates and lookup tables<br/>memory-mapped, SHA-256 verified at load")]
        OUT["Assemble one result<br/>status, next action, coordinates,<br/>corrections, temporal hint, repealed flag"]
        PRE -->|pass| SIG
        SIG -->|yes| SCAN
        SCAN --> CAP
        CAP -->|yes| LINK
        LINK --> RES
        IDX -.-> RES
        RES --> OUT
    end

    BOUND["ROUTE_BOUNDED"]
    AMB["ROUTE_AMBIGUOUS"]
    NF["INSTRUMENT_NOT_FOUND<br/>PROVISION_NOT_FOUND"]
    OOC["ROUTE_OUT_OF_COVERAGE"]
    UNR["ROUTE_UNRESOLVED"]

    subgraph PIPE ["Your pipeline"]
        FILTER["partition_filter<br/>instrument_id and coordinate"]
        VS[("Your vector store<br/>holds the provision text")]
        GEN["Generation"]
        ASK["Ask the user<br/>candidates and a clarification question"]
        REF["Refuse<br/>states the snapshot date"]
        DECL["Declare out of coverage"]
        DISC["discover<br/>concept index offers candidate provisions"]
        CONF["User confirms one<br/>routed again as a new query"]
        FILTER --> VS
        VS --> GEN
        DISC --> CONF
    end

    Q --> PRE
    PRE -->|fail| UNR
    SIG -->|no| UNR
    CAP -->|no| UNR
    OUT --> BOUND
    OUT --> AMB
    OUT --> NF
    OUT --> OOC
    OUT --> UNR
    BOUND --> FILTER
    AMB --> ASK
    NF --> REF
    OOC --> DECL
    UNR --> DISC

By design:

  • The only path to generation runs through ROUTE_BOUNDED and the partition filter. The filter function raises an error for every other status.
  • Every chunk in your vector store carries instrument_id and coordinate. The filter matches on those two fields.
  • Limits on query length, citation count and lookups keep routing time linear in the query length. A query past a limit returns ROUTE_UNRESOLVED.

Coverage

  • 134,219 UK Public General Acts and Statutory Instruments, bound to provision level.
  • Index snapshot 28 September 2026. Every refusal states that date.
  • Citation forms: full and short titles, abbreviations, chapter and SI numbers, pinpoints, ranges, exclusions, typos.
  • Recognised and reported out of coverage: Scottish, Welsh and Northern Ireland legislation, local Acts, EU and retained EU law.
  • Not covered in 0.1.0: Spanish law. It is on the roadmap.

How it was measured

The test batteries were hashed and tagged before the router ran on them. The router ran once. The results were hashed after. The seals are in the repository.

  • 2,112 labelled queries across nine batteries.
  • 1,030 of them are real citations taken from legislation text and held out from development.
  • v1 is the blind run, 29 September 2026. v2 reran after fixes to failures v1 exposed, so v2 is not blind on the hand batteries. Both are published. This article uses v1 unless it says otherwise.

Results

Measured. Sealed run v1 (blind), 29 September 2026, 2,112 queries, index snapshot 28 September 2026.

Results — Measure, Result (Data Table)
Measure Result
Real citations bound to the wrong instrument 0 / 1,897 (95 % upper bound 0.16 %)
Held-out real citations bound wrongly 0 / 1,030
Invented Acts bound 0 / 68
Same section number in different Acts, bound to the wrong Act 0 / 40
Real law refused 84 / 1,897 (v2: 82)
Rows with the expected outcome 1,903 / 2,112 (v2: 1,916)

Measured. Latency, 30 September 2026, full UK index, 306,336 calls per pass, 3 passes.

Results — Machine, p50, p99 (Data Table)
Machine p50 p99
Apple M4 0.14 ms 2.10 ms
Intel i5-3570 (2012) 0.50 ms 9.21 ms

Measured. 100,000 real citations taken from the text of UK legislation, index snapshot 28 September 2026. These citations were used to find and fix errors during development, so the router was partly tuned on them. The held-out results above use a separate set it never saw.

Results — Outcome, Share (Data Table)
Outcome Share
Bound correctly 80.02 %
Missed 8.54 %
Refused 6.33 %
Asked about 4.37 %
Bound to another instrument 7 citations

How it compares

Measured. Run 1 October 2026, scored 2 October 2026, with the sealed run's own scoring code. Fed the router's answers, that code reproduces the v2 run exactly.

  • Gemini 3.8 Flash at the high thinking level, through Google AI Studio. It was given the router's output contract, coverage, snapshot date and nine worked examples that do not overlap the battery. Its answer was read two ways. As a parser, it lists the citations and the router's index resolves them. As a router, it returns the outcome and coordinates on its own, from memory.
  • Jev (typesafe/jev-1.13-20260917), through OpenRouter's Decisions API. Jev is a decision model, so it was asked choice questions only.
  • Laya (421M base checkpoint), self-hosted. Speed and footprint only. Its accuracy is not reported, for the reason under Choice questions.

Routing the battery

Same 2,112 queries for every column. Router v1 is the blind run. Router v2 is the code that ships as 0.1.0, rerun after fixes to v1's failures, so it is not blind on the hand-written batteries. Gemini was run on the v2 battery and had not seen it.

Routing the battery — Measure, Router v1 (blind), Router v2 (0.1.0), Gemini parses, router resolves… (Data Table)
Measure Router v1 (blind) Router v2 (0.1.0) Gemini parses, router resolves Gemini alone
Real citations bound to the wrong instrument 0 / 1,897 0 / 1,897 0 / 1,897 102 / 1,897 (5.38 %)
Real law refused 84 / 1,897 82 / 1,897 55 / 1,897 233 / 1,897
Held-out real citations left unresolved 67 / 1,000 67 / 1,000 11 / 1,000 11 / 1,000
Invented law bound 0 / 68 0 / 68 0 / 68 0 / 68
Ambiguity battery, expected outcome 23 / 25 25 / 25 8 / 25 2 / 25
Typos bound that needed a question or refusal 1 / 22 0 / 22 4 / 22 3 / 22
Rows with the expected outcome 1,903 / 2,112 1,916 / 2,112 1,972 / 2,112 1,799 / 2,112
  • Gemini on its own bound the wrong instrument for about 1 in 19 real citations and refused about 1 in 8.
  • Gemini as a parser in front of the router was the most accurate configuration measured. It resolved 116 real-document rows the router left unresolved or refused, and lost 8 the router got right. Most were bare SI numbers, multi-citation sentences and schedule paragraphs.
  • That configuration bound nothing wrong and nothing invented, because every coordinate came from the router's index.
  • On ambiguous citations the router asked. Gemini picked one instrument.
  • Gemini refused all 68 invented citations, given a contract that told it how.

Choice questions

The right instrument is always among the options, each labelled with its title and citation, plus "none of these". This is the ceiling with a retriever that never misses. 2,050 questions per option count, instrument level only.

Choice questions — Options, Jev correct, Gemini correct (Data Table)
Options Jev correct Gemini correct
3 2,031 / 2,050 (99.1 %) 2,037 / 2,050 (99.4 %)
10 2,033 / 2,050 (99.2 %) 2,038 / 2,050 (99.4 %)
30 2,023 / 2,050 (98.7 %) 2,035 / 2,050 (99.3 %)

Laya is not in this table. Its vendor documents the base checkpoint as near chance without fine-tuning: 0.362 on the vendor's own typed-decisions benchmark, against 0.727 for Jev. Its only fine-tuned checkpoint is built for four synthetic workflows outside law, and its model card says to expect base-level results or worse elsewhere. No legal fine-tune of Laya exists. Scoring the base checkpoint would measure a use its vendor does not support. A legal fine-tune is future work, and it would need training data kept apart from the sealed batteries.

Both models choose well when the right instrument is offered. The open question is getting it into the options. Laya, at 30 options, cuts each title to 5 tokens and sees a median 29 % of it. "The Welfare Reform Act 2012 (Commencement No. 9, 21 and 23 (Amendment)) Order 2018" reaches the model as "The Welfare Reform Act". It raises no warning.

Speed

Measured. Time per call. The router and Laya were measured on 30 September 2026, Jev and Gemini on 1 October 2026. Laya ranges cover questions with 3, 10 and 30 options.

Speed — System, Where it ran, Calls, p50… (Data Table)
System Where it ran Calls p50 p99
Router Apple M4, in process 306,336 × 3 0.14 ms 2.10 ms
Router Intel i5-3570 (2012), in process 306,336 × 3 0.50 ms 9.21 ms
Router Modal, 1 CPU core, us-east, time inside the container 6,336 0.63 ms 3.65 ms
Router Same Modal service, round trip from Pakistan 6,336 276.6 ms 364.5 ms
Laya NVIDIA T4 GPU on Modal 900 34 to 37 ms 44 to 46 ms
Laya Apple M4 GPU 900 50 to 81 ms 88 to 117 ms
Laya Apple M4 CPU 900 89 to 139 ms 137 to 170 ms
Laya Intel i5-3570 CPU 900 1,165 to 1,987 ms 2,336 to 2,706 ms
Jev OpenRouter, from Pakistan 287 507 ms 2,222 ms
Gemini 3.8 Flash (high) Google AI Studio, from Pakistan 300 13.5 s 192 s
  • Router on Modal. The round trip is almost all network. A call with no routing on the same connection took 275.7 ms at p50. The router's answers matched the local router on all 6,336 calls.
  • Router inside the hosted-call pass. With caches cold after each network wait, its p50 on the M4 was 1.18 ms.
  • Laya. On its vendor's reference GPU, the T4, it took 34 to 37 ms. The vendor publishes 33 to 40 ms. Without a GPU it is 89 ms or more per call.
  • Jev. A call with no model on the same connection took 103 ms at p50.
  • Gemini. The time is the model's own. 22 of 300 calls took over a minute. The slowest took 249 s and 32,497 thinking tokens, on a dense amendment-note citation.

Cost, repeatability, footprint, data

Cost, repeatability, footprint, data — , Router, Laya, Jev… (Data Table)
Router Laya Jev Gemini 3.8 Flash (high)
Answer changed across 5 repeats, 200 rows 0 Not measured 2 / 197 34 / 200
Cost per 1,000 queries, list price $0 GPU time $0.041 $8.76
Install, plus model or index 0.6 MB, plus 362 MB index 693 MB, plus 807 MB weights Hosted Hosted
Query text leaves your network No No Yes Yes

Gemini's cost is the promotional price valid to 31 December 2026. The regular price is $17.52 per 1,000.

What was not controlled

  • One run, at one time of day, from one client in Pakistan. No off-peak run and no UK or EU client.
  • Gemini's figures hold for the high thinking level only.
  • Choice-question results are instrument level only, with the right answer always offered.
  • Run cost: Gemini $37.78 at the promotional price, Jev $0.30.

By design, not measured. Call the LLM only on queries the router leaves unresolved or refuses. In this battery that is 258 of 2,112 queries, 12 %. The router's ROUTE_UNRESOLVED status already routes to that step.

Every call, the seals and the scoring code are in the comparison report.

Where it fails

  • Real law is refused. The router refused 82 of the 1,030 held-out real citations. Most come from amendment notes in legislation and cite a section that has since been repealed, so it is no longer in the current statute book. The router refuses a provision it cannot find in the current text instead of binding the nearest one. That is a design decision. A question about the law before the repeal needs a point-in-time source.
  • Citations are missed. 8.54 % of the 100,000 development citations. The commonest missed shape is a bare SI number such as 1999/3434, which the router does not bind on purpose.
  • Some citations bind to the wrong instrument. 7 of the 100,000 development citations. None of the 1,030 held-out citations.
  • The latency target is missed. The design target was p99 under 2 ms. Measured p99 is 2.10 ms on the M4 and 9.21 ms on the i5. The tail is refusals, which rank suggested titles.
  • "As at" dates are not read. Verified in isolation, 2 October 2026: "As at 1 January 2012" binds the right provision but returns no temporal_hint. "As it stood on" and "as in force on" are read. "As at" and "as of" are planned for the next release.
  • Discovery is limited. Measured, sealed test run, 29 September 2026, 91 drafted queries: the right provision is in the top 10 for 88 % and first for 51 %. For the colloquial query "What is the cap on unfair dismissal?" the top 3 did not include section 124 (Verified in isolation, 2 October 2026).
  • The index is a snapshot. Law made after 28 September 2026 is refused until the index is rebuilt. There is no live register check in 0.1.0.

What it does not do

  • It does not hold or return provision text.
  • It does not resolve a date to the version in force. It passes the date through for a point-in-time layer.
  • It does not check what the model writes. A bound coordinate limits where retrieval looks. It does not establish that the answer is correct.

Install

pip install legal-rag-router

Python 3.11 or later. The index is a separate download attached to each GitHub release, with a SHA256SUMS file. Licence: AGPL-3.0-only. Index data is derived from legislation.gov.uk under the Open Government Licence v3.0.

Source code, documentation, the sealed batteries and every result file: github.com/azterizm/legal-rag-router.

Run this measurement against your own system.

What this costs

More Articles You Might Like

New entries are published to the feed.

RSS — /feed.xml