Source-linked AI summary
Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale
Liuyin Wang, Shuaipeng Jin, Jiwei Shi, Jensen Hsu
TL;DR
Normative-document QA depends on identifying effective versions, applicable scope, and source support—information that may not be in a single passage. This paper evaluates a governed system against a hosted retrieval service on a production-scale candidate set and reports a higher benchmark score for the governed system. It also releases benchmark materials and reproduction scripts.
Problem
Normative-document answers require identifying the effective version and confirming that the document applies to the relevant applicant, region, and date, which passage-level similarity alone cannot establish.
Source: Introduction · p. 1Method
The evaluation compares a hosted retrieval service with a governed system that resolves version and scope before generation on approximately 73,000 candidate documents using 200 questions with gold source documents.
Source: Introduction · p. 1, The Deployed Application · p. 3Results
97.7 overall for DeepKnown versus 88.1 for the hosted service, a 9.6-point gap computed from unrounded means.
Source: Introduction · p. 2Takeaways & Limitations
The paper releases the question set, expected answer points, gold sources, evaluated answer text, scores, and scripts for reproducing the benchmark statistics.
Source: Introduction · p. 2The comparison covers one question set, domain, corpus, and system pair, and the systems also differ in base model, ingestion, index, and answer template.
Source: Introduction · p. 2, Results · p. 7
Abstract
from arXiv · showhide
Correctly answering a question grounded in normative documents often depends on information outside any single passage: whether the retrieved document is the version currently in force; whether it applies to the jurisdiction, subject (such as an institution or applicant), and date at issue; and whether each normative claim can be traced to its supporting source text. Hosted retrieval services have substantially lowered the engineering cost of building an initial system over such corpora, making "upload the documents and ask" a common default. We evaluate this default on approximately 73,000 candidate normative documents supplied to a production deployment. The evaluation uses a stratified sample of 200 questions from our published benchmark, with a gold source document for every question; the released sampling rule reads no system outputs or scores. We compare the hosted service with a governed system that resolves version and scope through explicit rules before generation. The governed system scored 97.7 overall, while the hosted service scored 88.1, a gap of 9.6 points computed from unrounded means. The question set, the answer text evaluated for both systems, the scores, and the scripts used to reproduce the reported benchmark statistics are public. The governed configuration has operated as a commercial product since January 2026 and serves 1,126 registered users; named customer organizations include Zhipu AI and Lecheng Health. By mid-April 2026, it had reached roughly 100,000 calls per workday.
Introduction
Normative-document QA requires more than finding similar text: the system must identify the effective version and confirm that its scope covers the applicant, region, and date. The paper compares a governed system with a hosted retrieval service on the production candidate set and reports benchmark results and reproducibility materials.
Introduction
- Correct answers require confirming that the document is currently effective and that its scope covers the applicant, region, and date.An incorrect normative statement may be acted on and corrected only after a deadline has passed.Source: Introduction · p. 1
- Hosted retrieval lowers the engineering cost of an initial system, but its default performance on an operator’s production-scale candidate set is the question evaluated here.The hosted-service default can provide grounded answers with citations without requiring the team to implement several retrieval components.Source: Introduction · p. 1
- 97.7 overall for DeepKnown versus 88.1 for the hosted service, a 9.6-point gap computed from unrounded means.The comparison uses 200 questions and approximately 73,000 candidate normative documents supplied to the deployment.Source: Introduction · pp. 1–2
- The evaluation compares two deliverable systems that differ in governance as well as retriever, generator, and index.Questions and expected answer points were rule-guided and AI-assisted, grounded in original evidence, and reviewed by the business owner.Source: Introduction · p. 2
- The governed configuration had reached 1,126 registered users and roughly 100,000 calls per workday by mid-April 2026.The paper distinguishes these operational records from benchmark answer-quality measurement.Source: Introduction · p. 2
- The benchmark releases questions, expected answer points, gold source documents, both systems’ answer text and scores, judge rationales, the sampling rule, and reproduction scripts.The sampling rule reads no system outputs or scores.Source: Introduction · p. 2
The Deployed Application
The deployed application supports staff who must give traceable answers about normative documents during conversations or in written service work. It filters for effective versions and applicable scope before generation, and abstains when admissible evidence is insufficient.
The Deployed Application
- The service supports window and hotline staff answering in a conversation, as well as consultants and caseworkers preparing written replies or implementing services.Both groups are answerable for what they say while relying on materials they did not write; the 200 benchmark questions involve 191 distinct source documents.Source: The Deployed Application · p. 2
- The service’s workday call volume rose from roughly 10,000–20,000 per day in January to roughly 100,000 by mid-April 2026.The monitored record covers January 1 to April 19, and a call is a service invocation rather than necessarily a distinct end-user question.Source: The Deployed Application · p. 2
- Operational records document live use and call volume, while answer quality is measured separately on the same 200 questions and candidate corpus for both systems.The operator reports lower token use during knowledge interpretation, but the comparison baseline and measurement rule are not reported.Source: The Deployed Application · p. 2
- The governance layer filters retrieved candidates for version and scope, confines generation to admissible evidence, and links each assertion to its supporting clause.Version awareness concerns whether a clause belongs to an effective version; scope awareness concerns jurisdiction, applicant category, and time period.Source: The Deployed Application · pp. 2–3
- When evidence is insufficient to determine an answer, the system abstains and states what information is missing.This avoids substituting a merely plausible figure for an evidence-supported answer.Source: The Deployed Application · p. 3
- A fluent answer with a seemingly real citation can still be wrong, and a service-window error may carry consequences for the person acting on it.The system therefore begins answers with a fixed notice directing readers to consult the cited original text.Source: The Deployed Application · p. 3
Requirement
Normative knowledge differs from general knowledge because its correctness depends on authoritative documents and whether their scope applies. The paper frames the task as selecting and preserving usable, traceable evidence for the current request.
Requirement
- Normative knowledge is human-created and codified within a domain, with formal documents providing the authoritative basis for understanding and action.The category includes service catalogs, procedures, product terms, manuals, standards, and other materials, not only laws or public policy.Source: Requirement · p. 3
- Normative correctness depends on authority, effective status and date, jurisdiction, subject, and document replacement or hierarchy, not textual meaning alone.Nearly identical wording can support different conclusions under different versions, authorities, and scopes.Source: Requirement · p. 3
- Similarity retrieval can find textually close passages without establishing that they are in force or applicable to the question.Normative QA instead requires passages whose scope matches the request and whose version and conceptual boundaries are accurate.Source: Requirement · p. 3
- A system must preserve complete documents, effective versions, and applicability boundaries, then exclude unsupported, outdated, or out-of-scope content from conclusions.Claims should remain traceable to the issuing authority, source document, provision, version, and applicable scope for verification and audit.Source: Requirement · p. 3
AI Approach and Design Rationale
DeepKnown represents version and scope constraints explicitly and applies them before generation. On the evaluated candidate corpus, it scored higher overall than the hosted service, while requiring additional schema, ingestion, and review work.
AI Approach and Design Rationale
- DeepKnown stores extracted issuer, date, document-type, lineage, and geography, time, and subject-scope attributes as typed fields for enforcement before ranking.Extraction is model-assisted and human-reviewed; segmentation follows article structure so chunks inherit coherent attributes.Source: AI Approach and Design Rationale · p. 4
- Hard predicates prevent a high-similarity but out-of-scope document from remaining admissible, though extraction errors can remove the correct document and reduce recall.Filters are relaxed one dimension at a time; if admissible evidence still cannot be assembled, the system does not give an unsupported conclusion.Source: AI Approach and Design Rationale · p. 4
- Version lineage is stored as a field rather than chunk text, allowing a new supersession relation to update one edge instead of re-vectorizing an entire lineage.Attributes exclude documents only above a confidence threshold and otherwise adjust ranking; the approach carries schema-maintenance and extraction costs.Source: AI Approach and Design Rationale · pp. 4–5
- Generation begins only when the filtered candidate set satisfies the question’s version and scope constraints, making evidence sufficiency an external test rather than model self-assessment.The paper notes that the hosted service returned no output for six questions and returned uncited answers for two, all scored zero under the prespecified rules.Source: AI Approach and Design Rationale · p. 5
- 97.7 overall for DeepKnown versus 88.1 for the hosted service, a 9.6-point gap; DeepKnown received full credit on 189 of 200 questions and zero on none.The paper limits its claim to this domain and candidate-corpus scale, rather than claiming governance is always warranted.Source: AI Approach and Design Rationale · p. 5
Evaluation Methodology
The evaluation compares two deliverable systems on the same candidate corpus and questions, using answer-level scoring and a stratified sample selected without access to system outputs or scores.
Evaluation Methodology
- Both systems received the same approximately 73,000 candidate documents and 200 questions; the benchmark includes 106 simple, 81 complex, and 13 partial-answer questions.The set is a stratified random sample of 420 answerable benchmark items, with strata fixed before either system ran and selection independent of outputs and scores.Source: Systems, Corpus, and Controls · p. 5, Question Set · p. 5
- Questions and expected answer points were built from extracted source passages with business-owner review and checks for source, scope, current effectiveness, evidence, and format.The process used rule-guided, AI-assisted construction, programmatic checks, and human spot-checking and anomaly review.Source: Question Set · p. 5
Scoring and Rule-Assigned Records
- A deterministic scorecard assigns scores to predefined objective cases; remaining answers receive LLM-judge scores against expected points with partial credit.Eight of 400 records were directly assigned zero for empty returns or no citations; scoring used the same judge, temperature, formula, and rules for both systems.Source: Scoring and Rule-Assigned Records · p. 6
Evaluation Methodology
- The primary metric is the answer delivered to the user, because a relevant-file hit alone cannot establish a correct answer on version, scope, completeness, or citation consistency.Each question has a gold source document to support answer-level verification.Source: Why End-to-End Scores · p. 6
Systems, Corpus, and Controls
- Table 1 reports mean judge scores on a 0–100 scale for 200 questions over the candidate normative-document corpus, with the gap calculated as DeepKnown minus Gemini, the hosted service, from unrounded means.Source: Systems, Corpus, and Controls · p. 5, Table 1 · p. 6
Reproducibility
- The question set, per-question records, sampling rule, and reproduction script are public, enabling rescoring and recomputation of the reported benchmark statistics.Table 1 reports means and a gap computed from unrounded means.Source: Reproducibility · p. 6, Table 1 · p. 6
Results
DeepKnown scored higher overall and across all three question types. The results also show where the systems differ and limit interpretation to this whole-system comparison on one corpus and question set.
The measured difference
- 97.7 overall for DeepKnown versus 88.1 for the hosted service, a 9.6-point difference with a paired bootstrap 95% confidence interval of [5.7, 13.8].DeepKnown received full credit on 189 of 200 questions and zero on none; the hosted service received full credit on 166 and zero on 15.Source: Results · p. 6
Where the difference appears
- The hosted service scored 73.1 against DeepKnown’s 92.3 on partial-answer items, 86.6 against 96.8 on complex items, and 91.1 against 99.0 on simple items.All three confidence intervals exclude zero but overlap, so the results do not establish a reliable ordering of gap sizes.Source: Table 2 · p. 7, Results · p. 7
Anatomy of the gap
- The hosted service’s 15 zero-credit items comprised six empty returns, two uncited answers, and seven cited but substantively wrong answers.The cited wrong answers averaged 13 documents; examples include county-level figures inconsistent with city-wide targets and an incorrect subsidy amount.Source: Results · pp. 6–7, Table 3 · p. 7
Where the difference appears
- On partial-answer items, DeepKnown received full credit on 11 of 13 questions, compared with 6 of 13 for the hosted service.The authors suggest that these items require deciding what evidence is admissible, but identify this explanation as a hypothesis requiring per-item retrieval traces to test.Source: Results · p. 7
Scope
- The comparison covers one question set, document domain, corpus, and system pair; the systems differ in governance, model, ingestion, index, and answer template.The result supports a 9.6-point higher overall score for the governed configuration under this protocol, not an isolated effect of the governance layer.Source: Results · p. 7
Lessons Learned
The lessons emphasize resolving normative constraints before ranking and evaluating systems on the production corpus using the answers users receive. They also call for maintaining auditable question sets and separating accuracy from qualitative cost observations.
Lessons Learned
- Simple questions show a 7.9-point gap, complex questions a 10.3-point gap, and the governed configuration scores 99.0 on simple questions by enforcing version and scope constraints before ranking.The constraints are enforced at ingestion rather than left to the ranker or prompt.Source: Lessons Learned · p. 7
- End-to-end evaluation on the same candidate corpus and scoring protocol yielded a 9.6-point overall gap, with simple, complex, and partial-answer categories all pointing in the same direction.The comparison uses answers received by users rather than retrieval proxy metrics.Source: Lessons Learned · p. 7
- The hosted service’s 15 zero-point responses comprised 6 empty returns, 2 uncited answers, and 7 cited but substantively incorrect answers.The aggregate score treats all three outcomes as zero, although they correspond to different failure points.Source: Lessons Learned · p. 7
- Record output status, citation status, answer content, and the candidate set for each answer to help trace errors to retrieval, version or scope resolution, or generation.Source: Lessons Learned · p. 8
- Business-owner review and programmatic checks are both needed to keep normative-document question sets accurate and maintainable as the corpus changes.Checks cover source existence, applicability, current effectiveness, title conflicts, evidence, and format.Source: Lessons Learned · p. 8
- Lower token use during knowledge interpretation was reported during deployment growth, but the missing comparison baseline and measurement rule leave it qualitative.The benchmark does not isolate the cause of the observation.Source: Lessons Learned · p. 8
Related Work
Prior work addresses legal-text reasoning, evidence support, and hosted retrieval, but does not report the same production-scale, end-to-end comparison. This paper positions its contribution in explicit normative constraints and measurement of the answer users receive.
Related Work
- Legal and policy benchmarks cover reasoning and clause extraction, but do not isolate whether answers are version-aware and scope-aware for administrative normative documents.The cited examples include LegalBench, CUAD, LawBench, and LexEval (Li et al. 2024).Source: Related Work · p. 8
- Unlike RAG evaluations of retrieval relevance, answer faithfulness, or statement-level citation support, this system filters candidates before generation and evaluates final answers against gold sources.Per-question records preserve answer text, citation count, score, and judge rationale; common-format retrieval traces are unavailable for comparing the systems.Source: Related Work · p. 8
- Hosted retrieval studies and case reports do not provide end-to-end answer quality for a hosted service on an operator’s production-scale candidate document set.Prior evidence includes retrieval-reading studies and a qualitative failure taxonomy.Source: Related Work · p. 8
- 97.7 overall for the governed system versus 88.1 for the hosted service; the whole-system comparison differs simultaneously in model, index, prompting, and governance.The paper identifies typed version, scope, and effective-status fields enforced as predicates before ranking as its innovation.Source: Related Work · p. 8