Source-linked AI summary
The Vocabulary Gap Is an Equity Gap: Register Mismatch in Retrieval Systems for Public-Benefits Access
Krish Sapru
TL;DR
Public-benefits RAG systems retrieve from agency-written documents, while intended users may ask in plain or informal language, leaving retrieval across registers important to assess. The paper compares paired-register queries on fixed needs and gold passages, finding a large cross-retriever Recall@5 gap and testing a transparent lexicon bridge.
Problem
Public-benefits RAG evaluation often uses agency-like queries even though intended users may ask in plain or informal language, leaving user-register retrieval inadequately assessed.
Source: §7 · p. 3Method
The paper compares paired formal- and plain-register queries for the same needs and gold passages across lexical and term-graph retrieval, then tests an inspectable plain-to-formal lexicon bridge.
Source: §1 · p. 1, §5 · p. 2, §6.4 · p. 3Results
Across three retrievers, Recall@5 is 96–100% for formal queries but 36–44% for plain queries.
Source: §6.1 · p. 3Takeaways & Limitations
Benefits-access systems should report the register gap before deployment, and transparent lexicon bridges are presented as a governance-compatible design choice for high-stakes retrieval.
Source: §7 · p. 4The modest, authored benchmark cannot establish population-level prevalence, and its plain queries do not fully capture diverse inputs; all evaluated retrievers are lexical.
Source: §8 · p. 4
Abstract
from arXiv · showhide
Retrieval-augmented question answering is increasingly used to help people navigate public-benefits eligibility, yet the documents these systems retrieve from are written in agency register while intended users often ask questions in plain, informal, or non-native English. We show that this register mismatch can turn a high-performing retrieval system into an inequitable one. We construct a controlled benchmark of 51 publicly documented federal benefit-eligibility rules and 25 information needs, each phrased in both agency register and plain user register while keeping the gold passage fixed. Across BM25, TF-IDF, and a term-graph retriever, formal-register evaluation is nearly perfect (Recall@5 96-100%), but plain-register retrieval collapses (Recall@5 36-44%). For BM25, Recall@1 falls from 84% to 16% and Recall@5 from 100% to 44%, a 56-point equity gap on identical information needs. We trace the mechanism to a measurable vocabulary gap: formal queries share 0.63 of their content terms with the gold passage, while plain queries share only 0.11, a 5.9x reduction. A deliberately simple, auditable plain-to-formal lexicon bridge recovers much of the failure, lifting plain-query BM25 Recall@5 from 44% to 80%. The contribution is not a new retriever; it is an evaluation protocol, benchmark, mechanistic diagnosis, and transparent mitigation for a high-stakes social-impact failure mode that standard retrieval evaluation hides.
1 Introduction
Public-benefits access depends on people finding and understanding information written in agency language, while intended users may ask in plain or informal language. On identical information needs, standard retrievers perform well for agency-register queries but poorly for plain-register queries.
1 Introduction
- Agency-only evaluation can miss failures for intended users because benefit rules use controlled agency language while people may ask in plain, informal terms.Examples include asking about food stamps while working part time or help when heat is about to be shut off.Source: §1 · p. 1
- On identical information needs, three standard retrievers perform well in agency register and poorly in benefit-seekers’ register.For BM25, formal Recall@5 is 100% and plain Recall@5 is 44%.Source: §1 · p. 1
- The paper pairs a fixed information need and gold passage while varying only query register to measure the resulting equity gap.Source: §1 · p. 1
- The benchmark covers 51 public benefit-eligibility rules and 25 paired information needs across major federal benefit programs.Source: §1 · p. 1
- The gap appears across lexical, vector-space, and graph-based retrieval rather than being limited to one retrieval method.Source: §1 · p. 1
- A small, transparent plain-to-formal lexicon bridge is proposed to recover much of the lost plain-register Recall@5 without an opaque model or user data.Source: §1 · p. 1
2 Related Work
The paper connects retrieval research on lexical, dense, and graph-based methods with longstanding work on vocabulary mismatch and administrative burden. It frames register-sensitive retrieval as a social-impact evaluation problem, not solely an algorithm-optimization problem.
2 Related Work
- Sparse lexical, dense passage, and graph-based retrieval are established approaches, but evaluating with agency-like queries can overestimate quality for users who do not use agency language.Source: §2 · p. 2
- Vocabulary mismatch has long been studied in human-system communication and information retrieval, including query expansion, statistical translation, relevance feedback, and learned representations.Source: §2 · p. 2
- Administrative-burden research describes learning and compliance costs shifted to citizens, while public-service systems can reproduce or amplify those burdens when not accountable to affected communities.Source: §2 · p. 2
- AI-for-social-impact work emphasizes problem formulation, data collection, field evaluation, social significance, and engagement beyond computer science.This paper positions its benchmark and release-blocking metric within that framing rather than optimizing a new model in isolation.Source: §2 · p. 2
3 Problem Formulation
The formulation compares retrieval performance across formal and plain realizations of the same information needs, using the same gold passages. The formal-minus-plain difference captures the register gap that standard formal-only evaluation misses.
3 Problem Formulation
- The evaluation defines an information need with a gold passage set and realizes that need as a query in either formal or plain register.Source: §3 · p. 2
- The equity-relevant comparison computes both register metrics over the same needs and gold passages.Source: §3 · p. 2
- A system may have excellent formal-register performance yet a large formal-minus-plain gap, indicating it works in the lab but fails in deployment.Source: §3 · p. 2
4 Corpus and Benchmark
The benchmark combines a corpus of public federal benefit rules with paired formal- and plain-register queries for controlled information needs. Queries are authored rather than scraped from live logs to isolate register while avoiding collection of vulnerable users’ data.
4 Corpus and Benchmark
- The corpus contains 51 passages, each stating one publicly documented federal benefit-eligibility rule across benefits and cross-cutting procedures.It includes SNAP, health coverage, income support, housing, food programs, tax credits, recertification, hearings, verification, and mixed-status households.Source: §4 · p. 2
- Each of 25 information needs is phrased once in formal agency register and once in everyday plain register, with both queries mapped to the same gold passage or passages.Source: §4 · p. 2
- Authored query pairs are an ethical first-benchmark choice that isolates register without collecting potentially private, legally sensitive, or immigration-relevant live-search data.The authors characterize this as a lower-bound protocol, not a replacement for future IRB-governed, community-consented query collection.Source: §4 · p. 2
5 Methods
Methods compare standard retrieval systems on paired formal- and plain-register queries while holding each information need and gold passage fixed. They measure retrieval performance by rank and reciprocal rank, with the formal-minus-plain difference defining the equity gap.
5 Methods
- BM25 ranks tokenized passages, while TF-IDF ranks passages by cosine similarity between query and passage vectors.Both are standard lexical retrieval approaches.Source: §5 · p. 2
- The term-graph retriever seeds personalized PageRank with query terms and ranks passage nodes by their resulting mass.Its graph is designed to test whether corpus structure can help without a learned encoder; all evaluated retrievers are standard and unmodified.Source: §5 · pp. 2–3
- The paired evaluation reports Recall@1, Recall@3, Recall@5, and MRR@10 by register, defining the equity gap as formal minus plain.With 25 information needs, recall percentages change in four-point increments.Source: §5 · p. 3, Table 1 · p. 3
6 Results
Across three retrievers, formal-register retrieval performs strongly while plain-register retrieval falls sharply on the same information needs. The results connect this gap to lower query–passage vocabulary overlap and show that a simple lexicon bridge improves plain-query BM25 retrieval.
6.1 A large, consistent equity gap
- Formal Recall@5 is 96–100% and MRR is at least 0.817 across systems, while plain Recall@5 is 36–44% and Recall@1 is 12–24%.The graph retriever does not rescue plain queries and is weakest on them.Source: §6.1 · p. 3
6.2 Which needs are lost
- 14 of 25 BM25 information needs answerable in formal register are lost in plain register, including food-assistance, emergency-help, and tax-credit questions.The paper notes these losses can mean a family does not apply for a benefit it may be entitled to.Source: §6.2 · p. 3
6.4 A cheap, auditable remedy
- The hand-curated plain-to-formal lexicon bridge lifts plain-query BM25 Recall@5 from 44% to 80%.It recovers nine additional top-five successes among 25 needs; because the bridge and queries were authored by the same process, this result is an upper bound rather than a heldout estimate.Source: Table 2 · p. 3, §6.4 · p. 3
6.3 Mechanism: the vocabulary gap
- Formal queries share 0.63 of content terms with the gold passage, versus 0.11 for plain queries, a 5.9× reduction.The term-graph retriever inherits this blind spot because its graph is induced from the same agency corpus, explaining the gap across evaluated methods.Source: §6.3 · p. 3
7 Discussion
Agency-register evaluation can hide retrieval failures on plain user queries, while paired comparisons expose the gap and a transparent lexicon bridge recovers part of it. The discussion recommends measuring register disparities and making mitigation auditable.
7 Discussion
- A 56-point BM25 Recall@5 gap is partly recovered by a transparent lexicon bridge, which closes roughly 64% of the observed gap.The paired-register protocol holds the corpus, retriever, and gold passage fixed while varying query register.Source: Figure 2 · p. 4, Figure 1 · p. 4
- Benefits-access systems should report the register gap alongside aggregate retrieval quality and treat a large gap as release-blocking.The paper describes high aggregate performance alongside conditional accuracy for users who already speak institutional vocabulary.Source: §7 · p. 4
- A small, inspectable lexicon bridge is presented as a governance-compatible alternative to opaque query rewriting in high-stakes public-service retrieval.For example, a legal-aid lawyer can inspect a mapping from “food stamps” to SNAP.Source: §7 · p. 4
- The paired-register evaluation protocol may apply wherever institutions use controlled vocabularies and users use lived vocabularies.The paper names immigration assistance, tax credits, housing rights, disability accommodations, unemployment insurance, and student aid as possible domains.Source: §7 · p. 4
- Plain-register queries share far fewer terms with the correct agency passage, making vocabulary mismatch a measurable barrier.The paired design compares formal and plain queries for the same information need and gold passage.Source: Figure 3 · p. 4, Figure 1 · p. 4
8 Limitations and Ethics
The study’s benchmark and query phrasings constrain population-level claims, and its lexical retrievers leave dense retrieval untested. The illustrative bridge also requires careful governance, while retrieval success alone does not guarantee correct advice.
8 Limitations and Ethics
- The modest, authored benchmark does not establish population-level prevalence, and its plain phrasings approximate but do not capture the full diversity of user inputs.The passage calls for community-designed deployed benchmarks with informed consent and privacy review.Source: §8 · p. 4
- Because the evaluated retrievers are lexical, the study cannot establish whether dense retrieval narrows, preserves, or relocates the register gap.The open question is whether dense representations bridge the surface mismatch or shift the gap elsewhere.Source: §8 · pp. 4–5
- The illustrative lexicon bridge could encode regional, racial, or linguistic blind spots if maintained carelessly, and retrieval success alone does not guarantee correct benefit advice.The paper recommends versioning, public documentation, subgroup-relevant evaluation where ethical, and co-design with legal-aid or benefits navigators.Source: §8 · p. 5
9 Conclusion
The conclusion argues that agency-register evaluation can make retrieval systems appear reliable despite failures on plain-language queries. It identifies a vocabulary-overlap gap, partial remediation through auditable query expansion, and measuring the register gap before deployment as the core lesson.
9 Conclusion
- Across the evaluated lexical retrievers, the failure is large, explained by a 5.9× vocabulary-overlap gap, and partly remediable by cheap, auditable query expansion.The conclusion’s procedural lesson is to measure the register gap before deployment.Source: §9 · p. 5