Relevance is not enough: A Communication-Oriented Retrieval System for Consequential Scientific Question Answering

• In water-quality communication, a correct concentration alone may not explain regulatory limits, health effects, uncertainty, or available actions.Source: §1 · p. 1
• The system identifies evidential gaps to retrieve sufficient rather than merely relevant material, then separates evidence-based claims from audience-oriented formulation.Source: §1 · pp. 1–2
• About an order of magnitude less retrieved text accompanied higher human-rated completeness in some settings, while relevance explained about 1% of completeness variance.Source: §1 · p. 2
• The paper proposes human-centered completeness, evaluated through human annotations and model fine-tuning, as a criterion beyond relevance-based metrics.Source: §1 · p. 2

• Relevant documents can still be operationally beside the point in consequential science, leaving readers unable to understand a situation or act on it.Source: §2 · p. 2
• Prior consequential-domain systems emphasize accessibility, readability, and accuracy, but do not audit whether evidence is sufficient for understanding and action.Source: §2 · p. 2
• Existing adaptive retrieval approaches add reflection, confidence checks, sub-questions, or intermediate reasoning, while some supervised reasoning methods require costly annotations.Source: §2 · p. 2
• Completeness jointly captures informational coverage and transparent uncertainty, addressing limits of reference-based and relevance-focused evaluation for questions with multiple valid answers.Source: §2 · pp. 2–3

3 Methodology and System Design

• Community surveys and leader interviews informed a water-quality outreach pipeline that builds sufficient context, audits content, and articulates findings for laypeople.Source: §3 · p. 3

3.1 Constructing the knowledge base

• The knowledge base organizes authoritative water-quality, regulation, and health sources hierarchically, while 500 probe questions expand community and institutional FAQs.Source: §3.1 · p. 3

3.2 Retrieving content

• Retrieval first matches short queries to topic-focused headings, falls back to full text below a cosine similarity of 0.51, and tracks prior chunks across iterations.Source: §3.2 · p. 3

3.3 Progressively building sufficient context

• A prognostic audit removes low-utility documents and a diagnostic audit identifies evidential gaps; targeted retrieval fills remaining gaps for up to two iterations.The four strategies specify distinct evidence needs, and second-iteration relevant fetches replace irrelevant first-iteration chunks to keep context size stable.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

3.4 Curating responses

• A generator produces evidence-derived claims and states knowledge limitations, while a communicator rewrites them in persona-adapted language without changing factual content.The caring-neighbor and city-hall-translator personas use different tones and framing for uncertainty.Source: §3.4 · p. 4

4 Evaluation

• The evaluation co-designed completeness with community stakeholders to measure necessary information and transparent uncertainty, and annotated 648 generated answers across seven variants.Two annotators used a five-point Likert scale and achieved Cohen’s κ = 0.65.Source: §4 · pp. 4–5

4.1 Ablation study

• The ablation compared single-pass RAG, oracle prognostic filtering, implicit diagnostic retrieval, and explicit reasoning-guided diagnostic retrieval across matched query subsets.The four configurations were evaluated on 160 frequent community queries; retrieval metrics covered 89 queries and filtered-variant completeness covered 84.Source: §4.1 · p. 5

4.2 Results

• A 25-fold context reduction accompanied completeness rising from 0.60 to 0.711–0.751 across filtered variants, while answer relevance fell from 0.916 to roughly 0.65–0.70.The comparison uses the pipeline variants in Table 1.Source: §4.2 · pp. 5–6, Table 1 · p. 6
• Relevance scores explained little variation in human completeness ratings (R² = 0.014), while the fine-tuned judge aligned better with humans (ρ = 0.73) but had only moderate agreement (κ = 0.54).The reported association between completeness and answer relevance was ρ = 0.05; for context relevance it was ρ = 0.09.Source: §4.2 · p. 6
• In the explicit diagnostic check, the retrieval process resolved evidential gaps in 73.8% of cases, while the second pass introduced new gaps in 4 of 61 rerun queries (6.6%).Structured regulatory queries had 86.4% of gaps filled, versus 56.8% for causal queries. Filling evidential gaps did not yield downstream completeness gains.Source: §4.2 · p. 6
• The caring neighbor persona improved readability and emotional response, but persona conditions showed no significant differences in completeness, answer relevance, or context relevance.The caring neighbor produced emotional reactions in over 50% of responses; readability scores were µ = 51–52 versus µ = 41–42 for city hall translator and µ = 36 for no persona.Source: §4.2 · pp. 6–7, Figure 2 · p. 7

5 Discussion

• Reasoning-guided compression can reduce inference cost and latency and let interactions fit longer conversations, but diagnostic checks showed no significant final-completeness gain over prognostic filtering.The discussion notes that diagnostic checks detect and close evidential gaps despite that lack of a measured completeness gain.Source: §5 · p. 7
• Fine-tuning improved judge alignment, but the discussion recommends human validation of user-centered qualities before deployment.The proposed role for automated judges is screening large-scale systems during early development and reducing annotation workload.Source: §5 · p. 7
• Because relevance scores poorly reflect human completeness, the discussion argues that completeness should be evaluated independently of relevance in science communication.It describes complete responses as combining factual accuracy with the breadth needed for decision-making.Source: §5 · p. 7
• Personas altered emotional expression and readability without shifting completeness or relevance scores, supporting stylistic adaptation while preserving measured content quality.The discussion presents automated personas as a way to tailor information without labor-intensive manual editing.Source: §5 · p. 7

6 Conclusion

• The system combines reasoning-guided iterative retrieval with persona-based articulation to support communication of consequential scientific information.Its goal is to help laypeople interpret that information.Source: §6 · p. 7
• Completeness is introduced as a human-centered criterion that captures response qualities standard relevance-based metrics overlook.The paper also reports that gap-aware retrieval can substantially reduce context usage.Source: §6 · p. 7

Limitations

• Proprietary large language models are black-box components, preventing full characterization of the internal reasoning behind particular outputs.Alternative models or domain-specific fine-tuning could produce different outcomes.Source: Limitations · p. 8
• System components transfer across domains unevenly, and water-quality diagnostic questions must be customized for each domain.The core filtering, gap identification, iterative retrieval, and articulation components transfer; the four reasoning strategies generalize broadly, allowing adaptation primarily through knowledge-base substitution.Source: Limitations · p. 8
• The water-quality setting is presented as one application validating a reusable design, rather than as a domain-specific system.This qualification accompanies the need to customize diagnostic questions.Source: Limitations · p. 8

Ethical Considerations

• The system is designed to assist with communicating critical scientific data, not to replace public agencies or domain experts.Inaccurate or incomplete information could lead to inappropriate actions, and human oversight remains essential in high-stakes deployments.Source: Ethical Considerations · p. 8
• Retrieval and auditing mitigate risks from hallucinations, omissions, and training biases but cannot eliminate them, making human oversight essential in high-stakes deployments.Answers use authoritative sources, highlight uncertainty, and specify what evidence does and does not prove.Source: Ethical Considerations · p. 8
• The system relies heavily on its knowledge base; real-world deployments risk institutional bias without proper supervision.Practical use in consequential domains requires transparent governance and strict source-auditing procedures.Source: Ethical Considerations · p. 8
• Human evaluation annotations were used for research and development of an automated completeness evaluator, without collecting or using personally identifiable information for model training.Source: Ethical Considerations · p. 8

A.1 Single-pass RAG baseline (Configuration 1)

• The single-pass RAG baseline instructs the system to constrain its knowledge to the supplied context and answer the query using that context.Source: §A.1 · p. 10

A.2 Prognostic check

• Chunks scoring at least 2 are passed downstream, while a relevance evaluator uses a 0–3 scale from irrelevant to directly answering the query.Scores of 1 and 2 denote tangentially related and partially useful chunks, respectively.Source: §A.2 · p. 10

A.3 Diagnostic check

• The diagnostic check assesses whether evidence satisfies a reasoning-strategy schema; it is instructed not to retrieve evidence, answer the query, or judge general sufficiency.Source: §A.3 · p. 10
• The diagnostic setup either supplies a default template or classifies the query into causal, regulatory, comparative, or explanatory reasoning before supplying its matching template.Source: §A.3 · p. 10

A.3.1 Strategy templates

• Explanatory templates require accessible definitions and clear scope, while diagnostic questions test whether the relevant concepts or comparisons are explained in the context of the water-quality issue.Listed failure patterns include missing policy or synthesis conditions, circular definitions, absent historical context, jargon-heavy text, and outdated or unverified values.Source: §A.3.1 · pp. 10–11
• Causal templates check for a named cause, an effect linked to the problem, a plain-language pathway, and evidence about temporal order and alternative causes.The listed failure patterns include missing mechanisms or confounder controls, missing temporal evidence or enabling conditions, and correlation presented as causation.Source: §A.3.1 · pp. 10–11
• Regulatory templates check that a policy or standard is identified in context, its purpose and mechanics are explained, and compliance duties, updates, and personal implications are addressed.Failure patterns include missing threshold values, the policy regulating the issue, or a connection to isolated evidence.Source: §A.3.1 · p. 11
• Comparative templates require named comparators, compatible shared metrics, a baseline, clear direction, and an explanation for observed differences.They also call for disclosure of structural differences that could affect fairness.Source: §A.3.1 · p. 11

• The causal-query examples explain lead as entering water from corroding plumbing and list pipes, fixtures, solder, and some well components as sources.Source: B Representative Outputs · p. 11
• The city hall translator answer identifies lead pipes as the most common source and says contributions vary by home, plumbing system, and well configuration.Source: B Representative Outputs · p. 11
• The caring-neighbor example opens with reassurance, explains common plumbing sources, and acknowledges that each source’s contribution varies.Source: B Representative Outputs · p. 11
• Figure 3 reports that evidential gap filling has a statistically insignificant effect on completeness.Source: Figure 3 · p. 12