PaperlayerView exact revision on arXiv

Source-linked AI summary

From Noisy Retrieval to Decision Evidence: Silver Evidence Demonstrations for In-Context Learning

Masoud Monajatipoor, Fazlolah Mohaghegh, Hamid Reza Hassanzadeh, Akash Uttamrao Bhoite, Robert E. Tillman

arXiv:2610.32128v1cs.CL

TL;DR

Long, noisy retrieval can bury decision-relevant evidence, while expert evidence in development examples is often used only as an answer-side rationale. The paper replaces raw retrieval in demonstrations with label-blind, target-conditioned silver evidence paired with expert supervision. The full method improves over gold-only ICL on three of four datasets and has higher label accuracy than InstructRAG-style ICL on all four.

  • Problem

    Retrieval-augmented ICL can present redundant or distracting passages, while expert evidence annotations offer little guidance for filtering broad retrieval when used only as answer-side rationales.

    Source: §1 · p. 1

  • Method

    The method replaces raw passages in demonstrations with target-conditioned silver evidence generated without labels or gold evidence, then uses one structured call to generate target evidence before predicting the label and decision evidence.

    Source: §1 · p. 1, §5 · p. 5

  • Results

    The full method improves over equally sized gold-only ICL on three of four datasets and achieves higher label accuracy than InstructRAG-style ICL on all four.

    Source: §9 · p. 8

  • Takeaways & Limitations

    The reported results suggest that compact, target-conditioned silver evidence is particularly useful for decisions based on long and noisy text.

    Source: §9 · p. 8

    The experiments do not establish a causal retrieval-to-evidence-to-decision mechanism because compression, output structure, and intermediate supervision remain partly confounded.

    Source: §8 · p. 8

Abstract

from arXiv · show

RAG and retrieval-augmented few-shot ICL can struggle when decision-relevant evidence is buried in long, noisy context. Retrieved passages and demonstrations may contain redundant or distracting information. Although expert-labeled development examples may identify the text supporting each decision, these annotations are typically used only as answer-side rationales rather than as supervision for filtering noisy retrieval. We propose silver-evidence demonstrations, which replace raw retrieved passages in ICL examples with generated, target-conditioned evidence summaries. For each labeled development example, an LLM generates silver evidence without access to the expert label or gold evidence; this evidence is then paired with the expert label and gold evidence where available. At test time, a single structured call generates silver evidence from retrieved passages and then predicts the final label and decision evidence. Across EI 2.0, SciFact, synthetic policy retrieval, and HICRIC/IMR data, our full method improves over equally sized gold-only ICL (label-only for policy) on three of four datasets and improves evidence F1 on all annotated datasets. It also achieves higher mean label accuracy than InstructRAG-style ICL on all four datasets. These results suggest that replacing demonstration-side retrieved passages with compact, target-conditioned silver evidence is particularly useful for decisions based on long and noisy text, where it reduces distracting context while preserving decision-relevant signals and expert supervision.

1 Introduction

The paper addresses how to make decisions grounded in long, noisy retrieval while identifying the source text behind them. It replaces raw retrieval in demonstrations with label-blind, target-conditioned silver evidence paired with expert supervision, and reports gains over gold-only ICL on three of four datasets and higher accuracy than InstructRAG on all four.

Motivation

  • In high-stakes tasks, source text that supports a decision is part of the decision product and can support audit, appeal, expert review, and error correction.Examples include clinical evidence synthesis, scientific claim verification, and coverage or prior-authorization workflows.Source: §1 · p. 1
  • Retrieved passages can be redundant, irrelevant, or distracting, while prompt limits constrain how many demonstrations fit.Conventional ICL often presents development evidence only as an answer-side rationale, offering little guidance on turning broad retrieval into expert-style evidence.Source: §1 · p. 1

Approach

  • The method replaces demonstration-side retrieval with label-blind, task-conditioned silver evidence paired with expert supervision, then uses one call to generate target evidence before predicting the label and evidence.Development silver evidence is generated without the expert label or gold evidence; expert evidence is paired with it when available.Source: §1 · pp. 1–2, Figure 1 · p. 2

Results

  • The full method achieves higher accuracy than InstructRAG on all four datasets, with slightly lower policy Macro-F1.GPT-5.5 has the strongest aggregate label performance across the datasets; gains over gold-only ICL are largest on policy retrieval and SciFact, smaller on HICRIC/IMR, and minimal on Evidence Inference.Source: §1 · p. 2

2 Related Work

Related work addresses retrieval, compact intermediate representations, and evidence or rationale supervision. The paper distinguishes its approach by replacing demonstration retrieval with label-blind silver evidence paired with expert evidence when available.

Retrieval-augmented generation

In-context learning and intermediate reasoning

  • ICL research studies example selection, order, similarity, format, calibration, and prompt budgets; long retrieved passages motivate compact intermediate representations.Work on chain-of-thought and related methods finds that intermediate text can aid reasoning, while its content and structure matter (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2023; Wang et al., 2023, 2022).Source: §2 · pp. 2–3
  • InstructRAG retains retrieved documents while using answer-conditioned rationales (Wei et al., 2025); this paper instead replaces demonstration retrieval with label-blind silver evidence paired with expert evidence when available.The paper positions its format as an asymmetric ICL contribution that does not train a compressor.Source: §2 · pp. 2–3

Evidence and rationale supervision

  • Evidence and rationale resources connect predictions with explanations or annotated evidence, but prior work cautions that plausible explanations need not be grounded or faithful (Jain et al., 2019; Wiegreffe et al., 2019).Clinical, scientific-verification, and long-document policy datasets motivate distinguishing retrieval-derived silver evidence from expert gold evidence.Source: §2 · p. 3

3 Methodology

The method holds dataset-specific retrieval fixed, generates structured silver evidence from query and retrieval, and uses it in supervised demonstrations before a final structured prediction call. Evaluation compares retrieval and evidence-based ICL variants using label metrics and, where available, evidence overlap against dataset-native references.

3.1 Problem Setup

  • Each example pairs a query and fixed retrieved passages with a label and, optionally, expert evidence; the method predicts a label and supporting evidence where available.Retrieval is held fixed across systems, so the method begins from the query and retrieved passages rather than optimizing retrieval.Source: §3.1 · p. 3

3.2 Silver Evidence Generation

  • Silver evidence is generated from the query and retrieval without the label or gold evidence, then cached as concise, source-supported snippets organized around the decision.Structured items preserve available source metadata and are instructed to avoid broad background, unsupported inferences, and label-specific wording.Source: §3.2 · p. 3

3.3 Demonstration Construction and Leakage Control

  • Demonstrations replace raw passages with compact silver evidence and supervision; eight lexical-overlap nearest neighbors are selected, with eligible pools separated to prevent target leakage.Selections are fixed across controlled systems when fields are available; the policy format uses labels without expert gold evidence.Source: §3.3 · pp. 3–4

3.4 Final Inference

  • A single structured call takes the target’s raw retrieval and demonstrations, then returns silver evidence before the label and gold-style evidence when supported.The target silver evidence is not frozen in a separate call before prediction.Source: §3.4 · p. 4

3.5 Compared Systems

  • The full method improves labels over equally sized gold-only ICL on three of four datasets and evidence F1 on all annotated datasets; it also achieves higher label accuracy than InstructRAG on all four.The policy comparison is label-only because that dataset has no expert evidence annotation.Source: §3.5 · p. 4, §3.6 · p. 4

3.5 Compared Systems; 3.6 Metrics

  • Systems compare retrieval-only, gold-only, silver-only, full, InstructRAG-style, and oracle evidence conditions; metrics include label accuracy and macro-F1 plus evidence and sentence/span F1 where annotations exist.Evidence scores use dataset-native references, not the generated silver field, and are interpreted separately from label performance.Source: §3.5 · p. 4, §3.6 · p. 4

4 Datasets

The evaluation spans clinical evidence inference, scientific claim verification, synthetic policy retrieval, and adapted appeal-policy prediction. These datasets differ in label spaces, test protocols, and availability or strength of evidence annotations.

Evidence Inference 2.0

  • Evidence Inference 2.0 pairs clinical-trial prompts with comparative outcome labels and expert evidence spans, using 260 completed test examples.Source: §4 · p. 4

SciFact

  • SciFact pairs claims with abstracts and sentence-level rationales across support, contradict, and not-enough-information labels; evaluation uses 300 labeled validation examples because test labels are hidden.Source: §4 · p. 4

Prior-authorization policy retrieval

  • The synthetic prior-authorization dataset maps requests and CMS-derived policy chunks to approve, deny, or pend, but has no expert evidence spans and is evaluated on labels only.Source: §4 · p. 4

HICRIC/IMR

  • HICRIC/IMR combines California DMHC appeal records with HICRIC-aligned CMS passages and evaluates binary approve/deny prediction on a label-balanced 2,000-example test subset.Explicit outcomes are removed, and evidence labels provide weak reviewer-rationale supervision.Source: §4 · p. 4

5 Experimental Setup

The experiments hold retrieval fixed across systems and use separate training and evaluation pools. Two LLM stages generate cached demonstration evidence and, at inference, target evidence with a final prediction; a SciFact example illustrates filtering irrelevant context.

5 Experimental Setup

  • Demonstrations come from training examples, while targets come from each dataset’s designated evaluation split.The pools are disjoint.Source: §5 · p. 5
  • Each system receives the same fixed target retrieval, isolating comparisons to representation and prompting rather than corpus coverage.Retrieval is prepared once using each dataset’s corpus and retriever or passage selector; cached silver files can be audited against their source passages and expert fields.Source: §5 · p. 5
  • Stage 1 generates cached silver evidence without access to training labels or gold evidence; Stage 2 jointly generates target evidence and the final prediction in one structured call.The target label and evidence are not included in the Stage 2 prompt, and malformed evidence or invalid labels are rejected.Source: §5 · pp. 4–5
  • In SciFact, silver evidence filters a misleading result: although the abstract reports reduced nausea and vomiting, the claim concerns bleeding, producing a CONTRADICT prediction aligned with the expert rationale.Source: §5 · p. 5

6 Results

The full method improves label accuracy over equally sized gold-only ICL on three datasets, with a small decrease on EI 2.0, and exceeds InstructRAG in mean accuracy across all four. Evidence-overlap metrics also improve across annotated datasets, while backend rankings vary and some comparisons have important scope limits.

6 Results

  • Accuracy changes by −0.1 points on EI 2.0 and +33.0, +3.3, and +8.0 points on policy retrieval, HICRIC/IMR, and SciFact versus gold-only ICL.Policy retrieval uses label-only gold demonstrations; InstructRAG has higher policy Macro-F1, although full-method mean accuracy is higher on all four datasets.Source: Table 1 · p. 5, §6 · p. 5
  • In claim 338, silver evidence isolates the bleeding outcome and effect size from a genuine but irrelevant nausea benefit before a CONTRADICT prediction.The prediction aligns with the expert rationale.Source: Figure 2 · p. 6
  • Evidence F1 gains over gold-only ICL are 3.8, 8.0, and 3.2 points on EI 2.0, SciFact, and HICRIC/IMR; Sentence/Span F1 gains are 4.5, 8.7, and 2.2.Evidence F1 measures textual rationale overlap, whereas Sentence/Span F1 measures selection of the annotated evidence unit; HICRIC/IMR uses weak reviewer-rationale evidence and policy is omitted for lacking gold spans.Source: §6 · p. 5, Table 2 · p. 6
  • The remaining gap to oracle evidence indicates that retrieval and evidence localization are still unresolved.The result applies to the evidence-annotated datasets and evidence measures reported in the paper.Source: §6 · p. 6
  • No LLM backend dominates every metric: GPT-5.6-Terra and Gemini Pro 3.1 lead EI 2.0 label metrics, while other backends lead on different datasets or evidence metrics.GPT-5.5 is used for the primary tables because it has the strongest aggregate label performance across the four datasets.Source: §6 · p. 6
  • Table 4 is contextual rather than a leaderboard because dataset splits, decision protocols, and label spaces differ, especially for SciFact, policy retrieval, and adapted HICRIC/IMR.The paper therefore reports label-level comparisons only.Source: §6 · p. 6

7 Ablation Study

Against equally sized gold-only ICL (label-only for policy), the full method’s accuracy gains range from −0.1 points on EI 2.0 to +33.0 on policy retrieval, with evidence F1 gains on all annotated datasets. Ablations show that expert evidence adds value beyond silver-only demonstrations, while oracle results and resource costs qualify the findings.

7 Ablation Study

  • Accuracy changes versus gold-only ICL are −0.1 points on EI 2.0, +33.0 on policy, +3.3 on HICRIC/IMR, and +8.0 on SciFact; Evidence F1 gains are 3.8, 8.0, and 3.2 points on annotated datasets.The policy comparison uses a label-only counterpart; the controlled baseline otherwise uses eight demonstrations.Source: §7 · pp. 6–7
  • Adding gold evidence to silver demonstrations changes accuracy by 0.2 points on EI 2.0, 6.0 on SciFact, and 10.3 on HICRIC/IMR.On HICRIC/IMR, accuracy rises from 79.2% to 89.5% and Evidence F1 from 19.7 to 38.4; SciFact Evidence F1 rises from 37.8 to 64.1.Source: §7 · p. 7
  • The full method exceeds InstructRAG by 0.6, 3.0, and 5.3 accuracy points on EI 2.0, SciFact, and HICRIC/IMR, but the comparison is not a pure label-conditioning ablation because demonstration retrieval differs.InstructRAG has slightly higher policy Macro-F1 (99.2% vs. 99.0%) but lower accuracy (99.0% vs. 99.5%).Source: §7 · pp. 7–8
  • Evidence F1 rises from 58.4 to 62.2 on EI 2.0, from 56.1 to 64.1 on SciFact, and from 35.2 to 38.4 on HICRIC/IMR, while absolute HICRIC/IMR overlap remains limited.Span F1 changes from 56.4 to 60.9 on EI 2.0; Sentence F1 changes from 56.1 to 64.8 on SciFact and 22.9 to 25.1 on HICRIC/IMR.Source: §7 · p. 8
  • Oracle evidence remains stronger on SciFact (91.3% vs. 81.0%) and HICRIC/IMR (98.1% vs. 89.5%), indicating that retrieval and evidence selection remain bottlenecks.Source: §7 · p. 8
  • Full-method GPT-5.5 runs use 2.19M, 1.86M, 1.50M, and 32.67M tokens on EI 2.0, SciFact, policy retrieval, and HICRIC/IMR, respectively; these totals are not directly comparable across datasets.Compact demonstrations may reduce inference-prompt length, while offline silver-evidence generation adds preprocessing cost.Source: §7 · p. 8

8 Discussion

The method’s gains vary across datasets, consistent with differences in retrieval length and decision demands. The experiments support a representation benefit, but do not establish a causal retrieval-to-decision mechanism.

8 Discussion

  • The experiments do not establish a causal retrieval-to-evidence-to-decision mechanism because raw retrieval and silver evidence are not paired in demonstrations, and test-time evidence and labels are generated together.Compression, output structure, and intermediate supervision remain partly confounded.Source: §8 · p. 8
  • The full method has higher accuracy than InstructRAG on all four datasets, slightly lower policy Macro-F1, and stronger evidence results on annotated datasets.The largest accuracy gains are on policy and SciFact; gains are smaller on HICRIC/IMR and minimal on EI 2.0.Source: §8 · p. 8
  • Policy retrieval may benefit from filtering long, noisy criteria, while SciFact may benefit from isolating evidence for fine-grained entailment.HICRIC/IMR has a similar grounding need but weak reviewer rationales; EI 2.0 often uses compact intervention–outcome comparisons.Source: §8 · p. 8
  • On SciFact, summaries may blur support, contradiction, and insufficient-information distinctions; silver evidence is most reliable when it preserves decision-critical distinctions.Compact EI 2.0 passages leave little room for label gains despite improved evidence quality.Source: §8 · p. 8
  • The cross-dataset contrast supports a representation benefit rather than a universal reasoning improvement.SciFact labels depend on subtle entailment, directionality, population, and outcome distinctions that an over-general summary can erase.Source: §8 · p. 8

9 Conclusion

The paper introduces asymmetric silver-evidence demonstrations that replace demonstration retrieval with label-blind summaries paired with expert supervision. The full method improves labels on three of four datasets and evidence overlap on all annotated datasets, while exceeding InstructRAG in label accuracy on all four.

9 Conclusion

  • The full method improves labels over the corresponding baseline on three of four datasets and evidence overlap on all annotated datasets.It also achieves higher label accuracy than InstructRAG on all four datasets.Source: §9 · p. 8
  • Demonstrations replace raw retrieval with label-blind summaries paired with expert supervision, while test inputs retain raw retrieved passages.The summary is generated from target-conditioned retrieval without access to the label or expert evidence.Source: §9 · p. 8
  • At test time, one structured call generates silver evidence from raw retrieval before predicting the label and decision evidence.The approach is an asymmetric silver-evidence demonstration format for retrieval-augmented ICL.Source: §9 · p. 8

A.1 Prompting Details

The appendix specifies how silver evidence is generated and used in demonstrations and test prompts, along with baseline formats, dataset fields, backend identifiers, and experimental settings.

A.1 Prompting Details

  • Stage 1 receives each demonstration query and retrieval but excludes its gold label and evidence while generating concise, source-grounded information useful for the downstream decision.The prompt asks it not to predict the label or infer expert evidence, and to preserve available titles, headers, categories, sentence IDs, or policy sections.Source: §A.1 · p. 12
  • Stage 2 uses demonstrations with cached silver evidence and target retrieval; one structured call returns silver evidence, the label, gold evidence where applicable, and an explanation.Silver evidence may be header-keyed when metadata are available, and generated evidence is cached for later runs.Source: §A.1 · p. 12

A.2 Baseline Prompt Formats

  • The baseline formats include retrieval-only, gold-only or label-only ICL, silver-only ICL, the full method, oracle evidence, and InstructRAG-style ICL.Policy uses label-only ICL because expert evidence is unavailable; InstructRAG retains retrieval and uses label-conditioned rationales.Source: §A.2 · p. 12

A.3 Dataset-Specific Prompt Fields

  • EI 2.0 prompts use intervention, comparator, outcome, and article metadata; SciFact uses claims and titled abstract sentences with SUPPORT, CONTRADICT, or NOT ENOUGH INFO labels.EI 2.0 evidence is scored against expert spans, while SciFact evidence uses rationale IDs and text overlap.Source: §A.3 · p. 13
  • Policy prompts pair prior-authorization requests with retrieved criteria and approve, deny, or pend labels; expert evidence spans are unavailable.HICRIC/IMR prompts use treatment, diagnosis, background, and available rationale with retrieved CMS passages; labels are approve/deny and evidence uses weak rationale text.Source: §A.3 · p. 13

A.5 Other Experimental Details

  • All systems use the same within-dataset retrieval, and eight demonstrations are selected deterministically by lexical nearest-neighbor token overlap without manual curation or explicit class balancing.GPT-5.5 is used for both stages in the primary comparison; repeated final inference runs three times with cached silver evidence fixed.Source: §A.5 · p. 13

A.4 Backend Identifiers

  • Table 5 lists the provider-facing model and configured deployment identifiers used in the backend comparison.The identifiers and API access were last verified on October 12, 2026.Source: §A.4 · p. 13, Table 5 · p. 14

A.5 Other Experimental Details

  • Table 6 reports full-method backend sensitivity across accuracy, Macro-F1, Evidence F1, and Sentence/Span F1, with token and latency means and bold marking best available results.Repeated-run metrics include mean ± standard deviation; single-run summaries use ±0.0.Source: Table 6 · p. 14

Related papers

Loading 2610.32128v1…