PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Conformal Language Tasks with In-Context Ensembles

Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara, Kin Kwan Leung, Jesse C. Cresswell

arXiv:2609.03005v1cs.CLcs.LGstat.ML

TL;DR

NLP content-selection tasks need coverage of relevant information while excluding irrelevant content, but existing conformal methods rely on task-specific LLM prompts whose manual engineering is labor-intensive. The paper introduces Conformal Relevance, which curates in-context examples and ensembles diverse relevance scorers while retaining conformal coverage. Across seven datasets, one fixed configuration improves conciseness and MAP over manually crafted prompts and single-strategy baselines, with up to ∼50% less retained content.

  • Problem

    Conformal content selection guarantees coverage, but conciseness depends on scoring functions that typically require labor-intensive, task-specific LLM prompt engineering.

  • Method

    Conformal Relevance replaces manual relevance prompts with curated in-context examples and mean-ensembles diverse ICL-based scoring functions for conformal calibration.

  • Results

    Across seven datasets, one fixed Ens4 configuration improves MAP over ICL0 and the best single strategy, while removing substantially more irrelevant content at the same coverage.

  • Takeaways & Limitations

    A task-agnostic ICL ensemble can provide conformal coverage while improving conciseness across varied content-selection tasks with minimal manual input.

  • Takeaways & Limitations

    The framework assumes approximately 150–440 labeled examples, a static relevance criterion supporting exchangeability, and optionally a manually crafted one-line task hint.

Abstract

from arXiv · show

Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.

1 Introduction

NLP content-selection tasks require both coverage of relevant information and conciseness, while conformal prediction guarantees coverage but leaves conciseness dependent on the scoring function. Conformal Relevance replaces labor-intensive task-specific prompts with curated in-context examples and diverse score ensembling, improving conciseness across seven tasks.

  • NLP tasks such as summarization, extractive QA, legal review, and clinical evidence selection reduce to retrieving relevant document content under coverage and conciseness demands.
  • Conformal prediction provides distribution-free, finite-sample coverage guarantees, but conciseness depends strongly on the scoring function’s predictive power.
  • Task-specific LLM prompts define relevance criteria, but manual prompt engineering is labor-intensive, brittle, and not scalable.
  • Conformal Relevance uses curated in-context examples to let an LLM infer relevance instead of receiving a natural-language criterion through manual prompt writing.
  • A mean ensemble of mechanistically distinct ICL scoring functions improves performance by spanning different signal types whose failures are unlikely to coincide.
  • Up to ∼50% reduction in retained length is achieved at the same coverage, with gains attributed to retrieval-induced diversity and requiring 150–440 labels per task.

2 Background & Related Work

The background frames content selection as extracting task-defined relevant spans with conformal guarantees, while reviewing related work on conformal prediction, content selection, ensembles, and ICL example selection.

  • Content selection: Content selection extracts relevant document spans, with relevance varying across summarization, QA retrieval, and private-information detection.
  • Conformal prediction: Split conformal prediction calibrates a score function on labeled data to produce prediction sets with finite-sample coverage under exchangeability.
  • Conformal methods for content selection: Conformal factuality filters potentially hallucinated QA claims, whereas conformal importance provides recall-style coverage for retaining important extractive content.
  • Conformal methods for content selection: For content selection, a relevance function scores candidate spans, and the conformal score is the lowest relevance level retaining the target fraction of important spans.
  • Related work: Prior conformal relevance methods typically use manually crafted prompts, motivating a scoring function that generalizes across tasks without per-domain engineering.
  • Related work: Related work studies score-level conformal aggregation and ICL example selection based on similarity, learned retrieval, ordering, label mapping, and diversity.

3 Theoretical Framework

The theoretical framework shows that score-level mean ensembling preserves conformal coverage under exchangeability and improves the worst-case positive-span score when scorer failures are sufficiently complementary. Additional scorers provide diminishing returns, while the framework extends to partial recall targets.

  • Problem Setup: The conformal score is the lowest relevance assigned to any positive span, so raising this floor can shrink prediction sets when coverage remains unaffected.
  • Validity via ensembling: Score-level mean ensembling preserves the 1−α coverage guarantee under exchangeability, unlike set-level ensembling’s stated 1−2α floor.
  • Complementarity: Complementarity measures disagreement over the lowest-scoring positive spans, and ensembles benefit when scorers’ minima occur on different positives.
  • Complementarity: For two scorers, ensemble improvement occurs exactly when complementarity exceeds the individual floor gap: Comp > Smax − Smin.
  • Diminishing returns: An added scorer raises the ensemble floor when it assigns higher relevance to the current lowest-scoring spans.
  • Diminishing returns: The marginal gain from adding scorer K+1 is at most (1−S(K))/(K+1), establishing diminishing returns as ensemble size grows.
  • Extension to β < 1: For β < 1, the theory replaces the minimum with an rth order statistic and retains an analogous complementarity condition for raising the partial-recall score.

4 Method - Conformal Relevance

Conformal Relevance constructs task-agnostic relevance scores by retrieving labeled ICL examples through diverse strategies, averaging their scores, and conformally calibrating the resulting function. The fixed Ens4 configuration uses four strategies with two examples each and is applied across content-selection datasets.

  • Data split: The method partitions data into an ICL pool, calibration set, and held-out test set, with calibration and test sampled to satisfy exchangeability.
  • Relevance scoring: Each LLM relevance function scores content spans using only k labeled ICL examples rather than a manually crafted task-specific relevance prompt.
  • ICL example selection: The ensemble balances diversity benefits against cost by combining K = 4 ICL-selection strategies with distinct retrieval signals.
  • ICL example selection: The strategies include similarity-anchored DPP retrieval and pattern-based selection using positive and negative span centroids.
  • Ensembling: Because all scorers output values on [0, 1], their scores are averaged element-wise into a single conformal scoring function.
  • Ensembling: Ens4 is the mean ensemble of four ICL-selection strategies using k=2 examples per scorer, with validity established through the conformal argument.

5 Experiments

Across seven sentence-level relevance datasets, the fixed Ens4 configuration improves MAP and conciseness over manually crafted prompts and single-strategy baselines while preserving conformal coverage. Ablations and controls attribute the gains to retrieval-induced complementarity among diverse ICL strategies, with diminishing returns as ensemble size grows.

  • Main results: Ens4 improves MAP over ICL0 and the per-dataset Best Single strategy on all seven datasets.Improvements over ICL0 range from +0.090 to +0.170 MAP, with 95% confidence intervals excluding zero.
  • Ablations and validation: Using k=2 ICL examples per strategy is effective: k=1 underperforms on 6/7 datasets, while k ≥5 degrades performance on most datasets.The authors use k=2 as the more efficient of the two similar-performing options.
  • Ablations and validation: The default K=4 ensemble performs best on every dataset, while marginal gains decrease from K = 3 →4 compared with 2 →3.This empirical pattern is consistent with the theoretical O(1/K) diminishing-returns bound.
  • Ablations and validation: Ens4 outperforms a single strategy using the same eight-example ICL budget by +0.070 to +0.119 MAP on every dataset.The comparison distributes eight examples across four retrieval signals rather than concentrating them in one signal.
  • Ablations and validation: Retrieval-induced complementarity, rather than additional labels or stochastic temperature diversity alone, explains the ensemble’s improvement.Ens4 beats a supervised classifier on 6/7 datasets and temperature-diversity baselines on 12/16 settings and 6/7 datasets, respectively.
  • Ablations and validation: The method remains robust across alternative LLMs and removal of the task hint, with Ens4 beating both baselines on 7/7 GPT-5.6-terra dataset evaluations.Removing the hint changes MAP by at most 0.04 on 6/7 datasets; PhysioNet is materially affected.
  • Coverage and conciseness: Empirical coverage stays within 1 percentage point of the target 1−α coverage despite using 100 labeled calibration samples.Mean ensembling remains closest to target coverage while achieving the highest conciseness among the tested ensemble rules.
  • Coverage and conciseness: Ens4 removes more content than ICL0 at fixed coverage across all datasets at 1 −α = 0.8.The conciseness gains show that higher MAP translates into smaller, more useful prediction sets.

6 Conclusion

The paper introduces Conformal Relevance, which uses in-context learning and a four-strategy ensemble to provide coverage guarantees for extractive content selection with minimal manual input. Across seven datasets, it outperforms bespoke prompts in removing irrelevant content, while theory characterizes ensemble gains and their limits.

  • Conformal Relevance replaces task-specific, hand-crafted scoring prompts with a four-strategy mean ensemble of ICL-based relevance functions.The framework defines relevance implicitly through curated examples and provides coverage guarantees for extractive content selection.
  • A single fixed configuration outperforms bespoke prompts on seven datasets spanning five domains and filters substantially more irrelevant content across coverage and recall levels.
  • The framework’s theoretical analysis characterizes when ensemble complementarity improves worst-case scores and bounds additional gains from adding scorers.
  • Future work includes adaptive aggregation, graded relevance labels, and extending beyond extractive selection to generative tasks.

Limitations

The method’s scope is constrained by labeling, exchangeability, model coverage, computation, and data-distribution assumptions. The supplied passages also identify several practical and evaluation boundaries for deployment and replication.

  • Data and assumptions: The framework assumes approximately 150–440 labeled examples, a static relevance criterion, and optionally a manually crafted one-line task hint.The relevance criterion must not drift between calibration and test to preserve exchangeability; the hint was unnecessary on 6/7 datasets with little change.
  • Model scope: The main results use Gemini-2.5-Flash-Lite, with ablations on two 8B open-source LLMs, leaving model coverage limited.The authors note that stronger models have since surpassed this evaluated set.
  • Computational cost: Ens4 requires K = 4 LLM calls per document versus one for ICL0, increasing test-time computation and costs.
  • Reproducibility: Residual hosted-LLM server variation can shift individual sample scores between reruns despite fixed temperature, seeds, model version, and prompts.The authors recommend targeting aggregate MAP rather than per-sample scores for replication.
  • Coverage scope: The 1−α guarantee is marginal over the test distribution and does not imply conditional coverage across intents, lengths, or protected attributes.Mondrian conformal prediction is suggested when conditional coverage is required.
  • Data sparsity: When positives are very sparse, few positive ICL candidates may prevent sufficiently informative context, requiring larger or more carefully constructed pools at higher labeling cost.
  • Validity conditions: The conformal validity argument can fail if ICL examples leak from calibration or test data, or if aggregation depends on calibration data.The three-way split and fixed aggregation are intended to prevent these scenarios.

C.2 Ensemble floor advantage proof

The mean ensemble improves the worst-case positive-span score precisely when complementarity exceeds the gap between the stronger and weaker individual floors. The generalized result extends this condition to β < 1, while also showing diminishing returns as ensembles grow.

  • Ensemble floor advantage: At K=2 and β=1, the mean ensemble floor exceeds the better individual floor iff Comp > Smax − Smin.The decomposition expresses ensemble improvement through complementarity relative to the individual-floor gap.
  • Disjoint minimizers: Disjoint minimizers make the ensemble floor strictly exceed every individual floor because no positive span is minimized by all scorers.The condition is stated using the sets Aj of spans attaining each scorer’s minimum.
  • Worked example: In the worked K=2 example, complementarity 0.3 exceeds the floor gap 0.1, yielding S(2)=0.60 versus Smax=0.50.Each scorer compensates for the other’s weakness; shared weaknesses eliminate the improvement.
  • Extension to β < 1: For β < 1, the ensemble score exceeds Sβ,max iff at least ⌈β|y∗|⌉ positive spans have ensemble scores above Sβ,max.At β=1, this reduces to requiring every positive span to exceed the stronger individual floor.
  • Extension to β < 1: At β < 1, complementarity can be negative, so averaging scorers no longer guarantees that the conformal score will not decrease.The guarantee follows from concavity only for the minimum order statistic at β=1.
  • Diminishing returns: Adding scorers has diminishing returns: the new ensemble floor can rise only when the added scorer lifts every relevant order-statistic constraint, with an O(1/K) upper bound.The argument uses scores bounded in [0,1].

D Reproducibility Details

The reproducibility setup fixes models, seeds, temperature, data splits, calibration size, prompts, and ICL formatting. It also specifies deterministic contrastive examples and scoring behavior across task types.

  • Experimental settings: All experiments use Gemini-2.5-Flash-Lite at T=0 with fixed seeds, and calibration size n=100 across datasets.The fixed seeds cover pool sampling, strategy selection, negative subsampling, and calibration/test splits.
  • Data and ICL sizing: The ICL pool contains 50 examples for single-intent datasets or 20 per intent for multi-intent datasets, while each Ens4 scorer consumes k=2 examples.The larger pool supports diversity-maximizing selection strategies.
  • Dataset construction: Four datasets are reformulated into sentence-level binary relevance tasks, while SubSumE, PUMA, and PhysioNet already provide sentence-level labels.The reformulated positive definitions use task-specific annotations such as supporting facts, summary alignment, evidence spans, or PHI labels.
  • ICL format: Contrastive ICL examples present selected positives and a balanced sample of unselected negatives, with at least two negatives when available.Negative subsampling uses a deterministic seed and is held constant across ICL-free, single-strategy, and ensemble conditions.
  • Scoring protocol: The scoring prompt asks the LLM to infer the selection distinction from demonstrations and output a score between 0 and 1 for each sentence.Scores above 0.8 indicate clear matches, scores below 0.2 indicate non-matches, and intermediate values represent partial matches.

E.2 Windowed Context Algorithm and Ablation

The paper uses windowed ICL compression for long or sparse documents, retaining positives, local context, and remote negatives to reduce prompt length. Task-specific prompts define relevance for several application settings.

  • Windowed context: The compressed ICL representation retains all positive sentences, asymmetric local windows around positives, and sampled remote negatives.This design reduces prompt length while preserving the information used for ICL scoring.
  • Windowed context: Windowed context activates when average document length exceeds 30 sentences or average positive rate falls below 15%.SubSumE and Evidence Inference trigger the condition; the other listed datasets do not.
  • ICL selection strategies: Ens4 uses four complementary selection strategies: anchor_dpp, pattern_dpp, bm25, and random.The strategies target document similarity, relevance-direction diversity, lexical overlap, and query-independent sampling, respectively.
  • ICL selection strategies: Anchor-DPP selects a query-nearest anchor, then draws remaining examples with a conditional DPP favoring semantic proximity and mutual diversity.Its design seeks representative yet nonredundant examples for the scorer.
  • Task scoring prompts: The task prompts score relevance using task-specific criteria such as summary alignment, query answering, discussion perspective, PHI presence, and supporting-fact usefulness.Scores are requested for every sentence on a 0-to-1 scale.

H Main Empirical Results

Across seven datasets, a single Ens4 configuration improves MAP over ICL0 and the best single strategy, while ensemble gains increase with K but show diminishing returns. The ensemble uses more parallelizable LLM calls and therefore higher directional cost.

  • Main MAP results: Ens4 improves MAP on all seven datasets over ICL0 and the per-dataset best single strategy.Figure 2 visualizes these comparisons, while Table 3 reports the numerical values and confidence intervals.
  • Computational cost: Ens4 requires four independent scoring calls per document versus one for ICL0, but the calls can be parallelized.Serial input-token cost approaches 4× as document length grows, and latency measurements include provider-side effects.
  • Ensemble size: Mean MAP rises from 0.587 at K=1 to 0.625 at K=2, 0.650 at K=3, and 0.664 at K=4, with strictly diminishing marginal gains.Ens4 ranks first among all 11 non-singleton strategy subsets on every dataset.
  • ICL budget comparison: Ens4 exceeds the k=8 Best Single baseline by +0.070 to +0.119 MAP on every dataset.Increasing one strategy’s examples from k=2 to k=8 is not helpful, whereas distributing the budget across strategies improves performance.
  • Control comparison: Ens4 outperforms the supervised classifier on 6/7 datasets, with the classifier performing best only on ECTSum.The classifier uses the full ICL pool and calibration samples, while each Ens4 scorer sees only two examples at a time.

I.4 Complementarity vs. Stochasticity and Compute

The ensemble’s gains are attributed primarily to retrieval-induced diversity rather than stochastic temperature variation, while task hints and selection strategies affect performance unevenly. Windowed rendering improves ICL efficiency without changing selected examples.

  • Complementarity vs. Stochasticity and Compute: Ens4 outperforms temperature ensembles in 12/16 cases, with overlapping confidence intervals in the remaining 4 cases.The comparison evaluates temperature ensembles for each of four strategies across four datasets.
  • Complementarity vs. Stochasticity and Compute: Ens4 significantly outperforms equal-compute ICL0×4 on 6/7 datasets, with ContractNLI statistically tied.ICL0×4 uses four calls at temperatures T ∈ {0.3, 0.5, 0.8, 1.0}.
  • Robustness and Ablations: Across seven datasets, the default Ens4 configuration is evaluated alongside cross-model validation on Llama3-8B, Qwen3-8B, and GPT-5.6-terra.GPT-5.6-terra uses seeded 500-sample test subsets, while the other models use original test sets.
  • Task-Hint Format Ablation: Removing the task hint collapses PhysioNet MAP from 0.879 to 0.549 because regulatory criteria are not inferable from contrastive examples alone.Evidence Inference and ECTSum lose approximately 0.04 MAP without the hint, whereas four transparent datasets change by at most 0.01.
  • ICL Selection Ablations: Stratified ICL selection improves MAP on three of four multi-intent datasets by +0.018 to +0.028, while Evidence Inference is marginally better with non-stratified selection by −0.006.The exception may reflect weak correlation between Evidence Inference’s user-defined intents and relevance patterns.
  • Windowed ICL Selection Ablation: Windowed ICL improves SubSumE MAP by +0.10 at k = 2 and +0.16 at k = 5 by compressing rendering without altering example selection.The largest gains occur when full-document examples overflow the context window and degrade coherence.

J Empirical Validation of Ensemble Theory

Empirical checks support the proposed complementarity condition and show that Ens4 improves conciseness across coverage settings. The evidence also identifies a task-specific outlier that benefits from a fuller relevance description.

  • Ensemble Theory Validation: All 1,648 disagreements with Lemma 1’s condition are attributable to floating-point ties within 10^-12.The condition is Comp(R_j, R_k) > S_max − S_min ⇐⇒ S^(2) > S_max.
  • Conciseness Across Coverage Targets: Ens4 improves conciseness over ICL0 on all seven datasets at α = 0.20 and on most datasets at α = 0.05 and α = 0.10.These evaluations fix β = 0.8 while varying α across 0.05, 0.10, and 0.20.
  • Task-Specific Boundary: Evidence Inference is the largest outlier at α = 0.05 because its relevance criterion depends on agreement with experimental evidence rather than a semantic theme.Its criterion is non-semantic and benefits from a more descriptive task hint.
  • Task-Specific Boundary: Ens4-FullTask recovers a positive conciseness gap over ICL0 across all β on Evidence Inference.The variant uses a lengthier manually crafted prompt describing the task.
  • Alternative Ensemble Rules: Majority Vote can under-cover, COLA requires jointly optimized scorer-specific α_j values, and Union of Sets preserves coverage but typically over-covers.These alternatives combine conformal sets differently from score-level mean ensembling.
  • Alternative Ensemble Rules: The comparison averages coverage deviation across seven datasets and measures conciseness at (α, β) = (0.2, 0.8) for the tested ensemble rules.The Average Single Scorer baseline evaluates each constituent independently before averaging metrics.
Loading 2609.03005v1…