PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Evidence Highlighting for Frozen LLMs

Shaoang Li, Yanhang Shi, Yufei Li, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Jian Li

arXiv:2604.22565v2cs.CLcs.AI

TL;DR

Long-context LLMs can miss decisive evidence amid noise. HiLight trains a lightweight Actor to mark pivotal spans in unaltered inputs for a frozen Solver using only task reward, and it consistently improves performance while transferring across unseen Solvers.

  • Problem

    Long-context LLMs often overlook decisive evidence when it is diluted among irrelevant, redundant, or outdated information.

  • Method

    HiLight trains a lightweight Emphasis Actor with task-level reinforcement learning to mark spans in unaltered context for a frozen Solver.

  • Results

    HiLight consistently improves performance across recommendation and long-context QA, with the largest gains in high-distractor settings.

  • Takeaways & Limitations

    The learned emphasis policy transfers zero-shot across smaller and larger unseen Solver families, including an API-based Solver.

  • Takeaways & Limitations

    The study focuses on per-query evidence emphasis for frozen Solvers under fixed-context inference; cache-aware conversational reuse remains future work.

Abstract

from arXiv · show

Large Language Models (LLMs) can reason well, yet often miss decisive evidence when it is buried in long, noisy contexts. We introduce HiLight, an Evidence Emphasis framework that decouples evidence selection from reasoning for frozen LLM solvers. HiLight avoids compressing or rewriting the input, which can discard or distort evidence, by training a lightweight Emphasis Actor to insert minimal highlight tags around pivotal spans in the unaltered context. A frozen Solver then performs downstream reasoning on the emphasized input. We cast highlighting as a weakly supervised decision-making problem and optimize the Actor with reinforcement learning using only the Solver's task reward, requiring no evidence labels and no access to or modification of the Solver. Across sequential recommendation and long-context question answering, HiLight consistently improves performance over strong prompt-based and automated prompt-optimization baselines. The learned emphasis policy transfers zero-shot to both smaller and larger unseen Solver families, including an API-based Solver, suggesting that the Actor captures genuine, reusable evidence structure rather than overfitting to a single backbone.

1 Introduction

HiLight separates evidence selection from reasoning by marking pivotal spans in unaltered long contexts for a frozen Solver. It learns this emphasis policy from task reward and improves performance across diverse tasks, especially when distractors obscure sparse evidence.

  • Motivation: HiLight addresses long-context failures by decoupling evidence selection from multi-step reasoning.The framework uses a lightweight module to surface task-relevant spans before a frozen Solver reasons over them.
  • Framework: The Emphasis Actor inserts minimal structural markers around selected spans while preserving the original context.HiLight does not compress, summarize, or rewrite the input.
  • Evaluation: HiLight improves consistently over prompt-optimization and automated prompt-search baselines across sequential recommendation and multiple long-context QA tasks.The evaluation covers sequential recommendation, multi-hop QA, reading comprehension, and biomedical QA classification.
  • Evaluation: Up to ∼27% improvement over MI on HR@10 occurs in Amazon-Beauty, while gains are smaller but consistent on concentrated-evidence tasks such as PubMedQA.The largest gains occur in high-distractor, sparse-signal settings.
  • Analysis: Preserving the full context with highlight markers outperforms pruning, and learned Actors transfer zero-shot across unseen Solver families and scales.Robustness also holds across highlight budgets and marker formats.
  • Learning: HiLight learns highlighting with weakly supervised reinforcement learning using only downstream Solver rewards, without evidence annotations or Solver internals.This makes the approach compatible with frozen and API-based Solvers.

2 Related Work

Prior approaches reduce, rewrite, internally score, or statically format long contexts, often losing information or failing to select evidence instance by instance. HiLight instead learns where to place minimal markers in the raw context and applies this control across queries and Solvers.

  • Context reduction: Retrieval-style methods within an already supplied context perform hard selection by retaining top-k chunks or spans and discarding the rest.The Pruned ablation shows deletion can help sparse-signal recommendation but hurt multi-hop QA when connective context is removed.
  • Context reduction: Compression, summarization, and context distillation reduce input length but can omit subtle decisive details and bridging information.Prompted extraction also adds a generation step and couples evidence selection to the Solver's instruction-following ability.
  • Context markup: Structured prompting makes evidence boundaries explicit, but conventional formats are hand-designed and static rather than instance-specific.HiLight learns placement of minimal highlight tags for each input.
  • Prompt optimization: Unlike task-level APO, HiLight performs instance-level optimization on each data context while retaining the same raw evidence.This supports adaptation to varying noise and evidence distributions across queries.
  • Prompt optimization: HiLight targets evidence utilization through one emphasized input rather than extensive over-sampling used by some test-time scaling strategies.The related distinction concerns optimization scope and inference computation.
  • HiLight distinction: HiLight externalizes saliency cues in the raw input instead of discarding tokens or altering model weights, steering a frozen Solver's attention.This distinguishes input-side emphasis from pruning and gradient-based calibration.

3 Problem Formulation

The formulation treats evidence as latent and learns a budgeted highlighting policy for a frozen Solver. The Actor selects a binary mask, applies a non-destructive markup operator, and is optimized from task reward without token-level supervision or Solver gradients.

  • Task setup: A data instance consists of a query Q, tokenized context X, ground-truth output y*, and a frozen Solver that produces y.Task performance is measured by a task-specific utility such as EM, F1, accuracy, or NDCG.
  • Latent evidence: Relevant evidence is modeled as one or more latent subsets of the context because task information may be diluted by irrelevant, redundant, or outdated content.The evidence set is unobserved, need not be unique, and may depend on the task and Solver.
  • Evidence highlighting policy: The Actor policy selects a binary token mask M, and a deterministic operator inserts minimal highlight tags while preserving token content and order.Selected tokens may form coalesced highlighted spans.
  • Highlight budget: A highlight budget limits the maximum fraction of emphasized tokens to prevent all-highlight solutions and keep emphasis concise and auditable.The constraint is enforced through projection with k = ⌊γL⌋.
  • Objective and optimization: The objective is to maximize expected task reward from the frozen Solver operating on the emphasized context.Because the Solver is non-differentiable and evidence masks are unavailable, policy-gradient black-box optimization uses sparse task reward.

4 HiLight Method

HiLight trains a lightweight Actor to select and mark evidence while a frozen Solver reasons over the preserved context. Its method combines budgeted highlighting, deterministic tag insertion, and task-reward policy-gradient optimization.

  • Actor policy: HiLight parameterizes an Actor policy over context-token emphasis decisions conditioned on the query and context.The Actor produces token importance probabilities and a Bernoulli selection policy.
  • Highlight construction: The method samples masks, projects them to a highlight budget, coalesces adjacent selections into spans, and inserts boundary tags without changing token content or order.Inference uses deterministic top-k selection, while training preserves exploration through Bernoulli sampling before projection.
  • Frozen Solver prompting: The frozen Solver receives the emphasized context through a consistent prompt that encourages attention to highlighted spans while retaining access to surrounding context.The Solver remains frozen and may be open-weight or API-based.
  • Task-level reward: HiLight optimizes the Actor with grouped policy-gradient reinforcement learning using the Solver’s task metric as reward.Grouped sampling forms normalized within-group advantages, while the Actor is updated without gradients through the Solver.
  • Regularization: Entropy and target-length regularization stabilize exploration and encourage the expected highlight fraction to match the budget, alongside hard budget projection.The final loss combines policy-gradient, length, and entropy terms, with optional trust-region regularization described separately.

5 Experiments

HiLight consistently improves performance across recommendation and long-context QA settings, with the largest gains under sparse evidence and heavy distractors. Ablations, efficiency analyses, sensitivity tests, and transfer results support non-destructive highlighting as a robust and broadly useful intervention.

  • HiLight achieves the best performance across all four tasks and eight metrics, outperforming manual instructions and prompt-optimization baselines.
  • +5.38% HR@10 and +10.82% NDCG@10 over the strongest baseline occur on Amazon-Beauty, where decisive evidence is sparse amid heavy distractors.
  • HiLight maintains 1.08%–5.80% improvements across metrics on HotpotQA, SQuAD 2.0, and PubMedQA, although gains are smaller than on recommendation.
  • Pruning selected spans can recover predictive signal on Amazon-Beauty, but full-context highlighting performs best by steering attention while preserving disambiguating and preference-consistency information.
  • On HotpotQA, pruning substantially hurts EM, whereas HiLight improves it, indicating that multi-hop reasoning benefits from retaining connective material beyond selected supporting sentences.
  • HiLight reduces Solver-query budgets by ∼5–10× versus PRL and APE while achieving higher accuracy, with Solver-token overhead below 1.01×.
  • Performance remains broadly stable across highlight budgets, marker syntax, Actor capacities, and contexts extended to approximately 32K tokens.
  • HiLight transfers across smaller and larger unseen Solver families, with relative improvements of +8.78% to +19.17% on HR@10 and +8.48% to +31.56% on NDCG@10.

6 Conclusion

HiLight decouples evidence selection from reasoning by training a lightweight Actor to mark pivotal spans in unaltered inputs for a frozen Solver. Across recommendation and long-context QA, it improves over prompting baselines, especially with sparse evidence and distractors, while preserving connective context.

  • HiLight decouples evidence selection from reasoning for a frozen Solver.A lightweight Actor learns from task-level reward to insert minimal highlight markers into the unaltered input.
  • HiLight consistently improves performance over manual prompting and strong prompt-optimization baselines across recommendation and long-context QA.
  • The largest improvements occur in high-distractor regimes where decisive evidence is sparse.
  • Non-destructive emphasis outperforms hard pruning on reasoning-heavy tasks by preserving connective context.

A Benchmarks

The benchmark suite compares manual instructions with automated prompt-optimization methods across recommendation and long-context question-answering settings. These baselines differ in how they generate, search, and evaluate prompt candidates.

  • The benchmark compares HiLight against manual engineering and automated prompt-optimization methods.
  • MI uses human-written instructions without automated optimization or structural emphasis on the context.
  • PRL trains a prompt generator with reinforcement learning and intermediate reasoning traces, while synthesizing task-specific few-shot examples.
  • BFRS performs best-first search over prompt candidates and refines them using feedback from model executions.
  • OPRO iteratively proposes prompt candidates from prior candidates and scores, whereas APE generates and filters prompts from demonstrations.
  • DSPy jointly tunes instructions and few-shot demonstrations using a grounded proposer and Bayesian surrogate search.

B Additional Case Studies

Case studies show HiLight surfacing task-relevant evidence for frozen Solvers in multi-hop and biomedical question answering. The highlighted spans address entity confusion, evidence dilution, and the links needed for correct inference.

  • HiLight’s case studies cover HotpotQA multi-hop question answering and PubMedQA biomedical yes/no question answering.
  • The qualitative examples illustrate mitigation of entity-level confusion and evidence dilution, improving the frozen Solver’s final prediction.
  • HotpotQA Multi-hop QA: On HotpotQA, HiLight highlights Kirton End’s Boston district and Boston’s 2001 census population to support the correct answer.
  • PubMedQA Biomedical Yes/No QA: On PubMedQA, HiLight highlights ESBLEC prevalence and E. coli load in wastewater to support the correct “Yes” answer.

C.1 Actor Model Architecture

The Actor uses a pretrained causal LLM backbone as a feature extractor, combines the last four layers, and applies lightweight policy and value heads for token-level emphasis learning. The backbone is frozen, with optional LoRA adaptation for efficient task-specific training.

  • Backbone LLM: The Actor runs a pretrained Qwen causal language model over token IDs and an attention mask to obtain hidden representations.KV caching is disabled to reduce overhead and keep the Actor as a feature extractor.
  • Backbone LLM: The backbone is frozen during Actor training, preventing gradient updates to the full language model.This reduces trainable parameters and improves training stability.
  • Backbone LLM: Optional LoRA adapters provide task-specific adaptation while updating only adapter parameters and lightweight Actor heads.The reported configuration uses rank r = 8, scaling factor α = 16, and dropout 0.05.
  • Layer mixing: Token features use a learned convex combination of the last four hidden layers, with softmax-normalized mixing weights.The paper reports that this aggregation often provides more robust saliency features than using only the final layer.
  • Policy and value heads: A policy head converts token-level logits into temperature-controlled Bernoulli probabilities, while a value head predicts a scalar sequence baseline.The policy defines token-wise importance probabilities for emphasis learning.
  • Policy and value heads: Masked mean pooling feeds the pooled representation into a small MLP to obtain the value estimate.

C.2 Data Preprocessing

The preprocessing constructs next-item recommendation examples from chronological user histories and builds co-visitation candidates using recency-weighted item associations. Candidate sets are completed with popular items when necessary, using 40 candidates by default.

  • Each user’s final purchased item is the supervision label, while the preceding interaction prefix forms the model input history.
  • The method builds a directed item-item co-visitation graph by adding edges between items appearing within a sliding window in each training history.
  • Candidate retrieval uses neighbors of the last M history items in reverse chronological order, with M = 10, emphasizing recent interactions.
  • Candidate scores apply exponential recency decay with α = 0.85, where rank(r) = 0 denotes the most recent item.
  • The top-K scored items form the co-visitation candidates; missing slots are filled with popular items, and K defaults to 40 for training and evaluation.
  • Other datasets use the default dataset loader from Hugging Face, while consolidated hyperparameters are reported in Table 15.

C.4 Training Settings

The training settings define task-specific inputs and prompts for recommendation and question answering, including emphasized spans and strict solver output formats. Baseline prompt-optimization procedures generate or refine instructions while preserving task-specific formatting constraints.

  • For Amazon-Beauty, the actor reads only user history, while the solver re-ranks candidates using emphasized history together with item metadata.
  • For question answering, the actor receives the question and evidence but highlights only evidence spans, leaving the question intact for the downstream model.
  • Recommendation solver prompts request candidate likelihood scores based on historical interactions and require well-formed JSON enclosed in <FINAL_JSON> tags.
  • Question-answering prompts require concise answers based only on the provided evidence and enforce answer wrappers such as <answer>...</answer> or <FINAL_ANSWER>...</FINAL_ANSWER>.
  • OPRO, PRL, DSPy, and APE-style baselines generate or refine task instructions while retaining strict output schemas and task-specific scoring requirements.
  • The QA instruction requires answers from evidence only, concise responses, and strict final-answer formatting, with prompt optimization informed by top instructions and failure cases.
  • The recommendation instruction scores every candidate from 0 to 10, encourages meaningful score spread, and uses user-history signals such as brand, category, and phrases.
  • APE-style recommendation instructions additionally require every candidate exactly once, valid JSON, meaningful score spread, history signals, and a length of no more than 120 words.
Loading 2604.22565v2…