PaperlayerView exact revision on arXiv

Source-linked AI summary

Inhibitory Attention for Clinical Long-Context Reasoning: Characterizing and Mitigating Lost-in-the-Middle Effects in EHR Processing

Sanjay Basu

arXiv:2608.20348v1cs.CLcs.AI

TL;DR

The paper examines whether language models reliably retrieve clinically important information from any position in very long EHRs and characterizes the clinical lost-in-the-middle problem. It introduces QCCS, a query-conditioned sentence-selection gate, and finds that query-aligned context quality better predicts instruction accuracy than raw gold-sentence retrieval.

  • Problem

    Longitudinal EHRs can exceed 100,000 tokens, raising whether models faithfully retrieve information from every position rather than privileging some locations.

  • Method

    QCCS scores sentences in a chronological EHR stream against a clinical query, retains the top-k sentences plus recent events, and passes this compressed context to an LLM.

  • Results

    25.0% accuracy came from QCCS on instructions where it did not retrieve the gold sentence lexically, while retrieval arms reached at most 2.6% when they did retrieve it.

  • Takeaways & Limitations

    Context quality and query alignment, rather than raw gold-sentence presence, are the operative constraint for EHR instruction-following in this evaluation.

  • Takeaways & Limitations

    The Stage 2 evaluation is a proof-of-concept pilot with N=83, wide bootstrap confidence intervals, and differences between arms that are not individually statistically significant.

Abstract

from arXiv · show

Electronic health records now routinely exceed 100,000 tokens per patient. Yet large language models exhibit the lost-in-the-middle (LitM) effect: information near the center of a long context is retrieved less reliably than information near the edges. In clinical use this is not benign: the single most consequential fact in a note can sit at its center. We term this the clinical lost-in-the-middle (CLitM) problem, give its first systematic characterization using MedAlign, and compare context-selection strategies as remedies. Across 2,196 instruction-response pairs and six language models, we observe a 21.9 percentage-point gap between peak accuracy (59.5%, 95% CI [46.3, 71.0], 20-30% decile) and trough accuracy (37.6% [23.2, 52.5] at 70-80%); 67.8% of reference answers fall between the 10th and 90th percentiles of the EHR timeline, inside the CLitM trough. We introduce Query-Conditioned Clinical Suppression (QCCS), a lightweight query-conditioned selection gate, and evaluate it against BM25, BM25 with section-header filtering, dense retrieval, and cross-encoder reranking (N=83 held-out instructions). With Qwen2.5-7B-Instruct (16k context), QCCS outperforms all five comparators under LLM-as-judge scoring: for middle-position instructions QCCS reaches 16.7% versus BM25 3.3%, cross-encoder 0.0%, dense 0.0%, and full context 6.7%; overall QCCS reaches 25.3% versus at most 3.6% for retrieval-only comparators. This advantage is not explained by retrieval recall: at k=20, BM25 retrieves the gold evidence sentence in 98.8% of instructions (QCCS 34.9%), yet retrieval arms stay at most 2.6% accurate even when they retrieve it, whereas QCCS reaches 25.0% even when it does not. In this proof-of-concept evaluation, query-aligned context selection predicts EHR instruction-following accuracy better than gold-sentence retrieval recall.

1 INTRODUCTION

Longitudinal EHRs can exceed 100,000 tokens, making reliable retrieval across every timeline position a central clinical question. The paper frames the resulting positional retrieval failure as clinical lost-in-the-middle and proposes inhibitory attention as a principled remedy.

  • Clinical motivation: EHRs can exceed 100,000 tokens when longitudinal clinical information is concatenated into one record.Such records include encounter notes, laboratory results, medication histories, and social history entries.
  • Clinical motivation: The central question is whether models retrieve information faithfully from every position in a long patient history.The paper specifically asks whether attention systematically privileges some positions over others.
  • Clinical lost-in-the-middle: LitM describes lower retrieval accuracy for information near the center of long contexts than near their edges, producing a U-shaped accuracy curve.The paper attributes this pattern to preferential attention toward sequence beginnings and ends.
  • Clinical lost-in-the-middle: Clinical use makes positional retrieval failures consequential because relevant medication information may occur anywhere in a multi-year record.Existing retrieval, compression, and KV-cache methods address related problems but not the core architectural failure described here.
  • Contributions: The paper contributes systematic CLitM characterization, inhibitory-attention evaluation, and Query-Conditioned Clinical Suppression as a lightweight context-selection gate.It also reports failures of per-token QCCS-DiffAttn gating under class imbalance in structured EHR prediction.

2 RELATED WORK

Prior work establishes lost-in-the-middle behavior and inhibitory-attention mechanisms, while clinical long-context studies and query-conditioned selectors leave positional-bias correction in single-record EHRs insufficiently evaluated.

  • Lost in the Middle: Prior LitM studies reported U-shaped accuracy curves and 15–25 percentage-point troughs primarily in synthetic or multi-document settings.The paper identifies the absence of a prior clinical EHR study as the relevant gap.
  • Lost in the Middle: Mechanistic prior work links LitM to an attention basin, while EHR studies establish long-context baselines and report an 8.3% GPT-4 accuracy drop with context length.These works motivate testing positional effects in clinical records.
  • Inhibitory attention: Differential Transformer and related inhibitory mechanisms use trainable or negative attention to suppress copied or uninformative content.Their prior targets include generality and sparsity rather than positional-bias correction in clinical contexts.
  • Context selection: KV-cache compression and prompt-compression methods operate after attention, while query-conditioned selectors and sentence rerankers frame relevance differently.The paper evaluates compression, temporal reordering, and query-conditioned selection for positional-bias correction in single-record EHRs.

3 METHODS

The study combines MedAlign and EHRSHOT experiments with inhibitory-attention models and a query-conditioned sentence gate, then separates evidence retrieval from end-to-end LLM re-inference.

  • Datasets: MedAlign supplies 983 instructions across 275 patients for CLitM characterization and QCCS evaluation, while EHRSHOT supplies 6,739 patients and 15 binary prediction tasks.The experiments use each dataset under its respective data-use agreement.
  • CLitM characterization: The CLitM experiment labels answer positions in chronological MedAlign event streams and evaluates 2,196 model instances across six language models.Accuracy measures instruction following against clinician-authored references, distinct from retrieval recall.
  • Structured prediction: EHRSHOT compares standard attention, Differential Transformer, QCCS-DiffAttn, and H2O on laboratory abnormality prediction using AUROC and AUPRC.H2O is adapted with embedding norms as a heuristic heavy-hitter proxy in the discriminative setting.
  • QCCS selection: QCCS scores chronological EHR sentences using query-conditioned character n-gram representations and retains top-ranked sentences plus recent events for LLM inference.The gate is independent of the downstream LLM’s attention mechanism.
  • Inhibitory integration: The QCCS-DiffAttn extension replaces scalar inhibition with per-token weights that scale attention to key positions across heads.The study evaluates both scalar DiffAttn and the per-token gated integration.
  • Robustness check: Focal BCE is tested as a robustness check for gradient starvation under extreme class imbalance in per-token gate training.The check uses γ = 2 and positive-class reweighting.
  • Evidence retrieval and re-inference: Experiment 3 evaluates gold-answer retrieval at k=20 and then end-to-end re-inference on 83 held-out instructions using six context strategies and LLM-as-judge scoring.Retrieval hits use lexical overlap, with semantic entailment as a validation criterion.

4 RESULTS

The results show a pronounced clinical lost-in-the-middle curve and a strong separation between retrieval recall and reasoning accuracy. QCCS performs best in end-to-end evaluation despite retrieving the gold evidence less often than retrieval baselines.

  • CLitM characterization: 59.5% peak accuracy at the 20–30% position decile falls to 37.6% at 70–80%, a 21.9 percentage-point CLitM gap.The reported clustered 95% CI for the gap is [17.7, 45.2] pp.
  • Structured prediction: DiffAttn improves AUROC by 6.1 pp on anemia and 4.2 pp on hyperkalemia but decreases it by 1.9 pp and 1.5 pp on balanced tasks.The mean Differential-versus-Standard difference across four tasks is +1.7 pp AUROC.
  • Structured prediction: Per-token QCCS-DiffAttn does not improve over scalar DiffAttn, with gate training collapsing to chance under severe hyperkalemia imbalance.Focal BCE recovers +10.9 pp for hyperkalemia but remains 0.6 pp below scalar DiffAttn.
  • Retrieval recall: 98.8% BM25 lexical recall at k=20 contrasts with QCCS’s 34.9% recall on the 83-instruction held-out split.BM25 semantic recall is 45.8%, substantially below its lexical recall.
  • End-to-end re-inference: 25.3% overall QCCS LLM-as-judge accuracy exceeds the at-most-3.6% comparator accuracy, while middle-position QCCS reaches 16.7%.Middle-position BM25 reaches 3.3%, full context 6.7%, and dense and cross-encoder arms 0.0%.
  • Evaluation: Semantic LLM-as-judge scoring is necessary because token-overlap scoring inflates apparent performance for dense and BM25-filtered contexts.The paper uses semantic evaluation to identify false positives from lexical overlap.
  • Recall versus reasoning: Retrieval baselines remain at most 2.6% accurate even when they retrieve the gold sentence, whereas QCCS reaches 25.0% without retrieving it lexically.The conditional analysis supports query-aligned context as the operative constraint in this evaluation.

5 DISCUSSION

The discussion presents QCCS as the strongest evaluated remedy for clinical lost-in-the-middle effects, while showing that its advantage reflects query-aligned context quality rather than simple evidence retrieval. The authors also emphasize robustness, failure-mode evidence, and important deployment limitations.

  • QCCS reaches 25.3% overall accuracy versus at most 3.6% for all comparators.It is the only arm with consistently positive LLM-as-judge accuracy across position bands.
  • QCCS dominates every position band, including early-context instructions where it reaches 38.5% versus 0.0% for the comparator.The gate architecture remains robust across 27 variants within 6.25 percentage points.
  • Wrong-entity confabulation accounts for 64.2% of BM25 failures, while QCCS recovers correctness on 24.7% of the same cases.The failure analysis attributes errors to distractor anchoring rather than outright refusal.
  • QCCS reaches 25.0% accuracy even when it does not retrieve the gold sentence, compared with 2.6% for BM25 conditional on gold retrieval.The authors attribute this pattern to query-aligned, temporally anchored multi-event context rather than raw gold-sentence presence.
  • Map-reduce, structure-preserving retrieval, and MMR-diversified selection underperform QCCS, while oracle-blind judging and a larger Qwen model preserve its advantage.The reported limitations include single-site data, N=83 Stage 2 evaluation, QCCS training circularity, and no human evaluation.
  • The demonstrated accuracy remains insufficient for unsupervised clinical decision-making, so human oversight remains essential.The paper frames positional-bias mitigation as necessary but not sufficient for safe clinical AI deployment.

B LIMITATIONS

The paper’s limitations span dataset scope, evaluation uncertainty, model and task heterogeneity, baseline coverage, training-criterion circularity, and context-truncation artifacts. These constraints limit generalizability and clinical-readiness claims.

  • Data and generalizability: MedAlign’s 275 patients come from one academic medical center, limiting statistical power and generalizability across documentation styles, languages, and care settings.
  • Evaluation uncertainty: The Stage 2 evaluation is a proof-of-concept pilot with 83 instructions, wide bootstrap intervals, and differences between arms that are not individually statistically significant.
  • Task-type heterogeneity: The 83 held-out instructions mix retrieval, reasoning, and longitudinal summarization tasks, which may benefit differently from sentence selection.The reported 10.8% oracle accuracy reflects that multi-sentence integration can require more context than one gold event.
  • Reader model and evaluation: Results use Qwen2.5-7B-Instruct; larger or domain-tuned readers may produce higher accuracy or a different relative ordering.
  • Human evaluation and clinical deployment: No clinician evaluation was conducted, and the 25.3% QCCS accuracy is insufficient for autonomous clinical decision support.Prospective validation with practicing clinicians and diverse EHR systems is required before deployment.
  • Baseline and methodological scope: The study omits stronger neural rerankers and acknowledges that full-context truncation can bias comparisons toward compressed arms.The QCCS gate’s lexical-overlap training criterion also creates structural circularity, while H2O uses a non-equivalent embedding-norm proxy.

G FULL EHRSHOT RESULTS INCLUDING VALIDATION AUROC

The EHRSHOT analysis reports test AUROC and AUPRC across laboratory prediction tasks and attention conditions, with validation AUROC provided separately for reference. Hypoglycemia is excluded because its H2O run exceeded the wall-clock limit.

  • Validation reference: Validation AUROC values are reported separately for Standard, Differential, H2O, and QCCS-DiffAttn conditions, with em dashes marking inapplicable entries.
  • Test-set comparisons: Differential Transformer exceeds standard attention on Anemia by +6.0 AUROC points and Hyperkalemia by +4.2 points, while H2O wins Hyperkalemia AUROC.
  • Test-set comparisons: Figure A3 compares test AUROC and AUPRC across four completed EHRSHOT laboratory abnormality tasks and three attention conditions.
  • Scope of reported tasks: Hypoglycemia is excluded from Figure A3 because the H2O run exceeded the 7,200-second wall-clock limit.

H QUERY CONDITIONING ABLATION

The query-conditioning ablation tests whether QCCS benefits from question context and contrasts retrieval-oriented metrics with judge-based answer accuracy. Query conditioning improves gate recall, while token-overlap can substantially overstate comparator performance.

  • Query-conditioned gate: +14.7 percentage points overall and +23.8 points for middle-position recall result from adding query conditioning to the QCCS gate.The ablation uses 690 test rows before deduplication.
  • Metric comparison: Token-overlap inflates BM25-filtered and Dense scores in the 30–50% position band relative to the primary LLM-as-judge metric.BM25-filtered reaches 22.2% and Dense 33.3% by token overlap versus 0.0% by judge.
  • Metric comparison: Cross-encoder token-overlap is 9.6% overall versus 1.2% by judge and reaches 44.4% versus 0.0% in the 30–50% band.

I STAGE-2 ACCURACY CONDITIONAL ON STAGE-1 RETRIEVAL OUTCOME

Stage-2 accuracy remained low for retrieval baselines even when Stage 1 retrieved the gold evidence, while QCCS performed similarly with or without a lexical hit.

  • 2.6% BM25 accuracy with a gold-sentence hit contrasts with 0.0% when the hit is absent.
  • 25.0% QCCS accuracy without a gold-sentence hit was comparable to 22.2% when the hit was retrieved.QCCS’s accuracy was essentially unchanged across the two retrieval conditions.
  • 11.0% oracle accuracy with gold-sentence-only context exceeded BM25’s hit-conditioned accuracy but remained below QCCS’s 25.3%.The oracle used the gold sentence plus the five most recent events.
  • The primary judge classified responses as correct only when they conveyed the gold evidence’s factual answer, rejecting refusals or contradictions.The judge received the question, gold EHR evidence, and AI response.

L QCCS-DIFFATTN GATE: FOCAL BCE ROBUSTNESS CHECK

A focal-BCE robustness check shows that loss engineering helps under extreme class imbalance but does not generally rescue the QCCS-DiffAttn per-token gate.

  • ≤1 percentage-point changes across three tasks with positive prevalence of at least 28% support an architectural explanation for gate failure.The tasks were anemia, hyponatremia, and thrombocytopenia.
  • +10.9 percentage points on hyperkalemia raised QCCS-Focal from 0.500 to 0.609 under 2.38% positive prevalence.Extreme imbalance therefore permits partial recovery through focal BCE.
  • 0.609 QCCS-Focal remained below scalar DiffAttn at 0.615, while gate standard deviation stayed below the non-degenerate threshold.The reported gate standard deviation was 0.041 versus the 0.05 diagnostic threshold.

M SEMANTIC VALIDATION OF STAGE 1 HIT CRITERION

Semantic NLI validation shows that lexical retrieval hits often lack entailment, yet BM25’s downstream failure persists while QCCS remains effective without requiring semantic hits.

  • 54.2% of BM25 lexical hits were not confirmed as semantic entailments by NLI.BM25 semantic recall was 45.8%, versus 98.8% lexical recall.
  • 0.0% BM25 Stage-2 accuracy occurred even among 38 instructions with NLI-confirmed semantic entailment.This was stronger than the 2.6% accuracy conditional on lexical retrieval hits.
  • 60.0% QCCS Stage-2 accuracy occurred on 10 semantic-hit instructions, versus 20.5% on 73 semantic-miss instructions.QCCS therefore remained effective when retained sentences did not semantically entail the gold answer.

R TASK-TYPE STRATIFICATION OF STAGE-2 ACCURACY

Across task types and selection variants, QCCS’s advantage was largest for longitudinal summarization, while temporal ordering and diversity alone did not improve retrieval baselines.

  • Task-type stratification: 34.8% QCCS versus 0.0% BM25 was the largest task-type advantage in longitudinal summarization.The analysis covered binary retrieval, multi-step reasoning, and longitudinal summarization.
  • Task-type stratification: 14.8% QCCS accuracy for multi-step reasoning was below 27.3% for binary retrieval.Enumeration questions may require evidence from multiple positions that the gate can partially miss.
  • Limitations: The task-type patterns are preliminary because each category has only 23–33 examples and uses keyword-based classification.Larger splits and finer-grained annotations are needed for more definitive task-type analysis.
  • Temporal ordering: 1.2% BM25-temporal accuracy fell below standard BM25’s 2.4%, while QCCS reached 25.3%.Both contexts were presented chronologically, isolating selection rather than presentation order.
  • Diversity-aware retrieval: 2.4% MMR accuracy matched dense retrieval and remained far below QCCS.Diversifying dense retrieval therefore did not materially improve Stage-2 performance.

V QCCS GATE ARCHITECTURE ABLATIONS

QCCS architecture ablations show stable high recall across most tested configurations, while sparse attention produces near-total inhibition without materially improving classification performance.

  • QCCS architecture ablations: All 27 QCCS architecture combinations were evaluated across n-gram size, embedding dimension, and MLP depth.The production configuration uses ngram=3, embed_dim=64, and the standard MLP.
  • QCCS architecture ablations: 25 of 27 variants achieved ≥95% recall, with a 6.25 pp spread from 93.75% to 100.0%.N-gram size and embedding dimension showed no meaningful effect within the tested ranges.
  • QCCS architecture ablations: Shallow MLPs averaged 96.3% recall versus at least 99.2% for deeper alternatives, while the production standard MLP remained in the stable high-recall region.The positive–negative score gap was lower for shallow MLPs (0.261) than standard (0.294) or deep (0.300) models.
  • Sparse attention alternatives: Entmax15 and sparsemax produced 98.3–99.5% exact-zero attention sparsity, but AUROC changed by no more than 0.015 versus softmax.Both sparse variants occasionally underperformed softmax, including sparsemax on hyperkalemia (0.609 versus 0.632).
  • Sparse attention alternatives: The results indicate that sparse projection alone does not substitute for query-guided context compression in addressing CLitM ordering bias.The paper contrasts projection-based sparsity with QCCS pre-selection of query-relevant context.
Loading 2608.20348v1…