PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane, Sudip Mittal, Shahram Rahimi

arXiv:2610.00562v1cs.CLcs.AIcs.LG

TL;DR

It remains unclear how context strategies compare under controlled conditions across reasoning tasks, distant evidence, and questions unsupported by the available history. The study compares context strategies for longitudinal clinical reasoning across answer correctness, evidence distance, and abstention on unsupported questions. Episodic and Hybrid generally outperform Recent and Full Context, while Recent degrades with greater evidence distance and Episodic and Hybrid retain higher accuracy at long distances.

  • Problem

    It remains unclear how context strategies compare under controlled conditions across reasoning tasks, distant evidence, and questions unsupported by the available history.

    Source: §II · p. 2

  • Method

    The study compares context strategies for longitudinal clinical reasoning across answer correctness, evidence distance, and abstention on unsupported questions.

    Source: §I · p. 2, F. Discussion · p. 8

  • Results

    Episodic and Hybrid generally outperform Recent and Full Context, while Recent degrades with greater evidence distance and Episodic and Hybrid retain higher accuracy at long distances.

    Source: F. Discussion · p. 8

  • Takeaways & Limitations

    Answer accuracy when evidence is available does not necessarily translate to successful abstention when history does not support the requested conclusion.

    Source: F. Discussion · p. 9, §VI · p. 9

    MedLoCoMo histories are synthetic dialogues derived from MIMIC-IV, and LLM-judged correctness had moderate agreement and was not validated against clinician annotations.

    Source: §VI · p. 10

Abstract

from arXiv · show

Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence distance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for reasoning.

I. INTRODUCTION

Longitudinal clinical reasoning must connect evidence scattered across patient histories, but longer context windows do not ensure that models use relevant information effectively. This study compares five context strategies and finds that retrieval-based approaches, especially Episodic and Hybrid, can outperform access to more history.

I. INTRODUCTION

  • Long-context models can remain sensitive to where relevant medical information appears, and extended histories pose challenges for multi-session and temporal reasoning.The introduction cites lost-in-the-middle effects and degradation over long multi-session histories.Source: §I · p. 1
  • The study evaluates context strategies across reasoning tasks, query-evidence distance, and abstention when history does not support a requested conclusion.It frames these dimensions as three research questions.Source: §I · pp. 1–2
  • Episodic and Hybrid generally outperform Recent and Full Context despite Full Context providing access to more patient history.The comparison examines five strategies on shared histories and queries.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Prior work examines longitudinal clinical reasoning, long-term LLM memory, and long-context effects, but controlled comparisons of context strategies remain limited. The paper identifies open questions about task fit, distant evidence, and unsupported questions.

II. RELATED WORK

  • Longitudinal benchmarks and medical dialogue studies evaluate cross-admission reasoning, but access to patient histories alone does not guarantee effective reasoning [1] [6].The cited studies include MedLoCoMo, LongMedBench, and MediLongChat.Source: §II · p. 2
  • Long-context research documents degradation over extended histories and the lost-in-the-middle effect, motivating selective context construction.Relevant information may be difficult to use even when it remains inside the context window.Source: §II · p. 2
  • Existing clinical studies often assess prompting, retrieval, or complete memory architectures, leaving limited controlled evidence on strategy effects across tasks, evidence distances, and unsupported questions.Prior memory systems can bundle representation, retrieval, consolidation, and updating, complicating isolation of context effects.Source: §II · p. 2

III. PROBLEM STATEMENT

The paper treats context strategy as the selection and organization of history supplied to an answer model, then evaluates its effects on longitudinal clinical reasoning. MedLoCoMo provides the shared histories and questions for comparing strategies.

III. PROBLEM STATEMENT

  • A context strategy selects, organizes, and provides history to the answer model; more history does not necessarily mean more useful evidence.The study focuses on correctness, robustness to evidence distance, and abstention under insufficient evidence.Source: §III · pp. 2–3

IV. METHODOLOGY

  • The evaluation compares five context strategies on MedLoCoMo and also tests generalization on LongMemEval.The general-domain benchmark results are reported separately.Source: §IV · p. 3

A. Datasets

  • MedLoCoMo contains 100 patient timelines with admission-level histories and evidence-linked questions spanning single- and cross-admission reasoning.All strategies operate on a common flat representation of the same histories.Source: A. Datasets · p. 3
  • The study samples 1,000 questions across seven scope and question-type combinations, including 286 adversarial questions with unsupported premises.The same question set is used across context strategies and answer models.Source: A. Datasets · p. 3

B. Context Strategies

The five strategies differ in whether they provide recent or full history directly, or retrieve encounter-level evidence, patient-specific facts, or both. Retrieval methods select information under an evidence budget.

B. Context Strategies

  • Full Context provides verbatim history up to the model’s context limit, while Recent Context keeps the latest raw sessions within the evidence budget.Neither uses relevance-based retrieval or compression.Source: B. Context Strategies · p. 3
  • Episodic retrieval ranks overlapping encounter-level text chunks by query similarity and selects the highest-ranked chunks within budget.Chunks are embedded independently.Source: B. Context Strategies · p. 3, Figure 2 · p. 4
  • Semantic retrieval extracts patient-specific facts with source-session and temporal metadata, then ranks those facts by query similarity.Facts are selected within the evidence budget.Source: B. Context Strategies · p. 3, Figure 2 · p. 4
  • Hybrid retrieval maintains episodic chunks and semantic facts separately, interleaves candidates by rank, and selects them under a shared budget.It combines encounter-level evidence with patient-specific facts.Source: B. Context Strategies · pp. 3–4, Figure 2 · p. 4

C. Fact Extraction and Caching

The study constructs semantic facts per session and reuses cached representations, then sends each strategy’s retrieved context and query through a structured answer-generation call. Figure 3 illustrates the five context strategies.

C. Fact Extraction and Caching

  • Semantic memory extracts up to ten patient-specific, self-contained facts per session, preserving clinical details such as temporal information, negation, and uncertainty.Each session is processed independently with Qwen3-4B; cached facts are reused across questions and conditions referencing that session.Source: C. Fact Extraction and Caching · p. 4

D. Answer Generation

  • Answer generation provides the retrieved context and query in one structured-output call, returning an answer and supporting-excerpt identifiers.Identifiers absent from the presented context are discarded before evaluation; prompts, retrieval, and memory construction are held fixed across answer models.Source: D. Answer Generation · pp. 4–5
  • Figure 3 presents an illustrative example of the five context strategies.Source: Figure 3 · p. 5

E. Evaluation

The evaluation compares all context strategies on the same questions, measuring answer correctness, sensitivity to evidence distance, and abstention on unsupported questions. Supplementary metrics assess answer similarity and evidence grounding.

E. Evaluation

  • All conditions use the same sampled questions for paired comparisons, with semantic equivalence to the gold answer judged by a different model family.Generation failures count as incorrect; Gemma4 is the primary judge and MedGemma is the secondary judge, with agreement reported using raw agreement and Cohen’s κ.Source: E. Evaluation · p. 5
  • Evidence distance is the session gap from supporting evidence to the history’s end; a logistic regression tests whether its association with correctness differs by strategy.The model uses continuous standardized session gap, strategy indicators, and question-level cluster-robust standard errors; predefined distance bins are descriptive only.Source: E. Evaluation · p. 5
  • For adversarial questions with unsupported premises, the evaluation tests whether responses reject the premise or indicate that the history cannot answer it.Responses are judged against the benchmark’s unanswerable gold response using the RQ1 semantic correctness judge.Source: E. Evaluation · p. 5
  • Supplementary metrics include BERTScore F1 and scorable-response coverage, plus citation coverage, precision, recall, and F1 against gold evidence sessions.BERTScore excludes failed or empty generations; citation metrics are computed per question and averaged, with zero scores when nothing is cited.Source: E. Evaluation · p. 5

F. Implementation

The experiments run locally on four open-weight answer models, with context budgets and shared procedures varying by strategy.

F. Implementation

  • Experiments ran locally on an NVIDIA GeForce RTX 5080 workstation with 64 GB RAM and a 24-core Intel Core Ultra 9 285K processor.The system used Ubuntu 24.04.4 LTS and Ollama.Source: F. Implementation · p. 6
  • The answer models were qwen3-8b, mistral-7b, llama3.2-3b, and glm4-9b.Source: F. Implementation · p. 6
  • Full Context used each model’s available context window, while Recent and retrieval-based strategies shared a 7,000-token evidence budget.Semantic memory used qwen3-4b; episodic and semantic memories used all-MiniLM-L6-v2 embeddings ranked by cosine similarity, with answer prompts and decoding fixed per model.Source: F. Implementation · p. 6

V. RESULTS & DISCUSSION

Hybrid and Episodic provide the strongest aggregate answer accuracy, though their relative advantage varies by answer model. Evidence-grounding and answer-similarity measures offer complementary views of strategy performance.

V. RESULTS & DISCUSSION

  • Full Context results require caution because Qwen and Mistral have 32k-token windows, whereas LLaMA and GLM-4 have 128k-token windows.The other four strategies use the same evidence budget across models.Source: §V · p. 6

A. Overall Effect of Context Strategy

  • Hybrid had the highest accuracy for Qwen (52.6%), LLaMA (46.0%), and GLM-4 (55.5%); Episodic led for Mistral (62.0%), closely followed by Hybrid (61.4%).The two correctness judges agreed on 78.7% of evaluations (Cohen’s κ = 0.575), and the main strategy-level patterns were preserved with MedGemma.Source: A. Overall Effect of Context Strategy · p. 6
  • BERTScore F1 ranged from 0.83 to 0.87, with smaller strategy differences than judged accuracy; Qwen Recent and Hybrid scored 0.86 and 0.87 versus 31.6% and 52.6% judged accuracy.Semantic similarity alone did not capture the context-dependent differences in answer correctness; Mistral Full Context had 17.9% BERTScore coverage.Source: A. Overall Effect of Context Strategy · p. 6
  • Episodic achieved the highest citation precision (48.7%), recall (50.0%), and F1 (46.1%), while Hybrid led citation coverage (87.9%).Full and Recent Context had substantially lower citation precision and recall, indicating weaker alignment between cited and annotated evidence sessions.Source: A. Overall Effect of Context Strategy · p. 6, Table III · p. 6

B. Performance Across Reasoning Tasks

No context strategy leads across every reasoning task and model. Recent Context degrades as supporting evidence grows more distant, while answer accuracy and abstention both vary by strategy.

B. Performance Across Reasoning Tasks

C. Effect of Query-Evidence Distance

  • 83.8% to 36.1%: Recent Context’s Qwen accuracy falls from 0–2 to 21+ sessions; at 21+ sessions, Hybrid scores 61.1% for Qwen, 50.3% for LLaMA, and 59.6% for GLM-4, while Episodic scores 62.8% for Mistral.The Recent Context decline persists when analysis is restricted to successful generations; most strategy-by-distance interactions are not significant.Source: C. Effect of Query-Evidence Distance · pp. 6–7

D. Abstention on Unanswerable Questions

  • Hybrid has the highest answerable-question accuracy (63.1%) while retaining 30.9% accuracy on unanswerable questions; Semantic leads on unanswerable questions (33.5%).Semantic’s answerable-question accuracy is 50.2%, below Hybrid and Episodic; the strategy with the highest unanswerable-question accuracy varies across models.Source: D. Abstention on Unanswerable Questions · pp. 7–8

E. Cross-Benchmark Evaluation on LongMemEval

On LongMemEval, context strategy continues to affect accuracy, but results vary by model. The broader findings emphasize that evidence selection, distance, and reasoning demands all shape long-horizon performance.

E. Cross-Benchmark Evaluation on LongMemEval

  • On LongMemEval, Hybrid scores 45.4%, 41.6%, 43.8%, and 57.6% for Qwen, LLaMA, GLM-4, and Mistral, while Recent Context scores 9.0%, 7.6%, and 16.0% for the first three models.Episodic scores 41.2%, 36.8%, 44.2%, and 61.8%, respectively; Recent Context scores 66.4% for Mistral, illustrating model-dependent variation.Source: E. Cross-Benchmark Evaluation on… · p. 8, Figure 6 · p. 9

F. Discussion

  • Episodic and Hybrid generally outperform Recent and Full Context, despite Full Context providing access to more patient history.The discussion suggests selective retrieval can be more effective than simply increasing the context available to the answer model.Source: F. Discussion · p. 8
  • Recent Context degrades as evidence grows more distant, whereas Episodic and Hybrid retain higher accuracy at long distances, although their distance slopes do not differ significantly from other strategies.Strategy effectiveness also varies with the reasoning task and answer model; Semantic leads on adversarial questions, while Hybrid combines the highest answerable-question accuracy with relatively strong adversarial performance.Source: F. Discussion · pp. 8–9

VI. CONCLUSION

The study finds that reliable longitudinal reasoning depends on how relevant evidence is selected and presented, not only on how much history is available. Its conclusions are bounded by the evaluated models, retrieval settings, synthetic histories, and LLM-based judging.

VI. CONCLUSION

  • Across five context strategies and multiple open-weight models, Episodic and Hybrid generally achieve stronger answer accuracy, including on distant evidence.Strong performance when evidence is available does not necessarily translate to successful abstention when it is unsupported.Source: §VI · p. 9
  • Evaluation uses four locally served 3–9B open-weight models, a fixed set of retrieval settings, one embedding model, and a 7,000-token evidence budget.Results may differ for larger models or other configurations.Source: §VI · pp. 9–10
  • MedLoCoMo histories are synthetic dialogues derived from MIMIC-IV records, and LLM-judge agreement was moderate without validation against clinician annotations.Semantic facts were extracted with Qwen3-4B, which shares a model family with one answer model.Source: §VI · p. 10
  • Future work will investigate adaptive context strategies and evaluate reliability in more realistic clinical settings.Source: §VI · p. 10
Loading 2610.00562v1…