PaperlayerView exact revision on arXiv

Source-linked AI summary

ClinTraceBench: Source-Verifiable Longitudinal Clinical Reasoning over EHR-Derived Dialogues

Huimin Wang, Zhengyi Zhao, Yutian Zhao

arXiv:2609.01111v1cs.CL

TL;DR

ClinTraceBench asks whether compact history representations preserve the longitudinal signal needed for clinical reasoning over multi-visit EHR trajectories. It builds a source-verifiable benchmark and evaluates eight strategies across four backbones, finding relation loss in compressed memories, task-specific aggregation and abstention failures, and a cost-quality reversal favoring Haiku over Sonnet under full context.

  • Problem

    Whether retrieval, structured timelines, summaries, and agentic memories preserve sufficient signal for longitudinal clinical reasoning has not been measured.

  • Method

    The paper constructs 385 MIMIC-IV-derived verified dialogues with event-ID provenance and evaluates eight history representations across four backbones on a fixed nine-task benchmark.

  • Results

    Compressed strategies lose the finding–diagnosis relation even when given the attribution sentence before construction, while full context is the accuracy upper bound and Haiku is cheaper and slightly more accurate than Sonnet on its full-context Pareto frontier.

  • Takeaways & Limitations

    History representation choice creates task-specific preservation and aggregation trade-offs, so memory-equipped clinical agents should be evaluated across longitudinal tasks and the cost–quality frontier.

  • Takeaways & Limitations

    The dialogues are LLM-generated from structured MIMIC-IV fields, so benchmark comparisons are supported but transfer to real clinical conversations is not established.

Abstract

from arXiv · show

Clinical LLM assistants must reason over multi-visit patient trajectories, yet whether the compact history representations used to scale them---retrieval, structured timelines, LLM summaries, agentic memory---preserve the longitudinal signal clinical reasoning needs has not been measured. We introduce ClinTraceBench: 385 MIMIC-IV-derived verified dialogues with event-ID provenance, a nine-task taxonomy (T1--T9), and L0--L4 deterministic + L5 human-audit validation (98.92\% agreement). We evaluate eight history representation strategies---a no-context floor, \textit{last-visit-only}, \textit{full-context}, BGE-M3 \textit{dense-retrieval}, two compression schemes, and two agentic-memory systems (\textit{Mem0}, \textit{A-Mem})---across four backbones (DeepSeek-V3, GPT-4o-mini, Haiku~4.5, Sonnet~4.6) on 6{,}271 questions: 32 cells, 200{,}672 predictions. Four findings: (SP4) a controlled T3 injection probe isolates compression-induced \textit{relation} loss---with the attribution sentence present \textit{before} construction, \textit{Mem0}, \textit{A-Mem} and \textit{llm-summary} still recover only 0--5.3\% of the injected positives; (SP1) compressed strategies pay an aggregation tax on multi-visit trends and cross-patient comparisons; (SP2) the blind-to-full gap spans $+29.8$~pp (GPT-4o-mini) to $+62.7$~pp (Haiku); (SP3) abstention scales non-monotonically with context length. On the Pareto frontier Haiku dominates Sonnet under \textit{full-context} (\$25.76 vs.\ \$106.21), inverting the ``biggest backbone wins'' heuristic.

1 Introduction

ClinTraceBench addresses whether compact history representations preserve the longitudinal signal needed for clinical reasoning. It contributes a verified, source-anchored benchmark spanning diverse clinical operations and representation strategies.

  • Existing clinical benchmarks mostly use single-encounter or single-document prompts, leaving multi-visit aggregation, encounter-local linkage, and abstention under unstated facts insufficiently tested.
  • ClinTraceBench contains 385 MIMIC-IV-derived verified dialogues with event-ID provenance, nine clinical reasoning tasks, and 98.92% L0–L5 validation agreement.
  • The evaluation compares eight history representation strategies across four frontier backbones on 6,271 questions, producing 32 cells and 200,672 predictions.
  • A controlled T3 injection probe tests whether compressed representations preserve a finding–diagnosis relation when the attribution sentence is present before construction.
  • The paper reports full context as strongest on pooled accuracy, while dense retrieval remains competitive at lower cost and compressed or agentic memories show task-specific failures.

2 Related work

Prior benchmarks separately address clinical reasoning, temporal reasoning, long-context evaluation, or conversational memory. ClinTraceBench integrates these concerns through verified provenance, a broad task taxonomy, and controlled preservation testing.

  • Existing EHR-derived clinical evaluations are dominated by single-document or single-encounter prompts and do not stress multi-visit aggregation, encounter-local attribution, or abstention under unstated facts.
  • Temporal benchmarks evaluate longitudinal reasoning but lack event-level provenance from answers to dialogue turns and controlled probes separating signal-preservation failures from backbone reasoning capacity.
  • Long-context and retrieval benchmarks characterize attention drift and calibration, while ClinTraceBench operationalizes abstention and representation comparison in a clinical setting.
  • Open-domain memory benchmarks motivate agentic-memory methods but lack clinical semantics and event-level source provenance for tracing failures to specific dialogue turns.
  • The paper’s contribution is the integration of longitudinal EHR trajectories, verified dialogue provenance, nine reasoning axes, staged validation, and a controlled-injection preservation probe.

3 Benchmark design

The benchmark converts stratified MIMIC-IV patient records into verified multi-visit dialogues and evaluates nine tasks designed around distinct longitudinal reasoning demands. Its fixed, validated question set supports paired strategy comparisons.

  • Task design: The nine tasks span preservation/access, integrative reasoning across visits or patients, and diagnostic or epistemic probes for contradiction handling and abstention.
  • Task design: T3 tests same-encounter finding–problem attribution in post-construction staleness and pre-construction equal-input preservation settings.
  • Cohort and construction: Figure 2’s pipeline samples 400 balanced, complexity-stratified MIMIC-IV patients and produces 385 verified dialogues with deterministic event IDs and 98.92% audit agreement.
  • Evaluation: T9 contains only insufficient-information gold answers, making abstention rate its accuracy and requiring relative rather than absolute interpretation because always-abstain scores 100%.
  • Evaluation: The fixed evaluation set contains 6,271 questions across T1–T9, enabling paired McNemar tests while leaving T8 weakly powered and T4 under-powered.
  • Evaluation: Pooled accuracy is dominated by the unequal sizes of T1 and T9, so macro accuracy and per-task results are needed for task-balanced comparisons.

4 History representation strategies and backbones

The study compares eight context-access strategies, from no context and recency baselines to full dialogue, retrieval, compression, and agentic memory. Four frontier backbones answer questions, while memory artifacts are prepared once under a fixed model.

  • History representations: The eight strategies cover no-context blind, last-visit-only, full-context, dense retrieval, structured timelines, LLM summaries, and two agentic-memory systems.
  • History representations: Full-context feeds the entire multi-visit dialogue verbatim, whereas dense retrieval selects the top five complete clinical visits using BGE-M3 embeddings.
  • Backbones: The four answering backbones are DeepSeek-V3, GPT-4o-mini, Claude Haiku 4.5, and Claude Sonnet 4.6, with context windows ranging from 64k to 200k.
  • Agentic memory: Mem0 and A-Mem extract artifacts once before questioning, using DeepSeek-V3 as the fixed preparation model across all answering backbones.

5 Results

Across the strategy–backbone–task cube, full-context leads overall, while compressed representations lose performance especially on integrative longitudinal tasks. Controlled probes further distinguish aggregation, abstention, relation-preservation, and update-staleness failure modes.

  • Main accuracy: Full-context ranks first on every backbone, while dense-retrieval follows within 1–4 pp and structured-timeline remains third and backbone-invariant.The ordering survives alternative macro-accuracy checks and exclusions of selected diagnostic tasks.
  • SP1: The aggregation bottleneck: Compressed representations incur an aggregation tax on multi-visit trends, cross-patient comparisons, and treatment-response tasks.Agentic-memory systems omit per-visit value pairs, summaries cannot encode full trajectories, and retrieval can fail across non-contiguous chunks.
  • SP2: Backbone-specific extraction profiles: Haiku gains 62.7 pp from blind to full-context, whereas GPT-4o-mini gains 29.8 pp, showing backbone-specific context extraction profiles.Haiku has the lowest blind score but ties for the highest full-context score; GPT-4o-mini has the narrowest lift.
  • SP3: Non-monotonic abstention under longer context: Last-visit-only abstention exceeds full-context on every backbone, so abstention changes non-monotonically as dialogue context expands.The paired pooled comparison is 0.924 versus 0.763, with ∆= 0.161 and McNemar p = 1.7 × 10^-38.
  • SP4: Controlled preservation probe: With the attribution sentence present before construction, Mem0 and A-Mem recover at most 1/19 positives, while every llm-summary cell recovers 0/19.Inspection indicates compression usually retains the lab fact but omits the finding–diagnosis relation.
  • SP4: Controlled preservation probe: Post-construction injection leaves full-context, last-visit-only, and dense-retrieval intact, but four upstream-built strategies collapse to exactly 0.500.This secondary probe measures update staleness rather than discarding, because derived representations cannot contain a sentence introduced afterward.
  • SP5: Retrieval, cost, and Pareto efficiency: Dense retrieval reaches the Pareto frontier, while full-context × Haiku dominates full-context × Sonnet at $25.76 versus $106.21.Structured-timeline never reaches the frontier despite competitive accuracy because its per-call cost is not offset by sufficient compression.
  • Diagnostic interpretation: Aggregate accuracy can obscure distinct failure modes, including constant-No collapse, over-answering, trend failure, and cross-patient confusion.The paper uses task-level decompositions and case studies to preserve these diagnostic distinctions.

6 Discussion

Compact representations preserve locally retrievable information but lose relational and multi-visit structure, while cost-aware evaluation favors Haiku over Sonnet under full context.

  • What compact representations get right: Dense retrieval lands within 1–4 pp of full-context pooled accuracy and within 0–9 pp on every task except T4.BGE-M3 surfaces answers reliably when they are chunk-localizable.
  • What compact representations get wrong: Compressed strategies pay an aggregation tax because preprocessing discards per-visit value pairs needed for trends and cross-patient comparisons.This affects T2, T6b, T6c, and exploratory T8.
  • What compact representations get wrong: 0–5.3% of injected T3 positives are recovered by Mem0, A-Mem, and llm-summary even when the attribution sentence is present during construction.The underlying lab fact is typically retained; the finding–diagnosis relation is lost.
  • Cost flips “biggest backbone wins”: Full-context × Haiku dominates full-context × Sonnet on the cost–accuracy frontier at $25.76 versus $106.21 and 0.706 versus 0.698.This reverses the biggest-backbone-wins heuristic.
  • Memory size ≠ accuracy: Mem0 character budget is uncorrelated with patient-level T1 accuracy, with Pearson r = −0.129.The paper attributes greater importance to artifact architecture than to memory size.
  • Future work: Future work targets multi-EHR replication, longer horizons, dynamic updates, real longitudinal notes and conversations, and broader controlled-probe coverage.The authors also plan to rebalance T5 and enlarge T8.

7 Conclusion

ClinTraceBench measures whether history representations preserve the evidence needed for longitudinal clinical reasoning. Its results show that full context remains the accuracy upper bound, abstention changes non-monotonically with context, and smaller backbones can dominate on cost–quality.

  • ClinTraceBench is a source-verifiable benchmark with 9 tasks, 385 verified dialogues, 8 history strategies, 4 backbones, 6,271 questions, and 200,672 predictions.
  • Full context remains the accuracy upper bound, while compressed strategies lose relational signal even when the underlying fact is retained.Increasing backbone capacity does not recover the dropped finding–diagnosis relation.
  • Abstention discipline does not improve monotonically with context length.This complicates the case for feeding longer charts.
  • The cost–quality frontier inverts the biggest-backbone-wins heuristic, with smaller backbones dominating under full context.

Limitations

ClinTraceBench supports source fidelity and within-benchmark comparisons, but its synthetic dialogues, single-source scope, task design, residual artifacts, and agentic-memory preparation confounds constrain interpretation and generalization.

  • Data realism: Synthetic dialogues establish source fidelity and gold correctness, not the realism of ambiguous, redundant, temporally inconsistent, or incomplete clinical text.Key clinical fields achieve ≥98% coverage, hallucination checks pass at 99.8–100%, and gold recomputation mismatches occur on <0.5% of items.
  • Scope: The benchmark uses one English MIMIC-IV hospital-wide source and a single 385-dialogue time horizon, leaving other settings, languages, pediatric populations, multi-year follow-up, and dynamic memory updates untested.T5 uses controlled rather than natural-EHR contradiction, while T8 measures post-treatment lab change without retained concurrent-medication annotations.
  • Artifacts and sensitivity: Approximately 1,400 empty API responses comprise 0.7% of predictions, but excluding them changes Sonnet / Haiku full-context results by at most +0.2 pp and preserves SP1–SP4 contrasts within ∆< 0.5 pp.Roughly 13% of Sonnet T8 rows reached the original token cap, so limited residual truncation may remain.
  • Agentic-memory confound: Agentic-memory comparisons partly conflate answering and construction effects because non-DeepSeek memory cells use a fixed DeepSeek-V3 preparation model.In six backbone-matched cells, matching preparation did not improve accuracy (∆ from −0.045 to −0.244), and recovered ≤1/19 T3 positives, but this corroborates full-cohort ordering only at that subset scale.
  • Task design: Task discrimination is uneven: T5, T6, T8, and T9 are not distinguishable from majority-class predictors in accuracy, while T4’s 53 questions leave it under-powered at a 27.2 pp minimum detectable effect.Macro-F1 recovers usable signal on T5 and T8; planned revisions will rebalance T5 contradiction classes and enlarge T8.

Ethics statement

ClinTraceBench uses de-identified MIMIC-IV-derived synthetic dialogues and validates benchmark records through deterministic checks plus human spot-checking. The appendix reports task-specific stratifications, retrieval diagnostics, compression footprints, costs, and residual response failures.

  • Ethics and scope: 385 dialogues are synthesized from de-identified structured MIMIC-IV records rather than real clinical communications.The benchmark does not measure clinical safety, diagnostic accuracy, treatment appropriateness, or patient outcomes.
  • Validation: L0–L4 deterministic validation and L5 human verification are applied before questions enter the evaluation set.The pipeline checks schema, anchors, and independently recomputed gold answers.
  • Task-specific scope: T4 contains 53 questions and is treated as a limited rather than headline finding because its pool is biased toward a handful of condition families.Figure 10 reports pool composition, full-context set-F1 by backbone, and the 8 × 4 cell heatmap.
  • Compression: Pearson r(Mem0 size, Mem0 T1 accuracy) = −0.129 on n = 210 patients, indicating that larger compressed representations are not more accurate.The appendix attributes the bottleneck to what compressed structures encode rather than their raw size.
  • Diagnostics: About 0.7% of prediction rows remain empty after retries, concentrated on GPT-4o-mini and Haiku retrieval-style contexts.These rows are documented but not excluded from headline accuracy numbers.

E.2 Parse failures (D2)

Parse diagnostics show that a substantial minority of responses fail the strict gold-format matcher, with failures concentrated in specific backbone and context cells. Additional appendix checks examine truncation, patient and disease variation, calibration, and confidence intervals.

  • Parse failures: About 6.5% of 200,672 rows are un-parseable, dominated by Claude refusal-as-prose under no-context blind.The Haiku × no-context cell is the single largest contributor, while richer-context strategy ordering is unaffected.
  • Output length and truncation: T4 is the only task where the 120-token ceiling binds meaningfully, with 86–88% truncation on Haiku/Sonnet under the worst memory baseline.T1 and T8 are the next largest binders; T3, T5, T6, and T7 are essentially un-truncated.
  • Per-patient variation: Full-context accuracy distributions are uniformly narrower than compressed-representation cells across backbones.Richer memory produces less between-patient skew, not just a higher mean.
  • Per-disease drift: Hypertension has the weakest pooled full-context accuracy at 0.615, attributed to task-mix bias rather than a disease-specific knowledge gap.The hypertension pool over-indexes longitudinal and comparison families.
  • Tertile calibration: T8’s high-complexity tertile is the easiest under full-context, unlike the monotonic calibration observed for T1 and T9.This is flagged as T8 tertile anti-alignment.
  • Uncertainty: Bootstrap CI width is dominated by per-cell sample size rather than memory strategy or backbone.T4 and T8 are widest, while T1, T7, and T9 are tightest; the appendix concludes headline rankings are robust to within-cell sampling noise.

G.1 Class-balanced metrics for the skewed tasks

Class-balanced analyses separate representation performance from majority-class effects and test whether parser choices, memory construction, retrieval depth, summary budget, and task exclusions alter the headline comparisons. The main strategy ordering is generally stable, while T9 exposes a substantial over-answering problem.

  • T5 contradiction: Full-context, dense-retrieval, and structured-timeline exceed the T5 majority-class macro-F1 baseline of 0.478 on every backbone.The compressed strategies do not exceed that baseline.
  • T8 response tracking: Structured-timeline leads T8 macro-F1, followed by full-context and dense-retrieval, against a majority-class baseline of 0.247.T8 has n = 134 per cell.
  • Discriminative power: T6’s best cell reaches 0.609 against a 0.423 majority baseline, confirming discrimination beyond the trivial predictor.T5, T6, and T8 separate representations, while T9 remains a relative diagnostic probe.
  • T9 abstention: After parser misses are discounted, fabricated values comprise roughly 54% of genuine T9 over-answers.Plausible-but-unsupported detail is the largest observed error type behind SP3.
  • Ablations: Neither backbone-matched memory preparation nor larger retrieval depth consistently improves accuracy or removes the gap to full-context.Doubling the summary budget lowers overall accuracy for three of four backbones.

H Case studies

The case studies illustrate four distinct failure modes: representational loss, uncritical answering, reasoning-capacity ceilings, and cross-document aggregation breakdown. Two examples show that compressed memory can lose encounter-local linkage, while more context can worsen abstention.

  • Representational loss: Compressed memory loses the encounter-local linkage sentence in T3 and collapses to constant-No.In the example, full-context DeepSeek answers yes while Mem0 DeepSeek answers no.
  • Uncritical answering: Full-context Sonnet over-answers an unstated potassium value that last-visit-only correctly marks insufficient.The case exemplifies an abstention failure rather than missing retrieval alone.
  • Failure-mode taxonomy: The appendix frames the four cases as representational loss, uncritical answering, reasoning-capacity ceiling, and cross-document aggregation breakdown.The cases span T3, T9, T2, and T6.

I Full per-task strategy-by-backbone matrix

Table 14 reproduces the complete 9 × 8 × 4 accuracy matrix across nine tasks, eight history strategies, and four backbones. The matrix covers 6,271 stratified questions and 200,672 evaluations, with per-task accuracies reported for the strategy-by-backbone cells.

  • Matrix structure: 288 cells span nine tasks, eight history representation strategies, and four backbones.All per-task figures in the main paper and appendix are derived from this matrix.
  • Evaluation scope: 6,271 questions and 200,672 evaluations comprise the stratified evaluation set underlying the complete matrix.The table reports per-task sample sizes from T1=1500 through T9=1500.
  • T1: T1 full context reaches 0.806 for DeepSeek, 0.781 for GPT-4o-mini, 0.812 for Claude Haiku, and 0.843 for Claude Sonnet.T1 contains 1,500 questions for each backbone entry shown.
  • T2: T2 no-context accuracy ranges from 0.000 for Claude Haiku to 0.333 for GPT-4o-mini, while last-visit-only ranges from 0.177 to 0.354.The displayed T2 rows include DeepSeek, GPT-4o-mini, Claude Haiku, and Claude Sonnet, each evaluated on 889 questions.
  • T3: T3 no-context accuracies cluster at 0.492–0.500 across backbones, whereas last-visit-only ranges from 0.508 to 0.562.The T3 entries shown use 600 questions per backbone.
  • T6: T6 structured timeline records 0.609 for DeepSeek and 0.530 for GPT-4o-mini, exceeding the displayed T6 A-Mem values of 0.503 and 0.464.These entries are based on 700 T6 questions per backbone.
Loading 2609.01111v1…