HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care

1 Introduction

• Existing wearable QA benchmarks largely test statistical summaries or short-horizon classification, leaving long-term dynamics, therapy and activity context, and system failures underrepresented.These reasoning and analytical-rigor capabilities matter for safely guiding clinical decisions.Source: §1 · pp. 1–2
• HealthLoopQA organizes wearable monitoring evaluation into three analysis stages and eleven atomic reasoning abilities.Its benchmark tasks span process mining, anomaly detection, and prediction over 30-day monitoring horizons.Source: §1 · p. 2
• The fault-injected AID simulation testbed models eight device malfunctions and ten cyber-physical attack patterns for systematic safety and security evaluation.It is based on an open-source AID simulation testbed.Source: §1 · p. 2
• The benchmark contains 1,516 QA instances derived from simulated and real-world datasets.It comprises 127 diagnostic tasks covering process mining, anomaly detection, and prediction.Source: §1 · p. 2
• Evaluation of state-of-the-art LLMs with prompting-based and agentic frameworks reveals limitations in longitudinal reasoning, including complex pattern mining failures and in-context laziness under long-context prompting.The benchmark is designed to assess analytical rigor in medical wearable data interpretation.Source: §1 · p. 2

2 Related Work

• Earlier time-series benchmarks vary in scope: DeepSQA uses template questions, TimeSeriesExam lacks domain-specific tasks [Cai et al., 2024], and SensorQA emphasizes process mining and language similarity [Reichman et al., 2025].MTBench adds regression and classification metrics for financial and weather data but omits failure-aware medical-wearable monitoring [Chen et al., 2025].Source: §2 · p. 2
• Healthcare QA benchmarks include generic physiological questions in Time-MQA [Kong et al., 2025], while ECG-QA and LLM-CGM lack therapy or activity context and systematic failure modeling [Oh et al., 2023; Healey and Kohane, 2024b].These limitations leave domain-specific wearable monitoring only partly addressed.Source: §2 · p. 2
• CGM analysis methods capture temporal trends or integrate environmental data, but related work identifies limits in interpretability and coverage of diverse monitoring needs.Examples include LSTM models [Klonoff et al., 2025], AttenGluco [Farahmand et al., 2025], and GlucoBench [Sergazinov et al., 2024].Source: §2 · pp. 2–3
• HealthLoopQA’s overview combines closed-loop fault simulation, question design across task type and reasoning depth, and answer evaluation for numerical precision and reasoning alignment.Question prototypes draw on CGM frameworks, clinical guidelines, AID vulnerabilities, and patient monitoring needs.Source: Figure 1 · p. 3
• The paper uses QA as a diagnostic tool for reasoning over glucose dynamics, temporal patterns, and patient context, rather than treating linguistic diversity as the task itself.It contrasts this approach with costly human-centered experiments and offline proxy tasks that may not predict application performance [Bedi et al., 2025; Doshi-Velez and Kim, 2017].Source: §2 · p. 3

3 HealthLoopQA

• The single-turn task takes monitoring history, therapy and context, a natural-language question, and detailed answer instructions, and returns a precise answer plus natural-language reasoning.Answers may be numerical, categorical, or temporal.Source: §3 · pp. 3–4

3.1 Taxonomy

• The taxonomy assesses understanding of intent, reasoning, and task execution, with reasoning divided into descriptive, memory, and pattern levels and decomposed into atomic abilities.Process mining, anomaly detection, and prediction are the core time-series tasks.Source: §3.1 · p. 4

3.2 Tasks

• The 127 expert-curated tasks are based on CGM practices, clinical guidelines, and AID safety requirements, with questions and reasoning rationales generated from templates.Each question’s reasoning chain is manually analyzed and labeled by atomic capability.Source: §3.2 · p. 5

3.3 QA Data Collection

• Simulation enables controlled assessment of diverse reasoning and safety-critical failure scenarios, while real-world data evaluate robustness under naturally occurring physiological and behavioural variability.The simulation involves 16 virtual patients and 17 injected fault patterns; the real-world evaluation uses 10 months of data from five individuals with T1D and 44 suitable tasks.Source: §3.3 · p. 5

3.4 Quantitative Evaluation Metrics

• The evaluation uses SMAPE for continuous predictions, classification accuracy for categorical answers, and Affinity F1 for timestamped events.The metrics are normalized to a 0-to-1 scale, with higher values indicating better performance, and averaged across questions.Source: §3.4 · p. 6

4.1 Experimental Setup

• Prompting baselines receive monitoring data as text; agent baselines use attached CSVs and can iteratively execute Python, with Gemini evaluated in both frameworks.GPT and Claude were evaluated using the agent baseline.Source: §4.1 · p. 6

4.2 Quantitative Results

• Gemini agent tools raised overall scores from 43.7 to 65.8 on simulated data and 64.6 to 78.5 on real-world data; GPT led overall, followed by Claude and Gemini.Figure 4 shows per-question normalized-score distributions, and Table 2 summarizes performance by dataset portion, task type, and cognitive level.Source: §4.2 · pp. 6–7, Figure 4 · p. 6, Table 2 · p. 6
• On simulated data, GPT scores 72.9 on AD and 74.6 on PD, while Claude leads PM at 92.3; Gemini-Agent rises 67.4→85.5 on PM and 21.2→52.4 on AD.Gemini-Agent's PD gain is smaller, from 62.3 to 64.8.Source: §4.2 · p. 7
• Agents lift Gemini's descriptive and memory scores to 86.8 and 79.3, but Gemini-Agent's real-world PD score falls 73.0→61.5; GPT- and Claude-Agent score at least 64.0.The paper attributes Gemini-Agent's real-world prediction errors mainly to patient-specific local trends and inconsistent insulin-unit interpretation.Source: §4.2 · p. 7
• Descriptive/retrieval paths generally outperform pattern paths; as PM reasoning steps rise, scores generally fall, especially for prompting Gemini, while agent systems are more robust.Figure 5 plots overall normalized scores against the number of atomic skills required per question.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3.1 Prompting-based baseline

• Prompting models skip exact 30-day calculations or anomaly scans, misalign time windows, rely on unsupported estimates, guess over abstaining, and sometimes return malformed or overbroad outputs.The authors reviewed traces across all 30-day questions for Patient 0 and identify formatting drift as relatively uncommon.Source: §4.3.1 · p. 8

4.3.2 Agent-based baseline

• Agent errors include counting glucose measurements instead of distinct events, misreading insulin units, flagging normal exercise adjustments as anomalies, and merging false positives into broad intervals.The insulin-scale error produced predictions approximately five times larger than expected; GPT and Claude agents followed the scale instruction more consistently in that example.Source: §4.3.2 · pp. 8–9
• For prediction, models sometimes replace context-aware forecasting with averages from analogous historical events rather than estimating the trajectory from current conditions.In one example, the model averaged prior running events instead of simulating the immediate post-exercise glucose trajectory.Source: §4.3.2 · p. 9

• HealthLoopQA is a diagnostic benchmark designed to evaluate LLM reasoning and analytical rigor in medical wearable data interpretation.Source: §5 · p. 9
• Frontier models show strengths in surface-level description and memory retrieval but significant vulnerabilities in deeper temporal reasoning, including pattern mining, causal anomaly detection, and context-aware prediction.Source: §5 · p. 9
• The authors conclude that textual and coding fluency do not inherently provide the rigorous analytical capability required for safe physiological data interpretation.They identify causal, context-aware modeling as necessary for clinically robust decision support.Source: §5 · p. 9

6 Limitations

• In-silico patient models omit some biological variability, modeled faults may not capture real-world threat heterogeneity, and the OpenAPS setup excludes some diabetes-management practices.The 127 tasks cover core capabilities but not every patient question; other wearable domains and real-world deployment require further validation on relevant datasets and in clinical studies.Source: §6 · p. 9

A Ethics Statement

• The study reports no new human or animal experiments; real-world Type 1 Diabetes data came from public, fully anonymized datasets, and findings are for academic purposes, not medical advice or deployment safety conclusions.The authors call for rigorous clinical validation before drawing conclusions about safety or efficacy in real-world clinical deployments.Source: A Ethics Statement · p. 12

B Large Language Models (LLMs) Usage

• LLMs served as evaluation subjects and writing assistants; authors manually verified and curated the final failure patterns, interpretations, and conclusions.Source: B Large Language Models (LLMs) Usage · p. 12

C Reproducibility Statement

• The authors state that setup details, agent workflows, prompts, the benchmark, simulation environment, and evaluation scripts will be publicly released upon publication.Source: C Reproducibility Statement · p. 12

D Atomic reasoning abilities

• Atomic reasoning types were defined by reviewing benchmark questions and instructions, then refined through internal discussion to resolve disagreements and improve consistency.Source: D Atomic reasoning abilities · p. 12
• Each question was mapped to an ordered sequence of required reasoning types; for example, finding a day's peak glucose requires temporal retrieval followed by quantitative calculation.The paper reports performance grouped by complete ordered paths such as TR→QC.Source: D Atomic reasoning abilities · p. 12

E Failure analysis: in-context laziness

• In-context laziness produces an illusion of precision: models skip exact calculations, use plausible assumptions, and may confidently guess instead of expressing uncertainty.The paper says the pattern often begins with skipped computation or temporal misalignment, making later reasoning less meaningful.Source: E Failure analysis: in-context laziness · pp. 12–13
• Timelined prompting reduces temporal misalignment by grouping glucose values with events under timestamps, but models may still avoid full calculations.The two cases require either long-sequence quantitative computation or identifying a relevant time window before calculating.Source: E Failure analysis: in-context laziness · p. 13
• 77.3% accuracy for the Agent baseline exceeded the other PM methods, while Timelined reached 56.8% and Structured prompting scored 45.5% (30-day), 39.0% (7-day), and 64.7% (1-day).The authors attribute the Agent advantage to code-interpreter use, which alleviates computational and alignment failures.Source: E Failure analysis: in-context laziness · p. 14
• The authors distinguish this failure from chain-of-thought unfaithfulness: models anchor on rough intermediate values rather than computing over the full CGM record.They hypothesize that post-training shortcuts may contribute when models are rewarded for final answers without complete reasoning or recognizing when tools are needed.Source: E Failure analysis: in-context laziness · p. 14

F Data details

• The benchmark includes 127 QA templates, with task distribution summarized in Table 1 and frequently used words shown in Figure 10.Data simulation and collection details are described in later sections.Source: F Data details · p. 14

F.1 AID System Closed-loop Simulation Testbed

• The closed-loop testbed uses a clinically validated simulator with multiple controller strategies, 20 virtual patients with type 1 diabetes, and physical-activity submodels.Its rationale includes modeling meal, insulin, and activity effects; labeling faults without patient risk; representing inter-individual variability; and enabling controlled evaluation.Source: §F.1 · p. 15

F.2 Faults Modelling

• The benchmark defines 17 fault patterns and two physiological hazards through explicit simulation-loop manipulations, including CGM perturbations, basal-insulin changes, event replay, and dropped sensor signals.The modeled fault sources include documented glucose-sensing and insulin-infusion malfunctions and cyber-physical attack vectors.Source: §F.2 · pp. 15–16

F.3 Data simulation settings

• The settings describe three daily schedules and one fault-injected schedule, simulated over 30 days for 16 patients.The schedules vary exercise frequency and randomness; the passages identify separate scenarios for process-mining and prediction tasks.Source: §F.3 · pp. 16, 18
• Meal schedules specify breakfast, snacks, lunch, and dinner windows, with meals lasting 30–60 minutes and snacks lasting 5–15 minutes.Scenario-specific routines include running, cycling, a late-night snack, and an overeating lunch; the fault-injected scenario uses 17 designed faults.Source: §F.3 · p. 16
• Figure 8 compares reasoning behaviours across prompting baselines on pm_14, using colors to mark temporal or calculation errors, unsupported assumptions, guessing, and correct traces.Figure 10 identifies the semantic distribution and diversity of benchmark task prototypes.Source: Figure 8 · p. 17, Figure 10 · p. 18
• Figure 9 compares 24-hour glucose profiles across real-world, clean-simulation, and fault-injected simulation data.Table 7 summarizes CGM sensor and insulin-pump malfunctions; the supplied captions do not state comparative values.Source: Table 7 · p. 17, Figure 9 · p. 18

F.4 Real-world and simulation data statistics

• Table 9 and Figure 9 present characteristics and statistics for real-world, regular simulation, and fault-injected simulation data.Source: §F.4 · p. 18
• Figure 9 visualizes the data characteristics and statistics for the real-world and two simulation datasets.Source: §F.4 · p. 18
• Table 10 summarizes the QA data schema.Source: §F.4 · p. 18

G Experiment setting

• GPT-5.4, Claude-sonnet-4-6, and Gemini-3.1-flash-lite-preview are used for agent baselines, while only Gemini is used for the prompting baseline.The authors attribute the prompting-only model choice to practical reasons.Source: G Experiment setting · p. 18

G.1 Agent Baseline

• The agent baselines give models code-interpreter access to patient data and use structured answer extraction or output for evaluation.GPT-5.4 accesses patient data as a CSV and can write Python; Claude and Gemini also use code interpreters, with structured output enabled.Source: §G.1 · p. 18

G.2 Prompt based Baseline

• The prompting baseline converts glucose, insulin, meal, and exercise context into raw text and requests JSON output for post-processing.Source: §G.2 · p. 18

G.3 System Prompt

• The system-prompt templates contrast separate glucose and event sections, timestamp-organized glucose and event timelines, and an agent prompt with Python tools.Source: Figure 11 · p. 19, Figure 12 · p. 20, Figure 13 · p. 20