Source-linked AI summary
WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda
TL;DR
Existing benchmarks rarely test whether models can reason over real users’ longitudinal wearable records. WearableQA constructs a 16-type benchmark from authentic multimodal data using dual grounding and evaluates 14 LLMs, revealing wide performance variation and persistent difficulty.
Problem
Existing health benchmarks rarely evaluate reasoning over real users’ longitudinal wearable records combining wearable signals, biomarkers, and demographics.
Method
WearableQA builds 4,084 questions from 200 users’ records and uses dual grounding from literature findings and statistically validated population patterns, organized across 16 diagnostic question types.
Results
Across 14 models, accuracy ranges from 72.9% to 19.6% against a 10% chance baseline, with data and cross-signal reasoning remaining challenging.
Takeaways & Limitations
WearableQA provides a realistic and diagnostic testbed for evaluating LLM reasoning over real-world wearable data.
Abstract
from arXiv · showhide
Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record. We introduce WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to 500 days of daily measurements. WearableQA preserves authentic wearable distributions that include device noise and inter-individual variability. To evaluate distinct reasoning capabilities, we introduce 16 question types organized along two complementary axes: data versus health reasoning, which distinguishes computation over longitudinal measurements from physiological interpretation; and single- versus cross-signal reasoning, which separates reasoning about individual signals from the integration of multiple signals. To construct reliable questions at scale, we adopt a dual-grounding framework that combines literature-grounded physiological findings with statistically validated population-grounded physiological patterns. This enables the capture of meaningful relationships observed in real-world wearable data. Evaluation of 14 proprietary and open-source LLMs demonstrates that WearableQA effectively differentiates model capabilities, with performance ranging from 19.6% to 72.9% against a 10% chance baseline. Moreover, WearableQA remains far from solved: most models achieve accuracies below 60%. Overall, WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.
1 Introduction
WearableQA benchmarks reasoning over real users’ longitudinal wearable records, blood biomarkers, and demographics through a 16-type taxonomy and dual-grounded question construction. Across 14 models, performance varies widely, while data and cross-signal reasoning remain difficult.
- WearableQA contains 4,084 questions from 200 real users’ longitudinal wearable measurements, blood biomarkers, and demographic information.
- The benchmark separates data reasoning from health reasoning and single-signal from cross-signal reasoning to diagnose distinct model capabilities.
- Dual grounding combines peer-reviewed physiological findings with statistically validated population-grounded patterns, followed by deterministic computations and quality-control loops.
- 72.9% to 19.6% accuracy across 14 models, against a 10% chance baseline, demonstrates substantial variation in benchmark performance.
- Most models struggle with raw-measurement data reasoning, and stronger models show clear drops on cross-signal questions relative to single-signal questions.
2 Related Work
Existing health benchmarks primarily test medical knowledge or text-based interactions, while time-series benchmarks often use synthetic signals rather than real users’ longitudinal wearable data. WearableQA addresses this gap by evaluating health reasoning over authentic wearable records.
- Medical benchmarks such as MedQA, MedMCQA, PubMedQA, and HealthBench primarily evaluate medical knowledge through examination-style questions, biomedical text, or health-related interactions.
- Recent time-series benchmarks assess numerical reasoning, forecasting, and sequential-data question answering but largely rely on synthetic or simulated signals.
- WearableQA evaluates reasoning over longitudinal wearable measurements from real users in a health context.
3 The WearableQA Benchmark
WearableQA combines real-world longitudinal wearable, blood, and demographic data with a 2×2 reasoning-and-signal taxonomy. Its dual-grounding pipeline derives deterministic questions from literature findings and statistically validated cohort patterns.
- The benchmark draws on longitudinal wearable and blood-panel records from 200 real users sampled from a diverse in-the-wild cohort.
- Each user contributes 16 daily wearable metrics spanning cardio-fitness, activity and energy, and sleep domains.
- WearableQA crosses reasoning type with signal complexity and separately records whether each question’s ground truth comes from literature or population grounding.
- The 4,084 questions comprise 16 types, use 10 answer options, and balance answer positions uniformly, producing a 10% chance level.
- Data reasoning computes patterns from raw measurements, whereas health reasoning interprets physiological meaning; cross-metric questions integrate two or more signals.
- Dual grounding combines biomedical literature with statistically validated cohort patterns, re-deriving answers from real measurements rather than asserting them a priori.
- Literature-grounded relationships require independent same-direction support, while population-grounded candidates undergo effect-size, robustness, and cross-user authenticity checks.
- A discover-then-label pipeline converts each reasoning objective into reusable computation primitives executed deterministically on individual longitudinal trajectories.
4 Experiments
Experiments evaluate 14 LLMs on WearableQA under a common protocol and diagnose performance across reasoning type, signal complexity, prompting, input representation, and history context. Results show wide model variation, persistent difficulty with data and cross-signal reasoning, and substantial benefits from explicit computation and longitudinal history.
- Main results: 72.9% to 19.6%: overall accuracy varies widely across models under the default chain-of-thought protocol.Gemini-3.1-Pro reaches 72.9%, while Llama-3.2-3B reaches 19.6% against the 10% random baseline.
- Reasoning type: Most models perform better on health reasoning than data reasoning, indicating difficulty deriving answers directly from noisy longitudinal measurements.Exceptions and representative gaps are reported for Gemini-3.1-Pro, GPT-4o, Gemma-4-26B-A4B, and Mistral-Small-3.1.
- Signal complexity: Cross-signal reasoning remains challenging, with most proprietary models scoring higher on single-signal questions than on cross-signal questions.Gemini-3.1-Pro is an exception, achieving 72.4% on single-signal and 73.2% on cross-signal questions.
- Question types: Cross-signal prediction is especially difficult, ranging from 8.3% for Llama-3.1-8B to 40.5% for Claude-Opus-4.6.The task requires inferring one signal from another using longitudinal measurements and model knowledge.
- Prompting: Chain-of-thought improves most models, with gains up to 27 points for single-signal reasoning and up to 14 points for cross-signal reasoning in Claude-Opus-4.6.Open-weight models show smaller average gains of +2.3 points on single-signal and +3.0 points on cross-signal reasoning.
- Analysis: Python access raises overall accuracy from 51.2 to 71.3, whereas changing textual or visual representations provides only marginal or negative gains.Removing longitudinal history reduces history-dependent accuracy from 75.9% to 39.8%, and withholding all wearable time series lowers overall accuracy from 60.2% to 17.3%.
5 Conclusion
WearableQA is presented as a realistic benchmark for testing LLM reasoning over noisy, variable personal health trajectories. Its taxonomy and dual-grounding construction support fine-grained evaluation, while experiments reveal persistent difficulty with data and cross-signal reasoning.
- Conclusion: WearableQA contains 4,084 questions grounded in wearable records, blood biomarkers, and demographics from 200 real users.The benchmark preserves real-world noise and inter-individual variability.
- Conclusion: The benchmark evaluates 16 question types across data versus health reasoning and single- versus cross-signal reasoning.These axes enable fine-grained diagnosis of model capabilities.
- Conclusion: A dual-grounding framework combines literature-grounded physiological findings with population-grounded patterns instantiated on individual user trajectories.
- Conclusion: Across 14 models, accuracy ranges from 72.9% to 19.6% against a 10% chance baseline, while most models struggle with data and cross-signal reasoning.
A.1 Evaluation Protocol and Prompt
The evaluation protocol gives every model the same user context and differs the conditions only in the closing answer directive. Responses are parsed into a single option identifier and scored against deterministic ground truth.
- Prompt structure: Each prompt combines a user profile, sensor data, blood biomarkers, cohort reference, question, and answer options.The prompt ends with an answer-format directive that differs between chain-of-thought and direct-answer conditions.
- Prompt structure: The system instruction asks models to analyze the supplied health data and select the single best answer.
- Evaluation conditions: The chain-of-thought and direct-answer conditions use byte-identical context blocks, so their accuracy difference is attributed to reasoning elicitation.
- Answer parsing: Responses are parsed for an answer label using specified fallbacks, with direct answering emitting a bare letter and chain-of-thought reading the final line.
A.2 Experimental Details
The prior-knowledge analysis contrasts familiar definitional signal relationships with empirical associations that must be inferred from observed wearable measurements.
- Prior-knowledge analysis: Definitional pairs are relationships anticipated from prior knowledge, whereas empirical pairs are associations referred to from observed measurements.Examples include step count with active energy expenditure and heart rate with stress level.
A.3 Time-Series Representations
WearableQA represents longitudinal wearable histories in multiple text and image formats while preserving identical numeric content. The formats emphasize different reading patterns, including same-day cross-signal comparison and within-signal temporal trajectories.
- Representations: Four text formats—row, col, markdown, and csv—are generated alongside image and hybrid conditions from identical numeric content and column labels.The hybrid conditions are markdown+stats and csv+chart.
- Representations: The row format places one day per line with inline signal key=value pairs, emphasizing same-day cross-signal reading.An example includes daily Steps, RHR, Sleep(h), and HRV measurements.
- Representations: The col format groups each signal into a contiguous dated series, emphasizing within-signal temporal trend and trajectory reading.The example lists all dates for Steps, RHR, Sleep(h), and HRV in separate blocks.
- Representations: Markdown uses a days-by-signals pipe table, while csv uses the same grid as a header row followed by comma-separated value rows.Both formats encode the same three-day, four-signal excerpt.