PaperlayerView exact revision on arXiv

Source-linked AI summary

BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing

Yu Yvonne Wu, Arvind Pillai, Yuliang Chen, Yuwei Zhang, Sudarshan Regmi, Tess Z. Griffin, Michael V. Heinz, Lisa A. Marsch, Nicholas C. Jacobson, Andrew Campbell

arXiv:2608.27219v1cs.CL

TL;DR

Wearable sensing can provide longitudinal mental-health signals, but existing LLM agents have not been systematically evaluated for long-term wellbeing prediction with evidence-grounded rationales. BALMS benchmarks these capabilities and finds that reliable performance depends on backbone strength, selective evidence access, and semantically meaningful features.

  • Problem

    Existing LLM health agents mainly support short-term retrieval, leaving long-term wellbeing prediction and evidence-grounded rationale generation insufficiently evaluated.

  • Method

    BALMS benchmarks agentic longitudinal mental-health sensing across three real-world datasets, two task families, and multiple agent designs and LLM backbones.

  • Results

    Zero-shot agents achieve competitive prediction inconsistently, while chain-of-thought helps reasoning-oriented backbones and selective, semantically meaningful evidence access improves performance.

  • Takeaways & Limitations

    Longitudinal mental-health agents need selective memory, numerical grounding, and schema-aware reasoning to use extended sensing histories reliably.

  • Takeaways & Limitations

    The evaluation covers representative single-agent paradigms and retrospective datasets, while rationale quality relies on an LLM-as-Judge rather than expert or human-subject assessment.

Abstract

from arXiv · show

Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing. Wearable devices offer longitudinal behavioral and physiological signals for continuous, low-burden monitoring. Recent LLM-driven personal-health agents enable natural language queries over wearable signals, but mainly handle short-term, retrieval-based lookups (e.g., highest step count over a week). They do not evaluate whether agents can reason over long-term signals to predict wellbeing scores paired with evidence-grounded rationales. To address this gap, we introduce BALMS, the first systematic benchmark of LLM-based agentic systems for longitudinal mental health sensing. BALMS spans 3 real-world longitudinal datasets, 2 task families (closed-form wellbeing-score prediction and rationale generation auto-graded by an LLM-as-Judge), 3 agentic paradigms evaluated across 5 open- and closed-source LLM backbones. We find that zero-shot agents rarely outperform a simple mean baseline, except with stronger backbones or compact, semantically meaningful features. Chain-of-thought prompting improves reasoning-oriented backbones, but does not guarantee temporal grounding or numerical correctness. Together with more analysis on efficiency and temporal scaling, BALMS highlights the need for longitudinal mental health agents that selectively retrieve history, ground temporal evidence, and reason over interpretable behavioral features.

1 Introduction

BALMS addresses the gap between sparse self-report assessment and continuous wearable sensing by benchmarking agentic LLMs on long-term wellbeing prediction with evidence-grounded rationales. It evaluates task formulations and agent designs across datasets and backbones, showing benefits from stronger models, compact meaningful features, chain-of-thought, and selective memory.

  • Motivation: Wearables continuously collect multivariate behavioral and physiological signals over weeks to years, offering a lower-burden window into daily wellbeing than episodic self-report.Traditional clinical visits and questionnaires provide sparse snapshots and burden patients and providers (GBD 2019 Mental Disorders Collaborators, 2022; Substance Abuse and Mental Health Services Administration, 2025).
  • Motivation: Existing wearable-sensing agents mainly perform short-window factual lookups and arithmetic, leaving inference over long-term histories and mental wellbeing underexplored.Examples include finding the highest step count in a week or average resting heart rate over a month (Choube et al., 2025; Heydari et al., 2025; Wu et al., 2026).
  • Contribution: BALMS formalizes longitudinal mental health sensing as predicting verifiable wellbeing scores and evidence-grounded rationales from long-term passive sensing histories.The benchmark uses three real-world smartphone and wearable datasets and evaluates prompt-, tool-, and memory-based agents across five open- and closed-source LLM backbones.
  • Empirical Agent Design Insights: Zero-shot agentic prediction remains challenging, except with stronger backbones or compact, semantically meaningful sensor features.This finding motivates evaluating agent design choices rather than treating wearable sensing as a straightforward extension of short-window numerical lookup.
  • Empirical Agent Design Insights: Chain-of-thought prompting significantly boosts performance on reasoning-tuned backbones, while selective memory and meaningful features help more than raw streams or longer context windows.Agentic augmentations address context, computation, and auditability bottlenecks through retrieval, external memory, and tool use (Yao et al., 2022; Wang et al., 2023; Lewis et al., 2021).

2 Benchmarking LLM Agents for Longitudinal Mental Health Sensing

BALMS benchmarks LLM agents for longitudinal mental-health sensing across three real-world datasets, two task families, and three agentic paradigms. It evaluates wellbeing-score prediction and evidence-grounded reasoning over months-to-years of passive sensing data.

  • Benchmark scope: BALMS evaluates three agentic paradigms across three longitudinal datasets and five LLM backbones, scoring T1 by MAE and T2 with an LLM-as-a-Judge rubric.The benchmark uses shared longitudinal passive-sensing inputs for closed-form prediction and open-ended reasoning tasks.
  • Datasets: The benchmark includes DiversityOne’s 782 students across eight countries and 28 days, plus PMData’s 16 participants over five months with daily stress scores on a 1–5 Likert scale.DiversityOne combines smartphone signals with daily mood reports, while PMData combines Fitbit signals with daily PMSys wellness reports.
  • Datasets: GLOBEM covers 497 college students across four consecutive years of 10-week windows, pairing smartphone and Fitbit sensing with weekly PHQ-4 reports; BALMS predicts its anxiety subscale.This dataset provides the benchmark’s longest longitudinal setting among the described cohorts.
  • Task families: BALMS uses closed-form wellbeing-score prediction (T1) and open-ended rationale generation (T2), extending beyond fixed-window factual lookups such as weekly step maxima or monthly resting-heart-rate averages.Prior wearable agents primarily perform numerical perception over fixed windows (Choube et al., 2025; Heydari et al., 2025).
  • Agentic paradigms: The three paradigms expose history differently: prompt-based methods serialize numerical data, tool-based agents compute statistics over records, and memory-based agents retrieve relevant entries from external stores.Prompt-based prediction can produce explanations through designed prompts and chain-of-thought reasoning (Kim et al., 2024; Gruver et al., 2023), while tool-based systems use executable operations and memory systems can mix sub-day evidence with daily abstractions.

3 Experimental Setup

The evaluation spans five LLM backbones and measures closed-form prediction with MAE alongside LLM-judged rationales for temporal grounding and evidence consistency. Experiments use shared target-day splits and input windows across four agentic systems, with and without chain-of-thought.

  • Backbones: Evaluation covers five backbones: Qwen2.5-7B/14B-Instruct (Yang et al., 2025), Mistral-7B-Instruct-v0.3 (Jiang et al., 2023), DeepSeek-R1-Distill-Qwen-14B (DeepSeek-AI, 2025), and Claude-Haiku-4.5.The set includes three open-source instruct models, one open-source reasoning-distilled model, and one closed-source model.
  • Metrics: Closed-form predictions are scored by mean absolute error between integer self-reports and gold labels, averaged over held-out target days on each dataset’s native Likert scale.
  • Rationale evaluation: An Llama-3.3-70B-Instruct judge assesses whether rationales are temporally meaningful, grounded in sensing evidence, and consistent with the predicted score.The judge receives the sensing window, gold score, agent prediction, and rationale, evaluating more than fluency alone.
  • Rationale evaluation: The rubric evaluates six longitudinal reasoning operations—alignment, slicing, difference judgment, lag, structure, and interaction—using invocation rate, conditional accuracy, and coverage.Each operation is marked correct, incorrect, or not invoked, avoiding penalties for unnecessary omissions.
  • Experimental protocol: All systems use identical target-day splits and input windows; experiments compare four agentic systems with five backbones, reporting MAE with and without CoT.Open-source models run locally with vLLM, Claude-Haiku-4.5 uses the Anthropic API, and ReAct is capped at 10 iterations.

4 Results and Discussion

BALMS finds that zero-shot agents rarely beat a mean baseline unless using stronger backbones or compact, meaningful features, while performance depends on paradigm, temporal grounding, and schema compatibility. Chain-of-thought and longer histories help selectively, but do not ensure numerical correctness or efficient reasoning.

  • Rationale quality and temporal grounding: Claude-Haiku-4.5 is the only backbone that closes both rationale-quality and temporal-grounding gaps, while open-source instruct models produce fluent but weakly grounded rationales.Instruct agents rarely invoke alignment (C1) or structure (C5), indicating surface-level reasoning without reliable temporal operations.
  • Backbone and paradigm comparison: Claude-Haiku-4.5 with Health-LLM reaches 0.42 MAE on DiversityOne versus the 0.58 mean baseline, while PHIA reaches 0.54 MAE on PMData against 0.48, showing backbone- and paradigm-dependent performance.Larger, closed-source, or reasoning-tuned backbones more often approach or surpass the mean predictor, whereas smaller models such as Mistral-7B frequently do not.
  • Design implications: Scaling benefits HealthLLM and RAG more consistently than PHIA, whose success depends on generated code operating over compatible schemas, especially Fitbit-style aggregates in PMData.The findings support pairing capable backbones with structured or raw sensing representations rather than relying on zero-shot prompting alone.
  • Reasoning and grounding: Chain-of-thought reduces MAE by up to 41.4% across many DeepSeek and Claude configurations, but larger backbones still do not guarantee temporal grounding or numerical correctness.Reasoning-tuned models appear better able to organize temporal evidence before scoring.
  • Temporal scaling: RAG improves as histories lengthen, reducing PMData MAE by approximately 29% and GLOBEM MAE by approximately 9%, whereas HealthLLM degrades with longer textualized windows and PHIA remains stable but schema-sensitive.Adding records can dilute target-day evidence and increase numerical reasoning difficulty for HealthLLM.

5 Related Work

Recent benchmarks evaluate LLM-based agents in healthcare and mobile environments. They include interactive electronic-health-record tasks and smartphone application control through graphical interfaces.

  • Healthcare and mobile-agent benchmarks: Healthcare and mobile environments increasingly feature benchmarks for evaluating LLM-based agents.The cited examples include Schmidgall et al. (2025), Arora et al. (2025), and Tian et al. (2026).
  • Healthcare and mobile-agent benchmarks: MedAgentBench evaluates medical agents in an interactive FHIR-based electronic health record environment.The benchmark is attributed to Jiang et al. (2025).
  • Healthcare and mobile-agent benchmarks: Mobile-agent benchmarks assess whether agents can control smartphone applications through graphical interfaces.This capability is associated with Deng et al. (2024).

6 Conclusion

BALMS benchmarks LLM-based agentic systems for longitudinal mental-health sensing across diverse datasets, task families, and agentic paradigms. Its findings show that zero-shot prediction can be competitive, but reliable performance depends on grounding longitudinal evidence, with chain-of-thought helping reasoning-oriented backbones.

  • BALMS evaluates LLM-based agentic systems across three passive-sensing datasets, two task families, and three agentic paradigms.
  • Zero-shot agents achieve competitive wellbeing-score prediction without task-specific training, but reliability depends on how they access and ground longitudinal evidence.
  • Chain-of-thought prompting benefits reasoning-oriented backbones in longitudinal mental-health sensing.

Limitations

BALMS evaluates representative single-agent paradigms and retrospective prediction, rather than exhaustively covering agent designs or prospective deployment. Its scalable LLM-as-Judge rationale rubric does not replace expert or human-subject evaluation, motivating future work on real-world safety, calibration, and personalization.

  • BALMS covers single-agent prompt-, tool-, and memory-based systems but excludes multi-agent collaboration, planner-executor architectures, and dynamically combined tools, memory, and reflection.
  • The evaluation focuses on retrospective prediction from existing datasets rather than prospective deployment in real-world mental-health support scenarios.
  • The LLM-as-Judge rubric scales assessment of temporal grounding, numerical correctness, and explanation quality but does not replace clinical or human-subject evaluation.Future work should incorporate clinician and user assessments and study safety, calibration, and personalization.

Ethics Statement

BALMS is intended to support, not replace, professional medical care in continuous mental-health monitoring. The authors caution that autonomous, uncalibrated profiling could cause severe harms, including false reassurance or unwarranted clinical anxiety.

  • Potential Misuse: BALMS systems are intended as clinical decision-support tools rather than autonomous replacements for professional medical care.
  • Potential Misuse: Autonomous, uncalibrated mental-health profiling could lead to severe outcomes, including false reassurance or unwarranted clinical anxiety.

A Data Processing Details … F5. Schema-blind aggregation on raw streams.

PHIA’s apparent performance is driven by prediction collapse and execution failures rather than reliable longitudinal reasoning. Across datasets, silent execution errors, lost state, hallucinated rationales, hard-coded thresholds, and schema-blind aggregation undermine numerical evidence and are especially damaging on DiversityOne.

  • A Data Processing Details: The evaluation uses 7-day look-back windows for DiversityOne and 14-day windows for PMData and GLOBEM over daily-aggregated multivariate sensing signals.The processed evaluation set contains 1,355 DiversityOne, 1,448 PMData, and 1,640 GLOBEM target-day/user samples.
  • B.1 Predictions Collapse to a Single Label: PHIA collapses to a single label across datasets, yielding MAE 0.57 vs. mean predictor 0.48 on PMData, 1.52 vs. 0.80 on GLOBEM, and almost no parseable answers on DiversityOne.The modal PMData label is close to mean stress, while GLOBEM’s modal label is far from its low-skewed anxiety distribution.
  • B.2 Failure Taxonomy: Silent code execution leaves no output, yet PHIA proceeds as if computation succeeded, with traced windows averaging 2.0 empty observations on GLOBEM and 0.7 on PMData.These failures produce numerical claims unsupported by executed results.
  • B.2 Failure Taxonomy: Lost state across action blocks causes variables such as avg_steps, filtered_df, and anxiety_score to be referenced before redefinition, often triggering retries that reproduce silent-execution failures.The agent commonly responds with longer code blocks, which can re-trigger the missing-output failure.
  • F3. Force-finish with hallucinated rationale.: When the step budget expires, PHIA generates fluent but unsupported rationales; 8 out of 10 traced PMData windows contained deltas, thresholds, or trends that were never computed.The harness force-finishes by requesting a final label when no parseable answer is produced.
  • F3. Force-finish with hallucinated rationale.: Hard-coded thresholds such as steps > 20000 and sleep_minutes > 1680 on GLOBEM replace distribution-grounded reasoning and drive users into the same precomputed branches.This threshold heuristic is identified as the proximal cause of mode collapse because users usually fall on the same side of each cutoff.
  • F5. Schema-blind aggregation on raw streams.: On DiversityOne, applying mean or sum to roughly 18 half-hour slots per day without daily re-indexing conflates slot-level and daily aggregates, producing uninterpretable values on raw mobile streams.The same template works on PMData’s pre-aggregated Fitbit daily rows but fails on raw streams.
  • B PHIA Failure Analysis: F1 and F2 remove verifiable numerical evidence, after which PHIA either falls back on F4’s prior-driven heuristic or exhausts its budget and triggers F3; F5 makes DiversityOne worst because successful execution can still produce semantically wrong aggregates.The failure modes interact rather than operating independently.

B.3 Illustrative Traces · B.4 Implications

The illustrative traces expose execution-contract failures that can produce parseable but ungrounded predictions or hardcoded mode collapse. These failures recur across backbones, implicating sandbox interaction rather than language reasoning capacity and motivating memory-based agents.

  • B.3 Illustrative Traces: In PMData, repeated truncated code attempts produced no outputs, yet the agent returned a parseable prediction after force-finishing.The trace used Qwen2.5-14B-Instruct on participant p01 for target day 2019-11-25.
  • B.3 Illustrative Traces: The PMData rationale cited deltas that were never computed, making the prediction a prior-driven guess with post-hoc justification.It counted as successful in aggregate MAE only because a parseable integer was returned (F1+F3).
  • B.3 Illustrative Traces: In GLOBEM, lost execution state caused undefined-variable errors before the agent rebuilt data and predicted 3.The trace used participant INS-W_208 on target day 2018-05-16 and initially referenced variables defined only in earlier code blocks.
  • B.3 Illustrative Traces: The GLOBEM prediction was determined by the F4 magic-number threshold, producing 86% mode collapse across the held-out set.Recovered aggregates only selected between two hardcoded branches rather than determining the prediction from the data.
  • B.4 Implications: The same failure patterns appeared across Qwen2.5-14B, Mistral-7B, and DeepSeek-R1-Distill-Qwen-14B.This aligns with the observation that stronger backbones do not improve PHIA’s MAE.
  • B.4 Implications: The primary bottleneck is the model–sandbox contract—print discipline, state persistence, and threshold derivation—rather than language reasoning capacity.These observations motivate the memory-based paradigm benchmarked in Section 4.3.

C Latency Analysis

The latency analysis reports average per-sample latency as look-back windows grow and compares prompting with and without chain-of-thought across agentic systems, backbones, and datasets.

  • Latency vs. look-back window: Figure 7 reports Qwen2.5-14B-Instruct’s average per-sample latency as the look-back window grows on PMData and GLOBEM.
  • Per-backbone latency panels: Figures 8–10 report average per-sample latency for Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and DeepSeek-R1-Distill-Qwen-14B.
  • Per-backbone latency panels: The per-backbone panels contrast prompting without and with chain-of-thought across the three datasets, while Qwen2.5-14B-Instruct appears in main-paper Figure 6.

D Prompt Templates

The prompt templates define stress prediction inputs, outputs, and reasoning formats, while the LLM-as-Judge evaluates temporal reasoning and rationale quality using structured rubric scores. A case study shows that the rubric distinguishes evidence-grounded reasoning from similarly inaccurate predictions.

  • D Prompt Templates: The PHIA template uses an iterative ReAct trace that inspects prior sensor data, computes relevant features, and continues until emitting a final answer.The example begins by examining data shape and date range, with the loop continuing for up to M iterations.
  • D Prompt Templates: The LLM-as-Judge scores six temporal-reasoning dimensions with conditional accuracy and invocation rate, and six general-quality dimensions with mean Likert scores.Temporal dimensions use correct, incorrect, or not-invoked labels; general-quality dimensions use 1–5 ratings.
  • D Prompt Templates: Judge inputs include the rendered agent window, ground-truth and predicted labels, chain-of-thought rationale, and rubric, with justifications required to cite rationale text or specific values.Faithfulness assesses whether reasoning supports the prediction rather than whether the prediction matches ground truth; evidence grounding checks cited features, values, and days.
  • D.1 Reasoning Case Study: Same Window, Two Backbones: In the GLOBEM case study, both Qwen2.5-7B-Instruct and Claude-Haiku-4.5 over-predict a true anxiety score of 0, but their rationale failures differ across rubric dimensions.Qwen predicts 3 and Claude predicts 2; their comparable errors are assessed separately for evidence grounding and temporal structure.
  • D.1 Reasoning Case Study: Same Window, Two Backbones: Claude correctly attributes its numeric claims, whereas Qwen misattributes two of four, and Claude’s hedged conclusion follows its observations while Qwen makes an unsupported higher-level inference.The rubric therefore identifies qualitative reasoning differences that aggregate MAE alone cannot reveal.
Loading 2608.27219v1…