PaperlayerView exact revision on arXiv

Source-linked AI summary

BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing

Yu Yvonne Wu, Arvind Pillai, Yuliang Chen, Yuwei Zhang, Sudarshan Regmi, Tess Z. Griffin, Michael V. Heinz, Lisa A. Marsch, Nicholas C. Jacobson, Andrew Campbell

arXiv:2608.27219v1cs.CL

TL;DR

How reliably LLM agents can predict mental wellbeing from long-term sensing remains unclear. BALMS benchmarks wellbeing-score prediction and evidence-grounded rationales across three datasets and three agentic paradigms. Zero-shot prediction remains challenging except with stronger backbones or compact, semantically meaningful features, while chain-of-thought prompting improves reasoning-tuned backbones.

  • Problem

    It remains unclear how agents can reliably predict mental wellbeing from longitudinal sensing histories that challenge context capacity, numerical reasoning, and cross-dataset transfer.

    Source: §1 · p. 2

  • Method

    BALMS benchmarks three agentic paradigms on three longitudinal sensing datasets, evaluating wellbeing-score prediction and evidence-grounded rationale generation.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Zero-shot prediction remains challenging except with stronger backbones or compact, semantically meaningful features; chain-of-thought prompting improves reasoning-tuned backbones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight selective memory, numerical grounding, and schema-aware reasoning as priorities for longitudinal mental-health agent design.

    Source: §6 · p. 9

    BALMS evaluates retrospective prediction from existing datasets rather than real-world interactive or intervention settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing. Wearable devices offer longitudinal behavioral and physiological signals for continuous, low-burden monitoring. Recent LLM-driven personal-health agents enable natural language queries over wearable signals, but mainly handle short-term, retrieval-based lookups (e.g., highest step count over a week). They do not evaluate whether agents can reason over long-term signals to predict wellbeing scores paired with evidence-grounded rationales. To address this gap, we introduce BALMS, the first systematic benchmark of LLM-based agentic systems for longitudinal mental health sensing. BALMS spans 3 real-world longitudinal datasets, 2 task families (closed-form wellbeing-score prediction and rationale generation auto-graded by an LLM-as-Judge), 3 agentic paradigms evaluated across 5 open- and closed-source LLM backbones. We find that zero-shot agents rarely outperform a simple mean baseline, except with stronger backbones or compact, semantically meaningful features. Chain-of-thought prompting improves reasoning-oriented backbones, but does not guarantee temporal grounding or numerical correctness. Together with more analysis on efficiency and temporal scaling, BALMS highlights the need for longitudinal mental health agents that selectively retrieve history, ground temporal evidence, and reason over interpretable behavioral features.

1 Introduction

BALMS benchmarks agents that generate numerical wellbeing scores and evidence-grounded rationales from longitudinal sensing histories across three datasets. Its findings emphasize selective memory and semantically meaningful features over raw sensor streams or extended context windows.

1 Introduction

  • Episodic assessments provide sparse mental-state snapshots and burden patients and providers, whereas passive wearable and mobile signals capture longitudinal wellbeing dynamics.Assessment burden is highlighted by Substance Abuse and Mental Health Services Administration, 2025. Devices continuously collect behavioral and physiological signals over weeks to years.Source: §1 · p. 1
  • Existing wearable-agent evaluations emphasize short-window numerical lookups or qualitative insights rather than longitudinal mental-state inference with paired wellbeing predictions and explanations.Short-window lookup examples include Choube et al., 2025; Heydari et al., 2025; Merrill et al., 2026; Wu et al., 2026. Earlier clinical-agent benchmarks remain text-centric (Schmidgall et al., 2025; Arora et al., 2025).Source: §1 · p. 2
  • Longitudinal sensing challenges agents through context overflow, unreliable temporal numerical reasoning, and dataset differences in sensors, feature formats, and observation horizons.Numerical unreliability and dataset heterogeneity complicate longitudinal prediction and transfer (Pillai et al., 2025; Spathis and Kawsar, 2024; Luo et al., 2025). Retrieval, external memory, and executable tools support relevant-history selection, delegated computation, and auditable rationales (Yao et al., 2022; Wang et al., 2023; Lewis et al., 2021).Source: §1 · pp. 1–2
  • Three real-world smartphone and wearable datasets underpin BALMS, which requires verifiable numerical wellbeing scores paired with evidence-grounded rationales over long-term passive sensing histories.The benchmark formalizes longitudinal mental health sensing as an agentic task.Source: §1 · p. 2
  • Three agentic paradigms—prompt-, tool-, and memory-based—are evaluated across five open- and closed-source LLM backbones within standardized assessment infrastructure.This unified evaluation systematically compares agent designs across multiple backbones.Source: §1 · p. 2
  • Zero-shot agentic prediction remains challenging, with stronger backbones and compact, semantically meaningful features providing exceptions.Chain-of-thought prompting significantly improves reasoning-tuned backbones. Agents benefit substantially more from selective memory and semantically meaningful features than from raw sensor streams or extended context windows.Source: §1 · p. 2

2 Benchmarking LLM Agents for Longitudinal Mental Health Sensing

BALMS benchmarks longitudinal mental health sensing across three real-world datasets, two task families, three agentic paradigms, and five LLM backbones. It targets low-burden monitoring between clinical visits, assessing agreement with self-reports and rationales verifiable against sensing data rather than diagnostic capability.

2.3 Benchmarking LLM Systems

  • 3 agentic paradigms—prompt-based reasoning, tool-based analysis, and memory-based retrieval—are evaluated across 3 datasets and 5 LLM backbones on both task families.The paradigms differ in how longitudinal sensing data is exposed to the LLM; score prediction uses MAE, while rationale generation uses an LLM-as-a-Judge rubric.Source: Figure 3 · p. 3, §2 · p. 3

2 Benchmarking LLM Agents for Longitudinal Mental Health Sensing

  • BALMS targets continuous, low-burden monitoring between clinical visits rather than diagnosis, requiring scores aligned with self-reports and rationales verifiable against the sensing window.The benchmark therefore evaluates both wellbeing-score prediction and evidence-grounded interpretation of longitudinal passive sensing.Source: §2 · p. 3

2.1 Datasets

  • 3 datasets pair passive sensing with self-reports: DiversityOne measures daily mood, PMData daily stress, and GLOBEM the PHQ-4 anxiety subscale.DiversityOne (Busso et al., 2025) follows 782 students across 8 countries for 28 days using smartphone sensing; PMData follows 16 participants for 5 months using Fitbit signals and daily 1–5 wellness scales. GLOBEM combines smartphone and Fitbit sensing from 497 college students with weekly PHQ-4 reports across 10-week study windows over four consecutive years.Source: §2.1 · p. 3

2.2 Task Families

  • T1 predicts an integer wellbeing score from a target day’s sensing history; T2 requires a rationale citing sensor values and multi-day patterns.T1 follows Health-LLM’s zero-shot setting (Kim et al., 2024), separating prediction from text generation and retaining each dataset’s native scale. T2 evaluates justified assessments of trajectories, including trends, weekly cycles, and recovery, using an LLM-as-a-Judge rubric.Source: §2.2 · p. 4

2.3.1 Prompt-Based Reasoning Systems

  • Health-LLM (Kim et al., 2024) supplies daily sensor arrays, demographics, and task instructions in one fixed-context prompt, without retrieval, tools, or analysis planning.BALMS adopts its prompt template directly, varying only the look-back window and underlying backbone; the requested output is an integer target label.Source: §2.3.1 · p. 4

2.3.2 Tool-Based Agents

  • PHIA runs executable Python over preloaded sensing DataFrames, iteratively feeding results back to the LLM until a final answer.Its ReAct loop (Yao et al., 2022) generates reasoning traces and selects actions from a predefined tool space, supporting operations such as date filtering and groupby aggregation.Source: §2.3.2 · p. 4

2.3.3 Memory- and Retrieval-Based Agents

  • Memory-based agents retrieve similar historical days or combine sub-day records with daily summaries through an adapted RAPTOR tree (Sarthi et al., 2024).Day-level RAG embeds each day’s signals using a sentence-transformer (Reimers and Gurevych, 2019), retrieves top-k days by cosine similarity, and supplies those days with the query. Tree-based retrieval searches both sub-day leaves and LLM-generated daily summaries, allowing retrieved evidence to mix temporal resolutions.Source: §2.3.3 · p. 5

3 Experimental Setup

BALMS evaluates five LLM backbones using mean absolute error for wellbeing-score predictions and an LLM judge for rationale quality. Systems share target-day splits and sensing windows.

3.1 LLM Backbones

  • The evaluation spans three open-source instruct backbones, one open-source reasoning-distilled backbone, and one closed-source backbone.Instruct models are Qwen2.5-7B/14B-Instruct (Yang et al., 2025) and Mistral-7B-Instruct-v0.3 (Jiang et al., 2023). The others are DeepSeek-R1-Distill-Qwen-14B (DeepSeek-AI, 2025) and Claude-Haiku-4.5.Source: §3.1 · p. 5

3.2 Benchmark Evaluation Metrics

  • Prediction performance uses mean absolute error (MAE), while a Llama-3.3-70B-Instruct judge assesses rationale grounding, temporal meaning, and consistency with predicted scores.MAE compares integer predictions with gold labels on each dataset’s native Likert scale, averaged over held-out target days. The judge receives the sensing window, gold self-report score, agent prediction, and rationale.Source: §3.2 · p. 5
  • Temporal reasoning adapts TemporalBench (Cai et al., 2024) to assess alignment, slicing, difference judgment, lag, structure, and cross-channel interaction.These cover mapping temporal conditions to fields, comparing segments, quantifying changes, delayed effects, identifying peaks or change points, and joint channel effects. Operations are marked correct, incorrect, or not invoked; reporting includes invocation rate, conditional accuracy, and invoked-and-correct coverage without penalizing unnecessary omissions.Source: §3.2 · pp. 5–6
  • General quality adapts the PHIA expert rubric, assessing faithfulness, evidence grounding, domain knowledge, safety, clarity, and overall quality.Each receives a 1–5 Likert score. Faithfulness tests whether predictions follow logically from reasoning; evidence grounding tests whether cited facts are verifiable against the input window.Source: §3.2 · p. 6

3.3 Implementation Details

  • Systems share target-day splits and input windows, with matching prompt formats and decoding settings across locally served open-source backbones and API-served Claude-Haiku-4.5.Open-source models use vLLM on four A6000 GPUs; Health-LLM and PHIA follow official implementations with at most 10 ReAct iterations. Chunk RAG follows official LangChain implementations, while RAPTOR builds a two-level memory tree over sub-day records and daily summaries.Source: §3.3 · p. 6

4 Results and Discussion

Zero-shot mental-wellbeing prediction rarely beats a mean baseline, while reasoning quality depends strongly on the backbone. Longer histories, richer sensor inputs, and larger models do not uniformly improve performance: retrieval, interpretable features, and compatible tool schemas matter.

4.2 Reasoning and Rationale Analysis

  • Claude-Haiku-4.5 alone closes both rationale-quality and temporal-grounding gaps, while open-source instruct backbones produce fluent explanations but struggle with temporal reasoning.Alignment (C1) and structure (C5) invocation are near zero for instruct backbones, reflecting failures in historical-window slicing and cross-channel correlation tracking.Source: Figure 4 · p. 7, §4.2 · p. 7

4.1 Zero-Shot Mental-Wellbeing Prediction

  • 0.42 MAE for Claude-Haiku-4.5 with Health-LLM on DiversityOne beats the 0.58 mean baseline, although zero-shot agents rarely outperform the mean predictor.The baseline predicts the average training-set label without sensing inputs; stronger backbones or compact, semantically meaningful features provide exceptions to the general pattern.Source: §4.1 · pp. 6–7
  • Larger backbones improve HealthLLM and RAG, but PHIA depends on tool-schema compatibility rather than language capacity alone.PHIA excels on PMData’s Fitbit-style aggregates but fails on DiversityOne and GLOBEM’s raw mobile streams, which make generated code brittle. The proposed direction pairs capable backbones with structured sensor representations or raw signals rather than zero-shot prompting alone.Source: §4.1 · p. 7

4.2 Reasoning and Rationale Analysis

  • 41.4% is the largest reported MAE improvement from CoT prompting, which primarily benefits reasoning-tuned DeepSeek and Claude backbones.Improvements occur across most configurations, but larger backbones do not guarantee temporal grounding or numerical correctness.Source: §4.2 · p. 7

4.3 Long-Horizon Scaling

  • ∼29% on PMData and ∼9% on GLOBEM are RAG’s MAE reductions with longer histories, whereas HealthLLM degrades and PHIA remains schema-sensitive.Experiments use Qwen2.5-14B-Instruct with the target day fixed. Additional textual records can dilute HealthLLM’s target-day evidence; PHIA’s preloaded DataFrames stabilize PMData performance but remain weaker and noisier on GLOBEM.Source: Figure 5 · p. 8, §4.3 · p. 8

4.4 Efficiency and Sensitivity Analysis

  • Compact smartwatch-only features often match or surpass full multimodal inputs, while Health-LLM and RAG avoid PHIA’s costly iterative code execution.On GLOBEM, removing mobile streams favors semantically rich behavioral features over raw, high-frequency signals. PHIA’s latency penalty grows particularly with CoT because it repeatedly generates, runs, and debugs code.Source: Table 4 · p. 8, §4.4 · p. 8, Figure 6 · p. 8

5 Related Work

Existing healthcare and mobile-agent benchmarks measure tool use, planning, and interaction but do not capture the core challenges of longitudinal mental-health sensing.

5 Related Work

  • Existing benchmarks miss reasoning over multi-week to multi-year wearable histories, grounding predictions in numerical temporal patterns, and evaluating whether explanations support concrete wellbeing outcomes.Source: §5 · p. 9
  • MedAgentBench evaluates medical agents in an interactive FHIR-based electronic health record environment (Jiang et al., 2025).Source: §5 · p. 9
  • Mobile-agent benchmarks evaluate whether agents can control smartphone applications through graphical interfaces (Deng et al., 2024).Source: §5 · p. 9

6 Conclusion

BALMS benchmarks LLM-based agents for longitudinal mental-health sensing, finding that zero-shot prediction can be competitive but reliable performance depends on accessing and grounding longitudinal evidence. The findings highlight selective memory, numerical grounding, and schema-aware reasoning as design needs.

6 Conclusion

  • Zero-shot agents can achieve competitive wellbeing-score prediction without task-specific training, but reliable performance depends on accessing and grounding longitudinal evidence.BALMS evaluates LLM-based agentic systems across three passive-sensing datasets, two task families, and three agentic paradigms.Source: §6 · p. 9
  • Chain-of-thought prompting benefits reasoning-oriented backbones, while longer histories and additional sensors help only with selective context retrieval and interpretation of semantically meaningful features.Source: §6 · p. 9
  • The findings highlight the need for longitudinal mental-health agents with selective memory, numerical grounding, and schema-aware reasoning.Source: §6 · p. 9

Limitations

BALMS evaluates representative single-agent designs on retrospective data, without testing real-world interactive or intervention settings. Its LLM-as-Judge rationale assessment does not replace expert clinical or human-subject evaluation.

Limitations

  • BALMS covers prompt-based, tool-based, and memory-based single-agent systems, but excludes multi-agent collaboration, planner-executor architectures, and agents dynamically combining tools, memory, and reflection.The benchmark therefore covers representative agentic paradigms rather than all possible agent designs.Source: Limitations · p. 9
  • Retrospective prediction on existing datasets leaves agent performance in real-world interactive and intervention settings untested.Future work should evaluate prospective deployment and study safety, calibration, and personalization in real-world mental-health support scenarios.Source: Limitations · p. 9
  • LLM-as-Judge rationale evaluation enables scalable comparison of temporal grounding, numerical correctness, and explanation quality, but cannot replace expert clinical or human-subject evaluation.Future work should incorporate clinician and user assessments.Source: Limitations · p. 9

Ethics Statement

The systems benchmarked by BALMS are intended for clinical decision support, not autonomous diagnosis. The ethics statement highlights misuse risks, de-identified research data, and limits to generalization across populations.

Potential Misuse

  • Benchmarked systems should support clinical decisions rather than replace professional care, because autonomous, uncalibrated mental health profiling risks false reassurance or unwarranted clinical anxiety.The authors explicitly reject using these systems as autonomous diagnostic replacements.Source: Ethics Statement · p. 9

Population Representation

  • Zero-shot model weights should not be generalized to heterogeneous clinical settings without proactive calibration and local alignment.Sensing patterns vary across demographic, cultural, and socioeconomic groups; benchmarked models may inherit or amplify data-sparsity patterns and behavioral biases from student or regional populations.Source: Ethics Statement · p. 9

Data Privacy

  • BALMS uses secondary analysis of previously collected, fully de-identified DiversityOne, PMData, and GLOBEM research cohorts.No protected health identifiers were accessed, stored, or processed. The authors state that the anonymized research data make this study non-human-subjects research under standard institutional guidelines, requiring no IRB review.Source: Ethics Statement · p. 9

A Data Processing Details … F5. Schema-blind aggregation on raw streams.

PHIA’s apparently competitive PMData performance reflects mode-collapsed predictions rather than wearable-grounded inference. Trace analysis links this behavior to empty outputs, lost execution state, invented thresholds, forced completion, and schema-mismatched aggregation.

A Data Processing Details

  • The default zero-shot wellbeing-score prediction setup uses daily-aggregated multivariate sensing signals with 7-day look-back windows for DiversityOne and 14-day windows for PMData and GLOBEM, except in analyses explicitly varying window length.Evaluations use Mongolia’s DiversityOne cohort for mood prediction (1,355 samples), all 16 PMData participants for stress prediction (1–5 Likert; 1,448 samples), and GLOBEM’s INS-W_1 cohort for PHQ-4 anxiety prediction (0–3; 1,640 samples). GLOBEM evaluation covers only one continuous 10-week study term and only the anxiety subscale, leaving across-term discontinuities to future work.Source: A Data Processing Details · p. 12
  • PHIA receives full user histories with a 10-iteration cap, while Chunk RAG and RAPTOR retrieve three historical candidates.The prompt-based baseline uses the Health-LLM template verbatim; Chunk RAG encodes historical days with a sentence-transformer and combines retrieved days with the query day. RAPTOR builds a two-level tree of sub-day leaves and LLM-generated daily summaries, which compete under the same retrieval budget.Source: A Data Processing Details · p. 12

B PHIA Failure Analysis

  • PHIA does not benefit from stronger backbones and is competitive only on PMData.The failure analysis uses Qwen2.5-14B-Instruct on seed 0, with findings consistent across Mistral-7B, Qwen2.5-7B, and DeepSeek-R1-Distill-Qwen-14B.Source: B PHIA Failure Analysis · p. 12

B.1 Predictions Collapse to a Single Label

  • 0.57 MAE versus the mean predictor’s 0.48 on PMData, and 1.52 versus 0.80 on GLOBEM, reflect PHIA’s mode collapse rather than sensor-grounded prediction.PHIA predominantly predicts label 3 on PMData and label 2 on GLOBEM; DiversityOne produces non-parseable answers in almost every window.Source: §B.1 · p. 12, Table 7 · p. 13

B.2 Failure Taxonomy

  • 2.0 empty observations per GLOBEM window and 0.7 per PMData window accompany silent execution failures; lost execution state further undermines numerical evidence.Missing print() calls or truncated expressions yield no output, yet the agent cites unproduced numbers; GLOBEM records 168 such events across 82 traced windows. Variables lost between action blocks cause NameError, prompting longer retries that often repeat silent execution failures.Source: §B.2 · pp. 12–13

F3. Force-finish with hallucinated rationale.

  • 8 out of 10 traced PMData windows ended with forced completion and rationales citing numerical evidence never computed during execution.When the step budget expires without a parseable answer, the harness requests a final label; the resulting rationale references unsupported deltas, thresholds, or trends.Source: F3. Force-finish with hallucinated… · p. 13
  • Invented thresholds drive mode collapse by placing almost every user on the same side of a hard-coded decision boundary.Examples on GLOBEM include steps > 20000 and sleep_minutes > 1680, neither derived from the user’s prior distribution.Source: F3. Force-finish with hallucinated… · p. 13

F5. Schema-blind aggregation on raw streams.

  • Schema-blind aggregation makes DiversityOne the worst case: even successfully executed code can produce semantically incorrect summaries from sub-day sensor rows.DiversityOne has approximately 18 thirty-minute slots per day; applying mean or sum without daily re-indexing conflates slot-level and daily summaries, unlike PMData’s pre-aggregated rows. Empty outputs and lost state leave no verifiable evidence, pushing the agent toward invented heuristics or budget-triggered forced completion.Source: F5. Schema-blind aggregation on raw… · pp. 13–14

B.3 Illustrative Traces · B.4 Implications

The illustrative traces show that parseable predictions can conceal uncomputed rationales or hardcoded-threshold decisions. Similar failures across backbones implicate the model–sandbox contract and motivate a memory-based approach that exposes history directly.

B.3 Illustrative Traces

  • A parseable integer made the PMData trace count as successful in aggregate MAE despite uncomputed rationale deltas and an essentially prior-driven prediction.Four truncated execution attempts produced no output; force-finish nevertheless returned 4 with unsupported claims about activity, sleep efficiency, and mood.Source: §B.3 · p. 14
  • 3 was eventually returned in the GLOBEM trace after undefined total_steps and filtered_df errors forced state reconstruction.The agent hardcoded step_threshold = 20000 and sleep_minute_threshold = 1680.Source: §B.3 · p. 14
  • 86% mode collapse on GLOBEM arises from repeated hardcoded-threshold decisions, with recovered aggregates merely selecting between two fixed branches.The illustrated trace eventually executes, but its prediction remains entirely threshold-determined.Source: §B.3 · p. 14

B.4 Implications

  • The same failure patterns occur across Qwen2.5-14B, Mistral-7B, and DeepSeek-R1-Distill-Qwen-14B, consistent with stronger backbones not improving PHIA’s MAE.Source: §B.4 · p. 14
  • The identified bottleneck is the model–sandbox contract—print discipline, state persistence, and threshold derivation—rather than language reasoning capacity.Source: §B.4 · p. 14
  • The memory-based paradigm exposes relevant history directly to the LLM, avoiding the silent-execution surface implicated by these failures.Source: §B.4 · p. 14

C Latency Analysis

The latency analysis measures average per-sample latency as look-back windows grow and compares prompting with and without chain-of-thought across backbones and datasets.

Latency vs. look-back window

  • Look-back latency analysis evaluates Qwen2.5-14B-Instruct on PMData and GLOBEM as the history window grows.Figure 7 accompanies the main paper’s look-back analysis.Source: C Latency Analysis · p. 14

Per-backbone latency panels

  • Qwen2.5-7B-Instruct panels report average per-sample latency across three datasets, comparing prompting with and without chain-of-thought.The Qwen2.5-14B-Instruct counterpart appears in the main paper’s Figure 6.Source: Figure 8 · p. 15, C Latency Analysis · p. 15
  • Mistral-7B-Instruct-v0.3 panels report average per-sample latency across three datasets, comparing prompting with and without chain-of-thought.Figure 9 covers each agentic system.Source: Figure 9 · p. 15, C Latency Analysis · p. 15
  • DeepSeek-R1-Distill-Qwen-14B panels report average per-sample latency across three datasets, comparing prompting with and without chain-of-thought.Figure 10 covers each agentic system.Source: Figure 10 · p. 15, C Latency Analysis · p. 15

D Prompt Templates

The appendix provides prediction and ReAct prompt examples alongside a two-tier rubric for evaluating rationales. A same-window case study shows how prediction error alone misses differences in evidence grounding and reasoning quality.

D Prompt Templates

  • The ReAct example alternates thoughts, code actions, and observations to inspect longitudinal data and compute daily aggregates until a final answer or iteration limit.It examines 14 days with four segments per day, notes missing values, and considers sleep, screen use, mobility, and social activity.Source: D Prompt Templates · p. 16
  • Health-LLM templates predict stress from wearable signals and self-reported context; the reasoning variant requests a score plus step-by-step justification.Stress ranges from 0 to 5, with look-back inputs including steps, calories, resting heart rate, sleep, and mood. The example assigns stress 3 using stable heart rate, inconsistent sleep, and mood.Source: D Prompt Templates · pp. 16–17
  • The judge evaluates each rationale using the canonical input window, ground-truth label, predicted label, and two-tier rubric.Llama-3.3-70B-Instruct is applied uniformly across backbones with greedy decoding for reproducibility, served through vLLM using 8-bit quantization and tensor parallelism of two.Source: D Prompt Templates · p. 17
  • Tier 1 checks temporal alignment, slicing, difference judgment, lag, structure, and interaction, scoring invoked dimensions as correct or incorrect without penalizing non-invocation.Scores are 1 for correct invocation, 0 for incorrect invocation, and null when absent. The dimensions cover temporal mapping, segment comparisons, changes, delayed effects, broader patterns, and joint-variable reasoning.Source: D Prompt Templates · p. 17
  • Tier 2 evaluates faithfulness, evidence grounding, domain knowledge, safety, clarity, and overall quality, distinguishing logical support for predictions from agreement with ground truth.Each dimension uses a 1–5 Likert scale, with Overall the headline score. Grounding penalizes invented features, incorrect values, and wrong-day attributions; justifications must cite rationale text or input values.Source: D Prompt Templates · p. 17
  • Benchmark aggregation reports Tier-1 conditional accuracy and invocation rate separately, headlines macro-averaged conditional accuracy, and averages non-null Tier-2 scores.Conditional accuracy is #correct ÷ #invoked and is undefined with no invocations; invocation rate is #invoked ÷ N.Source: D Prompt Templates · p. 18

D.1 Reasoning Case Study: Same Window, Two Backbones

  • 3-point Qwen and 2-point Claude prediction errors conceal different rationale quality: both overpredict anxiety, but only Claude’s rationale survives both rubric tiers.The shared GLOBEM window has ground-truth PHQ-4 anxiety 0. Claude’s moderate conclusion follows its observations, whereas Qwen’s higher-anxiety conclusion lacks supporting evidence.Source: §D.1 · p. 19
  • Two of four Qwen numeric claims have wrong-day attributions; Claude grounds every numeric claim, recognizes a cross-window trend, and integrates four channels.Qwen’s misattributed values exist elsewhere in the window. Claude identifies declining location entropy and jointly reasons about sleep, activity, screen use, and mobility; Qwen cherry-picks a contradicted change and lists features separately.Source: §D.1 · p. 19, Table 8 · p. 20

Related papers

Loading 2608.27219v1…