PaperlayerView exact revision on arXiv

Source-linked AI summary

HiMA-MDD: A Hierarchical Multi-Agent Harness for Interpretable Multimodal Depression Detection in Clinical Interviews

Ao Chen, Xiaojiang Peng

arXiv:2608.21868v1cs.AI

TL;DR

Multimodal depression assessment must integrate sparse, context-dependent interview evidence into a coherent PHQ-8 profile, while existing holistic and generic multi-agent systems do not fully govern this coordination. HiMA-MDD introduces a three-layer, measurement-aligned harness with bounded evidence routing, item-score ownership, verification, and trace preservation. On E-DAIC, the paper reports that HiMA-MDD outperforms compared methods, with calibration achieving the best displayed Total MAE and Macro-F1 values.

  • Problem

    Existing depression-assessment systems do not necessarily coordinate evidence access, PHQ-8 item-score authority, bounded feedback, and state recording across interview, symptom, and profile levels.

  • Method

    HiMA-MDD uses three governed layers for QA-to-item grounding, single-owner factor-specialist scoring, and global audit with targeted revision and verified-profile reconstruction.

  • Results

    HiMA-MDD outperforms compared methods on E-DAIC; calibrated output achieves Total MAE 3.4107 and Macro-F1 0.8130.

  • Takeaways & Limitations

    The harness organizes PHQ-8 assessment while maintaining an inspectable path from interview evidence to the final symptom profile.

  • Takeaways & Limitations

    The four-group organization is an operational responsibility map, not a newly validated PHQ-8 psychometric structure.

Abstract

from arXiv · show

Depression assessment from multimodal clinical interviews requires integrating dispersed evidence from multiple symptoms into a coherent PHQ-8 profile. This process is hierarchical: relevant evidence is often sparse and context-dependent within local question-answer exchanges, multiple exchanges jointly support symptom-level judgments, and the final assessment depends on the coherence of the complete symptom profile. Existing LLM systems either process interviews holistically or distribute work across generic agent roles; neither design necessarily provides an explicit orchestration mechanism that coordinates evidence access, item-score authority, bounded feedback, and state recording across these levels. To address this gap, we introduce HiMA-MDD, a hierarchical multi-agent harness that aligns this assessment hierarchy with three agent layers. After non-agentic preprocessing constructs context-preserving multimodal QA units, Layer 1 identifies candidate QA-to-item relations and supports bounded item-grounded evidence routing. Layer 2 assigns symptom groups to operational factor specialists, with one specialist responsible for each provisional item score. Layer 3 audits the complete provisional profile, requests at most one round of targeted revision, and reconstructs the verified PHQ-8 profile. This layered design naturally yields a Hierarchical Evidence Trace, preserves all intermediate evidence, judgments, and revisions for auditability. The final item scores then deterministically produce the total score and screening decision. Using Qwen2.5-72B-Instruct as the harness backbone, our experiments on E-DAIC demonstrate that HiMA-MDD outperforms the compared state-of-the-art methods.

Introduction

HiMA-MDD addresses a hierarchical measurement–coordination gap in multimodal depression assessment by aligning evidence, agent roles, and PHQ-8 item responsibility across three governed layers.

  • Motivation: Depression assessment must organize dispersed information from multimodal clinical interviews into reliable and timely symptom-level judgments.The motivation is both clinical and public-health oriented.
  • Research Gap: Existing systems do not necessarily make evidence access, scoring authority, and global revision explicit across interview, symptom, and PHQ-8 levels.Holistic systems can leave local evidence attribution implicit, while generic multi-agent pipelines may not align responsibility with PHQ-8 items.
  • Approach: HiMA-MDD maintains evidence–item responsibility from local multimodal exchanges to the final PHQ-8 profile through three governed agent roles.The roles cover QA-to-item grounding, factor-level symptom reasoning, and global symptom verification.
  • Contribution: The harness preserves evidence links, intermediate judgments, audit findings, and revisions in a Hierarchical Evidence Trace for inspection.This trace supports auditability across the evidence-to-profile process.

Related Work

Prior work spans holistic interview prediction, finer-grained questionnaire-aware assessment, and functionally specialized multi-agent systems, but PHQ symptom structure remains non-universal.

  • Holistic Assessment: Early multimodal depression systems commonly predicted total scores or screening labels from complete interviews.This represents a holistic assessment route.
  • Finer-Grained Assessment: Recent methods use transcript, facial-expression, and multimodal language-model features, while others target questionnaire items, subscores, or separately retrieved evidence.These approaches move toward richer multimodal or finer-grained assessment.
  • Multi-Agent Assessment: Multi-agent systems distribute questioning, response assessment, scoring, updating, judging, and review across specialized functional roles.Examples include MDAgents, AgentMental, MAGI, and AI Psychiatrist Assistant.
  • Psychometric Context: Psychometric research distinguishes summed severity from individual symptom profiles and does not establish one universally accepted PHQ structure.HiMA-MDD therefore treats its four symptom groups as an operational responsibility map rather than a validated psychometric model.

Method

HiMA-MDD implements an explicit execution contract that routes context-preserving multimodal evidence through grounding, single-owner symptom reasoning, verification, and deterministic profile aggregation.

  • Harness Control Interface: The harness governs evidence access, provisional score ownership, revisions, outputs, and retained state transitions.It functions as an orchestration and control layer rather than merely a sequence of modules.
  • Outputs: Given a completed interview, the core harness produces eight raw verified PHQ-8 item scores, their total score, a screening decision, and a Hierarchical Evidence Trace.The optional calibration interface operates after the raw state is frozen.
  • Execution Contract: The formal harness interfaces define item rubrics, evidence routing, one provisional score owner per item, verification, deterministic aggregation, and retained provenance.These interfaces encode the system’s governance invariants.
  • Hierarchical Processing: The three layers ground QA-to-item relations, assign routed evidence to factor specialists, and audit, revise, and reconstruct the verified profile.The raw item vector is then passed through deterministic aggregation while the trace retains intermediate states.
  • Data and Measurement Structuring: Multimodal QA units pair each interviewer question with consecutive participant responses, turn and timestamp metadata, and aligned acoustic descriptors.Retaining the question supplies local context for short or elliptical responses.
  • Measurement Contract: The measurement contract provides eight 0–3 item rubrics over the preceding two weeks, symptom-group responsibilities, and fixed total-score and screening rules.Acoustic descriptors are aligned to QA units by timestamp and used as auxiliary cues.
  • Layer 1: The grounding agent identifies candidate PHQ-8 items and evidence polarity without determining symptom severity or producing item scores.Its outputs support bounded item-grounded evidence routing.
  • Layer 2: Four factor specialists divide provisional scoring across affective, somatic, internalizing, and sensorimotor symptom groups.The Affective, Somatic, Internalizing, and Sensorimotor specialists own the listed PHQ-8 symptoms.

Experiments

Experiments evaluate HiMA-MDD on E-DAIC and DAIC-WOZ using PHQ-8 item, total-score, and screening metrics, alongside baseline, ablation, granularity, and evidence-trace analyses. HiMA-MDD achieves strong held-out E-DAIC performance, robust DAIC-WOZ screening results, and component-level evidence supporting audit, reconstruction, and acoustic inputs.

  • Datasets and protocol: HiMA-MDD was evaluated on E-DAIC and DAIC-WOZ using ASR-derived transcripts and participant-speech audio, with E-DAIC as the primary held-out test evaluation.E-DAIC contains 275 participants and an autonomous-interview test set; DAIC-WOZ provides an additional development-split robustness evaluation.
  • Baselines and metrics: The comparison used Zero-Shot, 3-Shot, CoT, MDAgents, and AgentMental under a shared Qwen2.5-72B-Instruct backbone, PHQ-8 rubric, and output format.Methods predicted eight item scores, which were summed to produce total scores and screening decisions.
  • Main results: 3.4107 Total MAE and 0.8130 Macro-F1 were achieved by HiMA-MDD with post-hoc calibration on the held-out E-DAIC test split.The calibrated system improved every reported metric over the raw harness output and had the best displayed Total MAE and Macro-F1 values.
  • Robustness evaluation: 0.8381 Macro-F1 was achieved by raw HiMA-MDD on the DAIC-WOZ development split, rising from 0.7552 for the best-performing baseline.HiMA-MDD had the highest accuracy, screening κ, and Macro-F1, while its Total MAE was higher, showing different rankings for screening and absolute error.
  • Component ablations: Removing cross-factor audit and targeted revision produced the largest decrease among tested components, while removing centralized reconstruction or acoustic descriptors reduced screening performance.The ablation results support the contributions of audit–revision, verified-profile reconciliation, and participant-speech cues.
  • Evidence trace: The Hierarchical Evidence Trace records retained evidence for Depressed Mood and aggregates three evidence spans for Feelings of Failure into inspectable item judgments.For participant 716, the Depressed Mood score and rationale were retained after an untargeted re-evaluation, while Feelings of Failure received a multi-evidence score.

Conclusion

HiMA-MDD organizes PHQ-8 assessment across evidence grounding, symptom reasoning, and global verification, producing an inspectable path from interview evidence to the final profile.

  • HiMA-MDD produces item scores, total severity, screening decisions, and a recorded Hierarchical Evidence Trace from completed multimodal clinical interviews.
  • Results on E-DAIC support measurement-aligned governance for organizing PHQ-8 assessment while maintaining an inspectable path from interview evidence to the final symptom profile.

E-DAIC

This study uses E-DAIC and related clinical-interview resources to construct multimodal PHQ-8 inputs, evaluate screening from complete item profiles, and preserve the measurement contract underlying those decisions.

  • Dataset: E-DAIC contains 275 participant sessions with audio, video-derived descriptors, automatic transcripts, and depression labels; its official test partition contains autonomous-agent sessions.
  • Dataset: The study uses processed transcripts and participant-speech audio from E-DAIC, while DAIC-WOZ supplies an additional 35-participant development-split robustness evaluation.
  • Preprocessing: The preprocessing reconstructs interviewer structure from time-stamped Whisper transcripts using the 85-question DAIC-WOZ inventory and repairs unresolved turn boundaries.
  • Multimodal Inputs: Participant-response audio is represented with eGeMAPSv02 functionals and auxiliary depression and anxiety outputs from the DAM model.
  • Measurement Contract: PHQ-8 covers eight symptom categories, assigns each item a value from 0 to 3, and sums items to a total ranging from 0 to 24.
  • Measurement Contract: The four responsibility groups are an operational evidence-routing choice rather than a newly validated psychometric structure.
  • Measurement Contract: Every system predicts the complete eight-item profile, whose scores are summed deterministically; totals of 10 or above enter the depressed screening range.

Appendix B: Baselines and Evaluation Protocol

The evaluation compares HiMA-MDD with prompt-based and multi-agent baselines under a shared backbone, PHQ-8 task, and output format.

  • Output Protocol: For each participant, systems predict eight PHQ-8 item scores that are summed into a total score and screening decision.

Metrics

The metrics evaluate both aggregate PHQ-8 severity estimation and thresholded screening, with exploratory tests aligned to specific comparisons and outcomes.

  • Total-Score Metrics: Total MAE and Total RMSE measure aggregate severity-estimation error from predicted and gold PHQ-8 totals.
  • Screening Metrics: Screening uses a threshold of 10 on the PHQ-8 total to derive predicted and gold classes for accuracy and agreement measures.
  • Agreement: Cohen’s κ compares observed and expected class agreement, while item κ applies the same unweighted definition across four ordinal score categories and averages across eight items.
  • Reported Metrics: The evaluation reports control-range F1, depressed-range F1, Macro-F1, Total MAE, Total RMSE, accuracy, and screening κ.
  • Statistical Tests: Exploratory nominal p-values are unadjusted for multiple comparisons, with metric-specific tests using aligned predictions and zero-shot LLM as the sole star-marker reference.

Implementation and Reproducibility Details

The implementation uses specified preprocessing, multimodal feature, hardware, evidence-retention, and calibration settings. These settings constrain evidence access and reproducibility across controlled runs.

  • Preprocessing and evaluation: The protocol thresholds both gold and predicted PHQ-8 totals at 10 and uses Whisper turbo with time-stamped transcript segments.The interview inventory contains 85 DAIC-WOZ questions with primary and follow-up types.
  • Preprocessing and evaluation: Question matching uses Sentence-Transformers all-mpnet-base-v2 with normalized embeddings and cosine-equivalent dot products.Audio is represented at 16 kHz mono using openSMILE eGeMAPSv02 functionals.
  • Resource and orchestration limits: The system permits up to 8 item-grounded QA records per PHQ-8 item, 24 deduplicated records per operational factor, and 1 targeted specialist revision round.Experiments run on an NVIDIA RTX 5880 Ada Generation GPU with 48 GB memory.
  • Calibration: The full post-hoc calibration feature dimension is 376, with standardized Ridge, Bayesian Ridge, and Elastic Net models compared using development-selected hyperparameters.Models are fitted on the training partition and selected on the development partition.
  • Caching and controlled runs: The evidence-index cache keys on the symptom schema, prompt version, model, temperature, and multimodal QA-unit content hash.Shared cached QA-to-item relations let controlled-granularity runs vary Factor-Specialist configuration and evidence access without rerunning Layer 1 grounding.

Appendix C: Significance-Test Details

The appendix reports the statistical tests underlying star markers in the main comparison. Tests use Zero-Shot as the reference, with exploratory nominal p-values that are unadjusted for multiple comparisons.

  • Test setup: Table 7 reports metric-specific one-tailed p-values for tests using Zero-Shot as the reference condition.These tests underlie the star markers in the main comparison.
  • Calibration condition: The HiMA-MDD + Post-hoc Calibration condition uses the selected supervised Bayesian Ridge ordinal score-correction model.This specifies the calibration model used for the tested condition.
  • Statistical qualification: The reported significance values are exploratory, nominal, and unadjusted for multiple comparisons.Interpretation should therefore retain the appendix's stated qualification.
Loading 2608.21868v1…