Source-linked AI summary
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment
Merna Bibars, Bolaji Omofojoye, Allan I. Levey, Rachel Hershenberg, Gari D. Clifford, Hyeokhyen Kwon
TL;DR
Depression and anxiety in older adults with MCI are underdiagnosed, motivating scalable analysis of remote interviews. TAMI aligns multimodal features, encodes time-varying missingness, conditions fusion on questions, and attributes predictions across modalities, questions, and time. In 49 participants, it reached AUROC 0.68 for depression and 0.69 for anxiety, while open-ended responses alone produced depression AUROC 0.67 versus the full interview’s 19 minutes.
Problem
Existing multimodal interview methods overlook temporal alignment, time-varying modality missingness, question context, and joint interpretability for depression and anxiety screening in older adults with MCI.
Method
TAMI aligns multimodal features within question-answer segments, encodes modality-timestep missingness, conditions fusion on question context, and derives multi-level attributions.
Results
0.68 depression AUROC and 0.69 anxiety AUROC were achieved, with temporal alignment providing the largest performance gain; open-ended responses yielded 0.67 depression AUROC.
Takeaways & Limitations
A shorter open-ended protocol may be sufficient for depression screening in older adults with MCI.
Takeaways & Limitations
The study is limited by its small sample size of N = 49 and feature extractors pretrained on general-population data.
Abstract
from arXiv · showhide
Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68 (depression) and 0.69 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain ($Δ{\geq}0.1$). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1min), the depression model achieved an AUROC score of 0.67, which was not significantly different from using the full interview (19min) ($p>0.05$). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI.
I. INTRODUCTION
Older adults with MCI face clinically important, underdiagnosed depression and anxiety, while remote multimodal interviews offer scalable behavioral assessment. TAMI addresses temporal misalignment, modality missingness, and limited question- and time-specific interpretability.
- Depression affects 32% of adults with MCI, while anxiety affects 10–50%, and both are associated with increased dementia risk.
- Remote interviews can quantify facial, vocal, language, and physiological cues for scalable mental-health screening.
- Prior methods pair multimodal features by sequence index rather than original timestamps, associating multi-second windows with single frame observations.
- TAMI also targets joint attribution across modalities, questions, and interview moments, which prior approaches do not provide comprehensively.
- Remote recordings exhibit time-varying modality dropout from blur, occlusion, and frame loss, while zero-filling obscures missingness versus valid near-zero measurements.
- TAMI combines fine-grained temporal alignment, modality-level missingness, and question-conditioned multimodal fusion for depression and anxiety prediction.
C. Handling Missingness across Modality and Time
TAMI represents interviews as question-answer sequences and uses answer-aligned temporal bins with fine-grained missingness masks. It combines question-conditioned fusion with joint attribution across modalities, questions, and time.
- Prior methods either ignore question context or condition limited modalities and temporally pooled representations rather than joint cross-modal behavior at each moment.
- TAMI derives modality, question, and interview-wide temporal importance from integrated-gradients attributions at cohort and individual levels.
- TAMI processes interviews through aligned multimodal time series with per-modality-feature timestep masks, question-conditioned fusion, and post-hoc multi-level attribution.
- Each participant is represented by observed question-answer segments, allowing the number of answered questions to vary across participants.
- The model uses facial, physiological, acoustic, and language feature series, with interview questions encoded separately to condition participant-side features.
2) Answer Segmentation and Temporal Alignment:
TAMI partitions each answer into adaptive temporal bins that preserve answer coverage and align features according to timestamps and temporal overlap. It then projects masked feature series and fuses them into multimodal tokens.
- Answer Segmentation and Temporal Alignment: Each answer is divided into adaptive temporal bins rather than truncating long answers or padding short ones.
- Answer Segmentation and Temporal Alignment: Frame-level features enter the bin containing their timestamp, while window-level features enter every bin their windows overlap.
- Modality-Timestep Missingness: Missing feature vectors are zero-filled, but availability masks distinguish missing observations from valid zero-valued measurements.
- Projection Backbones and Fusion: The linear backbone projects each feature independently, whereas the nonlinear backbone applies per-feature temporal self-attention before fusion.
- Projection Backbones and Fusion: Projected feature representations and availability masks are concatenated and projected into fused multimodal tokens across the participant sequence.
2) Question Conditioning and Prediction:
TAMI conditions each aligned multimodal token on the question associated with its answer before sequence modeling and prediction. Integrated-gradients attributions then summarize feature importance across modalities, questions, and time.
- Question Conditioning and Prediction: Each temporal-bin token receives the embedding of its associated question before processing by a Transformer encoder.
- Question Conditioning and Prediction: A prepended [CLS] representation is mapped by a linear head to the depression or anxiety prediction logit.
- Interpretability: Integrated gradients assigns a scalar attribution to each feature at each temporal bin using a standardized mean feature-value baseline.
- Interpretability: Attributions are aggregated into modality importance, question importance, within-question modality importance, and temporal importance.
- Dataset: The dataset contains remote semi-structured interviews from 49 older adults with MCI, with participant-side interview durations ranging from 21 to 67 minutes.
B. Implementation and Evaluation
TAMI was evaluated through participant-independent repeated cross-validation and ablations of temporal alignment, question conditioning, and missingness masking. Temporal alignment produced the strongest AUROC results, while feature importance differed between depression and anxiety.
- Evaluation: 15 runs used participant-independent 5-fold cross-validation across 3 repetitions, with training-fold standardization and AUROC reporting using 95% confidence intervals.Additional metrics were reported in the Supplementary.
- Ablation design: Temporal alignment and question conditioning were evaluated under linear and non-linear projection backbones against sequence-index-paired baselines.T denotes temporal alignment and Q denotes question conditioning.
- Missingness masking: The modality–feature-timestep mask improved AUROC in three of four model-task combinations, with the largest gain for Linear+T anxiety and mixed effects for depression.The no-mask variant made zero-filled bins indistinguishable from observed values.
- Temporal alignment: 0.68±0.04 AUROC was achieved for depression by Non-linear+T, a Δ=+0.10 improvement over the Non-linear baseline.The temporally aligned model achieved the highest depression score in the reported comparison.
- Temporal alignment: 0.69±0.09 AUROC was achieved for anxiety by Linear+T, a Δ=+0.11 improvement over the Linear baseline, with p=0.05.Temporal alignment achieved the highest anxiety score in the reported comparison.
- Feature importance: Eyegaze accounted for 66% of depression attribution, while anxiety attribution was divided between eyegaze at 44% and head pose at 37.2%.Together, the two leading features accounted for more than 77% of total attribution in each task.
C. Question Importance
Question-level attribution was distributed across the interview but differed by task: depression emphasized open-ended questions and early interview moments, whereas anxiety showed a more uniform temporal profile. Open-ended responses alone retained depression performance close to the full-interview model.
- Question-group attribution: Depression showed wider question-group attribution dispersion, with open-ended questions receiving the highest mean attribution.The open-ended set comprised Q1–Q4, including self-description, best and worst events, and picture description.
- Question-group attribution: Anxiety showed a tight group-mean attribution range of 0.24 × 10^-3, indicating roughly uniform reliance on the full interview.Its highest-attribution questions came from multiple question groups.
- Question-group attribution: Neither GDS nor GAD-7 ranked among the top question groups for its corresponding task, indicating reliance on behavioral responses across the video.The result supports examining multimodal behavioral biomarkers beyond the screening tools.
- Temporal attribution: Depression attribution peaked early and declined from ∼2.0 to ∼0.75 × 10^-3 by mid-interview, aligning with the open-ended questions.Eyegaze attribution declined from ∼1.25 to ∼0.50 × 10^-3 over the same period.
- Temporal attribution: Anxiety showed a uniform temporal importance profile, with total attribution decreasing near the interview end before returning to its earlier level.Eyegaze and head-pose attributions remained relatively uniform throughout the interview.
- Interview shortening: 0.67 ± 0.13 AUROC was achieved by the depression model using only four open-ended responses over a median 5.1-minute duration.The open-ended-only model was evaluated with the Non-linear+T model.
- Fusion design: Temporal alignment was the primary driver of performance across both tasks, while question conditioning did not improve temporally aligned models.Unaligned Q-only variants showed no consistent gains over baselines.
D. Modality Importance
TAMI’s attributions concentrated on eyegaze for both outcomes, while depression emphasized open-ended questions and anxiety-related importance was distributed more broadly across the interview.
- Feature-level attribution: Eyegaze accounted for 66% of total depression attribution and was the dominant feature for both depression and anxiety.Anxiety attribution was distributed more evenly between eyegaze and head pose.
- Question-level attribution: Attribution was distributed across question groups rather than concentrated in the groups defining each label.GDS ranked last of six groups for depression, while GAD-7 ranked second to last for anxiety.
- Question-level attribution: Depression attribution was highest during the open-ended question block and declined across the later structured section.This pattern is consistent with prior evidence favoring spontaneous speech over constrained sentence reading for depression prediction.
- Interview-length analysis: Open-ended responses produced a depression AUROC of 0.67 versus 0.68 for the full interview, without a significant difference (p > 0.05).The open-ended responses had a median duration of 5.1 minutes, compared with 19 minutes for the participant-side full interview.
- Limitations: The study’s principal limitation was its small sample size (N = 49), alongside possible reduced reliability of facial and acoustic extractors for older adults with MCI.The authors call for validation in larger and more diverse cohorts.
A. Feature Extraction
TAMI extracts facial, acoustic, language, and physiological signals at different temporal resolutions, then represents them as aligned, missingness-aware feature time series for attribution analysis.
- Feature inputs: The model uses eight modality-feature time series spanning facial, acoustic, language, and physiological inputs.Inputs include action units and landmarks, head pose, eyegaze, eGeMAPS, ComParE, Wav2Vec2, rPPG heart rate, and transcript embeddings.
- Temporal resolutions: Window-level features summarize signal content over intervals, whereas frame-level features are measured at individual timestamps.This distinction reflects the different temporal supports of acoustic, physiological, and facial measurements.
- Facial features: Facial features are extracted at 1 fps, and bins are marked missing when face detection or gaze estimation fails.Failure sources include camera viewpoint, internet connectivity, and lighting.
- Acoustic, language, and physiological features: Acoustic and physiological features use overlapping windows, while transcript embeddings are computed from words assigned to temporal bins.Acoustic windows are 2 seconds with 1-second overlap; transcript embeddings have 768 dimensions per bin.
- Attribution preparation: Integrated-gradient attributions are averaged across feature dimensions and gated by availability masks so missing bins contribute zero attribution.The resulting per-bin scores support aggregation across modality-features, questions, and time.
2) Modality Importance:
TAMI aggregates feature-level integrated-gradient scores into modality importance while accounting for observed bins and participant-level availability.
- Missingness-aware aggregation: Attributions are averaged per observed bin so modality-features with more missing temporal bins are not penalized for contributing fewer terms.The normalization uses the number of observed bins for each modality-feature.
- Feature-level attribution: Feature-level attribution scores are reduced to one score per temporal bin by averaging absolute attributions across feature dimensions and applying missingness masks.Unavailable bins contribute zero attribution.
- Aggregation outputs: Per-bin attribution scores are aggregated to produce modality, question, within-question modality, and temporal importance measures.These aggregations organize attribution across the interview’s main interpretability axes.
- Participant normalization: Modality-feature contributions are normalized within participants over observed modality-features, with entirely absent features assigned zero importance.This makes contributions comparable across participants despite modality availability differences.
3) Question Importance:
Question importance is computed by normalizing attribution within each answered question and averaging across participants who provided observations for that question.
- Question-level normalization: Question importance is normalized by each question’s number of observed bins to reduce bias toward longer answers.The resulting values are averaged over participants who answered the question.
- Within-question modality importance: Within-question modality analysis tracks observed bins for each modality-feature and excludes modality-features absent throughout a question.Per-bin attribution is normalized by observed-bin count before within-participant comparison.
- Dataset-level aggregation: The dataset-level within-question modality-feature importance is reported as an aggregation over participants for each question and modality-feature.This measure is shown in Fig. 11.
5) Temporal Importance:
Temporal importance is summarized on a normalized interview timeline by averaging within-answer bins and across participants. The data also show substantial modality- and question-dependent missingness, especially for language features.
- Temporal aggregation: U = 20 fixed-width buckets represent normalized interview positions, with each bucket spanning 5% of interview duration.Buckets enable cohort-level temporal summaries despite participant-specific bin counts and interview lengths.
- Temporal aggregation: Each participant’s bin importance is averaged within a bucket before participant-level averaging, preventing participants with more bins from contributing more.This produces a mean per-bin temporal importance at each relative interview position.
- Interview protocol: The interview protocol combined open-ended prompts with social, depression, anxiety, loneliness, and social-engagement question groups.The fixed-order protocol comprised six consecutive groups.
- Interview protocol: Some questions were omitted through skip logic or protocol changes, and occasional spoken answers were marked unanswered because timestamps were manually annotated.These omissions represent design- and annotation-related missingness in addition to feature-extraction missingness.
- Modality missingness: 37.0% of language-aligned bins are missing overall, compared with less than 0.1% for acoustic features and 1.4–3.0% for visual and physiological features.One participant had no valid rPPG signal across the full interview.
- Question-dependent missingness: Open-ended questions elicited longer responses and 16% text missingness, whereas structured question groups had shorter answers and up to 49.3% missingness.The UCLA Loneliness Scale had the highest reported rate at 49.3%.
2) Question-Conditioning Models:
The question-conditioning models incorporate answered-question embeddings into fused multimodal sequences using additive or cross-attention mechanisms. The models are evaluated alongside temporally aligned and unaligned variants, with question-level attributions identifying task-specific contributors.
- Pooled additive conditioning: Question-conditioned multimodal tokens are formed by adding a learned mixture of a participant’s question embeddings to each fused timestep.The conditioned sequence is then passed to the Transformer encoder.
- Cross-attention conditioning: Cross-attention conditioning lets each fused multimodal timestep attend to the participant’s answered-question embeddings.The timestep supplies the query, while projected question embeddings supply keys and values.
- Model comparisons: T denotes temporally aligned features and Q denotes question conditioning in the classification comparisons.Linear and non-linear refer to the alternative projection backbones.
- Model comparisons: Pairwise model comparisons report mean AUROC differences and one-sided paired t-tests testing whether the first model outperformed the second.Bold p values indicate p < 0.05.
- Question-level attribution: For depression, the highest-attribution questions concerned social contact and support, self-rated health, and open-ended self-description.The five listed question attributions ranged from 1.509 × 10^-3 to 2.151 × 10^-3.
- Modality-feature attribution: Eyegaze remained the top within-question contributor for depression, while eyegaze and head pose retained their relative importance for anxiety across question groups.The modality-feature rankings from the global analysis were preserved at the question level.
C. Single Participant Temporal Importance
Single-participant integrated-gradients analysis shows distinct temporal attribution patterns for depression and anxiety. Depression attribution peaks during the open-ended block, whereas anxiety contributions remain comparatively stable across questions.
- Analysis setup: The temporal plots show integrated-gradients attribution for the three highest-contributing modality-features for one representative participant per task.Selected participants came from test folds of the best-performing models, and scores were smoothed using a 15-s window.
- Depression: For depression, eyegaze had the highest attribution across all questions, with total attribution peaking during the open-ended block.ComParE contributed mainly at the beginning of the interview, while attribution decreased through the middle.
- Anxiety: For anxiety, eyegaze and head pose contributed comparably across all questions, while rPPG contributed mainly during the open-ended block.The rPPG contribution decreased afterward.
- Analysis setup: The depression analysis considered eyegaze, ComParE, and head pose, whereas the anxiety analysis considered eyegaze, head pose, and rPPG.These were the three highest-attribution modality-features for the respective task.