Source-linked AI summary
Multimodal Detection of Higher-Order Behavioral Constructs: Self-Compassion in Structured Reflective Interaction
Siddhant Jain, Dimitra Tsovaltzi
TL;DR
Higher-order constructs such as self-compassion are difficult to model from behavior unfolding across time, and their expression during reflective interaction has been little studied beyond self-report. The paper builds a 51-session dataset and a multimodal, window-based pipeline, then compares unimodal models with probability-level late fusion. Fusion adds a small but real improvement over the best single modality, while performance varies with class support.
Problem
Higher-order behavioral constructs unfold across modalities and time, while self-compassion in moment-to-moment reflective speech has been little examined beyond self-report questionnaires.
Source: §1 · pp. 1–2Method
The authors annotate 51 reflective-dialog sessions, consolidate six theoretical components into three modeling classes, align video, audio, and text, and evaluate unimodal models with probability-level late fusion.
Source: §1 · p. 2, §3.2 · p. 3Results
Fusion adds a small but real improvement over the best single modality, while detection varies by class support and a majority-class baseline competes for the best-supported class.
Source: §12 · p. 8Takeaways & Limitations
The conclusion is that a theory-grounded higher-order construct can be modeled with a three-class scheme and shared multimodal pipeline, though class detection remains uneven.
Source: §12 · p. 8The effective sample size is bounded by 51 sessions, despite 155,960 extracted windows, and the evaluation was descriptive without significance testing or split-sensitivity analysis.
Source: §9 · p. 7
Abstract
from arXiv · showhide
Many of the qualities that matter most in how people learn and grow, how someone regulates their emotions, reflects on a setback, or stays aware of others during a difficult conversation, are not directly observable. They have to be inferred from how someone speaks, moves, and sounds over time, and they resist the kind of clean labeling that most machine learning pipelines are built around. We study this challenge through a case that is well grounded in psychological theory but rarely modeled computationally: self-compassion, the tendency to respond to one's own setbacks with patience rather than harsh self-criticism. We examine how it appears during structured reflective interviews in a technology-mediated training setting, where people naturally talk through socio-emotionally demanding situations. Since no existing dataset captures this kind of construct in this kind of setting, we collected and annotated 51 reflective dialog sessions using an independent, temporally overlapping annotation scheme grounded in established theory. We consolidate the underlying six-component psychological model into a three-class supervision space, balancing self-kindness and mindfulness against self-critical or overwhelmed states, and build a reproducible window-based pipeline that aligns video, audio, and text on a shared timeline. Unimodal models trained on each modality separately are compared against a simple probability-level fusion strategy, which yields modest but consistent gains over the best single modality. We close by discussing where each modality succeeds or struggles, what this suggests about how this kind of construct is actually expressed in reflective speech, and what would be needed to model it, and constructs like it, more effectively.
1 Introduction
The paper studies self-compassion as a theory-grounded but rarely computationally modeled example of higher-order behavior expressed during reflective interaction. It motivates a multimodal, temporally aligned approach because such constructs unfold across language, prosody, and movement, while prior relevant work has narrower settings and supervision.
1 Introduction
- Higher-order constructs are difficult to model because their signals unfold over time across language, prosody, and movement, while annotations are sparse, extended, and overlapping.These theory-driven constructs are not directly measurable like basic emotion categories.Source: §1 · p. 1
- Self-compassion has an established psychological framework but has rarely been studied beyond self-report questionnaires or as it appears moment to moment in spoken reflection.Reflection can intensify rumination or harsh self-evaluation when emotional regulation is ineffective.Source: §1 · p. 2
- The paper contributes a three-class modeling representation, a shared window-based video-audio-text pipeline, unimodal evaluations, and interpretable probability-level late fusion.The authors present baseline results for a construct, setting, and formulation they say have not previously been addressed together.Source: §1 · p. 2
2.1 Self-Compassion as a Regulatory Construct
- Self-compassion comprises three bipolar dimensions, but its positive and negative poles can co-occur and fluctuate independently, supporting a multi-label formulation [17, 18, 25].The dimensions distinguish kindness from judgment, common humanity from isolation, and mindfulness from over-identification.Source: §2.1 · p. 2
2.2 Behavioral Correlates of Self-Compassion
- Prior self-compassion studies found measurable facial and vocal behavior, but used therapy sessions, unimodal signals, and event-based supervision rather than aligned multimodal windows.The paper extends this evidence to structured professional reflection and overlapping multi-label components.Source: §2.2 · p. 2
2.3 Multimodal Modeling of Complex Behavioral Constructs
- Because reflective-dialog data involve long sessions, overlapping labels, and few sessions, the paper favors modular unimodal models and low-capacity late fusion over higher-capacity joint architectures.Probability-level fusion avoids additional trainable cross-modal parameters and keeps modality contributions interpretable.Source: §2.3 · p. 2
- The authors identify limited evidence on whether general-purpose or instruction-following language models capture theory-grounded, multi-component self-compassion without task-specific adaptation, motivating text fine-tuning.Source: §2.3 · pp. 2–3
- The paper positions self-compassion in structured reflection as one concrete route into the broader underexplored problem of modeling higher-order, theory-grounded constructs beyond benchmark emotion categories.Source: §2.3 · p. 3
3 Problem Formulation
The formulation asks whether a six-component theory can be represented faithfully under the corpus’s support constraints, aligned across modalities, and evaluated with interpretable late fusion. The authors retain overlapping labels while consolidating sparse components for modeling.
3.1 Research Questions
- The research questions ask how to consolidate theory under empirical support limits, align facial, acoustic, and linguistic signals, compare modalities, and assess low-capacity late fusion.Source: §3.1 · p. 3
3.2 Label Consolidation
- The modeling scheme retains Mindfulness (MF), Self-Kindness (SK), and Negative components (NEG), with NEG combining Self-Judgment and Over-Identification.Common Humanity and Isolation are excluded because their temporal coverage is sparse.Source: §3.2 · p. 3
- The original six constructs were annotated as overlapping tiers, but Common Humanity and Isolation had only 121s and 15s of annotated duration, compared with 7,137s for MF and 3,056s for SK.The consolidation is a response to corpus constraints, not a claim that excluded or aggregated constructs are theoretically equivalent.Source: §3.2 · p. 3
3.3 Component Co-Occurrence
- Because self-compassion components can fluctuate independently, a single temporal window may contain multiple labels, such as mindful awareness alongside partial self-criticism.Source: §3.3 · p. 3
- Figure 2 depicts independent ELAN tiers that permit temporal overlap between self-compassion components.Source: Figure 2 · p. 3
3.4 Multi-Label Formulation
- The task assigns independent binary labels for MF, SK, and NEG, with separate sigmoid-linked probabilities rather than cross-class normalization.Source: §3.4 · p. 3
4 Dataset and Annotation
The dataset comprises 51 reflective-dialog sessions, independently annotated across overlapping construct tiers and converted into window-level supervision. Limited coverage across original constructs informed consolidation, and video-level splits were used to limit temporal leakage.
4.1 Data Collection
- 51 annotated reflective-dialog sessions capture participants’ post-simulation reflection, with synchronized audio and video and timestamp-aligned transcripts derived through automatic speech recognition.Sessions ranged from 600s to 4,956s, with a mean duration of approximately 3,489s.Source: §4.1 · p. 3
4.2 Annotation Scheme
- Constructs were annotated as independent ELAN tiers with start and end times, allowing temporal overlap; about 15% of sessions received independent second coding, without a numeric agreement statistic.Source: §4.2 · p. 3
- The annotation audit found that MF and SK covered most annotated duration, SJ and OI were unevenly represented, and CH and ISO had limited coverage.Table 1 reports total annotated duration and video coverage per original construct; the imbalance motivated consolidation.Source: §4.2 · p. 4, Table 1 · p. 4
4.3 Window-Based Supervision
- A positive window label requires at least 0.8 seconds of overlap with a class interval in each 4-second window, reducing sensitivity to marginal boundaries while retaining short annotations.The 1-second-hop segmentation produced 155,960 windows, of which 10,658 (6.83%) had at least one positive label.Source: §4.3 · p. 4
4.4 Splits
- Splits were made at the video level in an 80/10/10 train/validation/test ratio to prevent temporal leakage, with fitting, standardization, and imputation restricted to training data.Evaluation used the 1,425 test windows carrying at least one valid supervision label.Source: §4.4 · p. 4
5 Multimodal Preprocessing and Alignment
The preprocessing pipeline standardizes recordings before aligning video, audio, transcripts, and annotations to a common temporal reference. Modality-specific features are then extracted or projected onto the shared four-second windows.
5 Multimodal Preprocessing and Alignment
- Recordings with heterogeneous resolutions, frame rates, and bitrates are standardized before cross-modal alignment, and modalities share a common temporal reference.Figure 3 illustrates the alignment of raw video, audio, transcribed language, and ELAN annotations into learning segments.Source: §5 · p. 4, Figure 3 · p. 4
5.1 Video
- Video features comprise MediaPipe facial blendshape coefficients and upper-body pose landmarks extracted at 10 frames per second, yielding 40 time steps per four-second window.Figure 4 illustrates the feature extraction.Source: §5.1 · p. 4, Figure 4 · p. 4
5.2 Audio / 5.3 Text
- Audio uses diarization and eGeMAPSv02 acoustic features pooled to windows, while text uses Faster-Whisper German transcripts classified by segment and projected onto the same four-second windows.Source: §5.2 · p. 4, §5.3 · pp. 4–5
6 Modeling and Fusion
The section describes a shared multi-label modeling and evaluation protocol for video, audio, and text, followed by probability-level late fusion of independently trained modality models.
6 Modeling and Fusion
- Each modality is modeled independently with three sigmoid outputs for Mindfulness, Self-Kindness, and Negative components under the same multi-label formulation.Text is projected from segment-level predictions to the window-level evaluation protocol.Source: §6 · p. 5
6.1 Unimodal Models
- Video uses a GRU over frame sequences, audio uses one-vs-rest logistic regression over eGeMAPS features, and text uses context-enhanced LLaMA-3.2 fine-tuning.The exploratory audio MLP did not yield consistent gains and is not reported in detail.Source: §6.1 · p. 5
6.2 Evaluation Protocol
- Models output independent class probabilities; threshold-dependent metrics use 0.5, and model selection relies on validation data without test-set input.PR-AUC and ROC-AUC use continuous probabilities; macro metrics average across classes, while micro metrics aggregate counts across classes.Source: §6.2 · p. 5
6.3 Late Fusion
- Late fusion combines modality probabilities for each class using shared weights selected on validation macro-F1, with unimodal models left unchanged.The weights are chosen on a fixed grid and applied without modification to the held-out test split.Source: §6.3 · p. 5, Figure 5 · p. 5
7 Results
The results show modest gains from late fusion over the strongest window-level unimodal baseline, alongside important differences by modality and class. Text is evaluated separately at the segment level, and class imbalance complicates interpretation of Mindfulness F1.
7 Results
- For Mindfulness, the majority-class baseline F1 of 0.814 exceeded the strongest unimodal model’s F1, Video at 0.723, making F1 alone misleading under severe imbalance.The passage recommends giving PR-AUC and ROC-AUC greater weight when interpreting Mindfulness performance.Source: Table 3 · p. 5, §7 · p. 5
- Audio failed to detect Negative components at the 0.5 threshold, with F1 0.000; the passage relates this to NEG’s low support of 0.051 and audio’s weaker overall discrimination.Evaluation included 1,425 test windows with at least one valid supervision label.Source: §7 · pp. 5–6, Table 2 · p. 5
- Text achieved the highest individual-modality macro-F1, 0.49 at the segment level, with MF F1 0.58, SK F1 0.38, and NEG F1 0.50.Text was evaluated separately because its supervision unit differs from the window-level video and audio models.Source: §7 · p. 6
- Tri-modal fusion achieved macro-F1 0.350, an absolute improvement of 0.019 over the strongest window-level unimodal baseline (Video, 0.331), and improved micro-recall over every unimodal or pairwise configuration.The test-split comparison is at the window level.Source: Table 5 · p. 6, Figure 6 · p. 6, §7 · p. 6
- Validation-selected weights favored video in every pairwise configuration that included it, while tri-modal fusion used uniform weights of 0.33 per modality.The weights describe validation selection; the tri-modal test result is reported above.Source: Table 6 · p. 6, §7 · p. 6
8 Discussion
The discussion finds that class support and modality-specific signals shape performance, with no modality dominant across classes. Fusion suggests partial complementarity, while low-capacity probability-level fusion prioritizes stability and interpretability under limited data and class imbalance.
8.1 Class-Specific Behavior
- MF has the highest support ratio (0.687), but its majority-class baseline F1 of 0.814 exceeds every trained unimodal model’s F1, making PR-AUC and ROC-AUC better guides to discrimination.Text performs consistently for MF, and video captures non-verbal correlates of reflective engagement within short windows.Source: §8.1 · p. 6
- SK has lower support and variable explicit self-address across speakers, while video and audio provide indirect affective correlates with limited standalone reliability.Its majority-class baseline is zero, so model F1 on SK reflects real signal rather than majority-class prediction.Source: §8.1 · p. 6
- NEG retains the lowest support ratio (0.051); lexical ambiguity and indirect audiovisual cues make classification difficult despite a zero majority-class baseline.For example, “I didn’t handle that well” can be balanced acknowledgment or harsh self-judgment depending on tone and context.Source: §8.1 · p. 6
8.2 Modality-Specific Strengths and Constraints
- Video models short-term facial and posture dynamics in 4-second windows, while audio captures less semantically direct paralinguistic information and text offers more direct semantic access.The window scale limits modeling gradual affective trajectories; text’s segment-level predictions must be projected onto windows for fusion, and transcription errors constrain it.Source: §8.2 · pp. 6–7
- No single modality dominates across classes; performance differences reflect class-dependent expression in reflective dialog.Video’s frame-level signals, audio’s paralinguistic features, and text’s reflective and evaluative content provide different kinds of access to the constructs.Source: §8.2 · pp. 6–7
8.3 Interpretation of Fusion Behavior
- Incremental pairwise and tri-modal fusion gains indicate partial complementarity, with validation-selected weights favoring video and audio and text adding more modest correction.Uniform tri-modal weighting suggests no modality fully replaces the others; the authors prioritize stability and interpretability over higher-capacity joint architectures given limited sessions and pronounced class imbalance.Source: §8.3 · p. 7
9 Limitations and Future Work
The paper’s findings are bounded by a new, single-corpus dataset, limited effective sample size, and modeling and evaluation choices. Future work is proposed to test generalization and explore more expressive fusion and uncertainty-aware methods.
9 Limitations and Future Work
- Although sliding windows yield 155,960 labeled instances, statistical independence is determined at the session level, bounding effective sample size by 51 sessions.Isolation and Common Humanity were excluded because of sparse coverage; the structured reflection setting may also reduce isolating or negatively valenced expression.Source: §9 · p. 7
- All results are corpus-specific to one structured, interviewer-guided training environment, and generalization to spontaneous reflection or other educational settings has not been assessed.The 4-second, 1-second-hop windows impose uniform discretization on reflective processes of variable duration and may not align with construct boundaries.Source: §9 · p. 7
- Automatic speech recognition without manual correction and imperfect speaker diarization add alignment noise and bound text performance.The evaluation is descriptive, with no statistical significance testing or split-sensitivity analysis.Source: §9 · p. 7
- Probability-level late fusion uses shared static weights, does not model cross-modal feature interactions, and treats classes symmetrically in weighting.Probability calibration was also not performed because the focus was relative discriminative performance rather than deployment-ready estimates.Source: §9 · p. 7
- Future work includes cross-corpus and cross-domain validation, more expressive class-aware fusion, probability calibration, uncertainty estimation, and analysis of fine-tuned language models.The proposed language-model analysis would examine theory-grounded linguistic markers relative to pretrained, non-finetuned counterparts.Source: §9 · p. 7
10 Data and Code Availability
The paper plans to release its code, while dataset access remains restricted by participant consent and is handled case by case.
10 Data and Code Availability
- Code for preprocessing, feature extraction, model training, and fusion will be released publicly upon acceptance.Source: §10 · p. 7
- Identifiable participant video and audio cannot be publicly redistributed under the consent protocol, and further data access is handled case by case without guarantee.Source: §10 · p. 7
- The paper reports annotation, window construction, and feature extraction details to support replication on comparably collected data.Source: §10 · p. 7
11 Ethical Considerations
The study uses de-identified aligned features rather than raw recordings and does not present predictions as suitable for judgments about individuals. Participant consent and data access constraints shape how recordings can be used and shared.
11 Ethical Considerations
- Modeling uses de-identified, timestamp-aligned features rather than participants’ raw video or audio, and no identifying information is reported at the individual level.Source: §11 · p. 7
- The authors treat the work as a modeling and measurement study, not a diagnostic or evaluative tool for individual clinical or pedagogical judgments.Source: §11 · p. 7
- Recordings concern emotionally sensitive material and were collected under ethics review and informed consent, including withdrawal and recording-exclusion rights.The consent scope constrains data sharing; the teaser figure’s right panel is an AI-generated illustration, not a real individual.Source: §11 · p. 7
12 Conclusion
Probability-level fusion modestly improves on the best single modality, while detection remains uneven across classes and the majority-class baseline competes for the best-supported class. The results are constrained by the 51-session scale, with more data, richer fusion, and calibrated evaluation identified as next steps.
12 Conclusion
- Probability-level fusion adds a small but real improvement over the best single modality.Source: §12 · p. 8
- The six-component theory reduces to a grounded three-class supervision scheme, and shared preprocessing aligns video, audio, and text on one timeline.Source: §12 · p. 8
- Class detection tracks the amount of annotated support, and a majority-class baseline competes with trained models on the best-supported class.Source: §12 · p. 8
- With 51 sessions, gains are modest; the authors identify more data, richer fusion, and calibrated evaluation as next steps for testing generalization.The proposed approach should also be tested on other theory-grounded constructs and application settings.Source: §12 · p. 8