PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

Naoto Nishida, Yoshio Ishiguro

arXiv:2609.02510v1cs.CVcs.HCcs.LG

TL;DR

The paper studies underdetermined 12-class acted-emotion recognition from body-only skeleton motion under leave-performer-out evaluation. It combines eleven models with orthogonal error modes and tests whether their explanations are motion-grounded. The ensemble improves protocol-matched Macro-F1, while the audit finds stronger LMA than kinematic alignment and diffuse within-window temporal saliency.

  • Problem

    Body-only 12-class acted-emotion recognition under performer-disjoint evaluation is underdetermined, with chance at 8.3% and a reproduced STGCN++ baseline at 25.73 ± 4.03% Macro-F1.

  • Method

    The paper combines eleven models spanning four inductive-bias families with equal-weight logit-mean fusion and applies part-masking, stability, counterfactual, and semantic LMA attribution tests.

  • Results

    36.80 ± 4.00% per-fold Macro-F1 yields a protocol-matched +11.07 pp (+43% relative) over the reproduced STGCN++ baseline, while region saliency correlates with LMA at ρ = +0.500 versus +0.033 for classical kinematics.

  • Takeaways & Limitations

    Reliable gains come from combining orthogonal error modes, and tested explanations support motion-grounded body-region evidence for both a strong member and the submitted ensemble.

  • Takeaways & Limitations

    Claims concern the fixed 64-frame, 74-performer training-split LPO setting; the held-out test score is pending, and longer or multiwindow inference regressed.

Abstract

from arXiv · show

We study body-only, 12-class acted-emotion classification from skeleton motion under leave-performer-out (LPO) evaluation, a hard, underdetermined setting: chance is 8.3%, and a protocol-matched reproduced STGCN++ baseline reaches only 25.73 +/- 4.03% Macro-F1. We show that reliable gains come not from a new architecture but from combining eleven models with orthogonal error modes: under 10-fold LPO cross-validation on the labeled training performers, an equal-weight logit-mean ensemble reaches 36.80 +/- 4.00% per-fold Macro-F1, a protocol-matched +11.07 pp (+43% relative) over the same-split reproduced baseline. Our central contribution is a tested explanation suite: for a strong ensemble member, part-masking and counterfactual edits show (rather than assert) that its decisions depend on motion-grounded body-region evidence, and this region saliency aligns with rule-based Laban Movement Analysis (LMA) attributes far more than with classical kinematics: region-level saliency-LMA Spearman rho = +0.500 versus +0.033, roughly 15x, and the alignment holds for the submitted 11-way ensemble itself at rho = +0.517; the audit is post hoc and needs no retraining. The same suite faithfully reports a negative: within-window temporal saliency is diffuse rather than localized.

I. INTRODUCTION

DIEM-A poses underdetermined 12-class body-only emotion recognition under leave-performer-out evaluation. The paper finds that combining diverse models improves protocol-matched performance and supports motion-grounded explanations through tested attribution audits.

  • Chance is 8.3%, while reproduced STGCN++ reaches only 25.73 ± 4.03% Macro-F1 under leave-performer-out evaluation.The task excludes face, audio, and scene information and evaluates across disjoint performers.
  • 36.80 ± 4.00% per-fold Macro-F1 is reached by averaging eleven models spanning four inductive-bias families.The protocol is 10-fold leave-performer-out on the 74-performer training split.
  • ρ = +0.500 versus +0.033 shows region saliency aligns more with rule-based LMA attributes than with classical kinematics.Part-masking and counterfactual edits test whether decisions depend on motion-grounded body-region evidence, without retraining.
  • +11.07 pp / +43% is achieved by an orthogonal-error 11-way logit-mean ensemble over the same-split reproduced baseline.The gain is attributed to measured error-space orthogonality rather than a new architecture.
  • The explanation suite reports five positive verdicts and one negative verdict rather than hiding unfavorable findings.The paper also contributes a deterministic motion-to-rationale narrator, while dataset release remains pending consent and license review.

II. RELATED WORK

Prior work provides strong skeleton-recognition architectures and attribution techniques, but this paper targets the harder leave-performer-out body-emotion setting and validates whether explanations drive predictions.

  • Skeleton action-recognition work is dominated by graph-convolutional and attention architectures, which this paper builds on for body emotion.Those models are typically evaluated within-performer, whereas this study reports leave-performer-out results.
  • Few attribution studies test whether cited evidence actually drives predictions, especially under performer shift.The paper addresses this gap with a validated audit protocol rather than a new attribution method.
  • DIEM-A uses 12 acted emotions, performer-disjoint splits, and fixed 64-frame windows from 24-joint skeleton sequences.The training split contains 7,992 clips from 74 performers, while test labels are withheld.
  • The official STGCN++ baseline is 25.21 ± 4.49% Macro-F1 on 92 performers, while the reproduced same-split baseline is 25.73 ± 4.03%.Improvements are reported against the protocol-matched reproduction, not the full official result.

IV. METHOD

The method combines models with deliberately different inductive biases using equal-weight raw-logit averaging. Logit-mean fusion is selected as the canonical aggregation because it outperforms probability averaging without added cost.

  • A. Model pool: four inductive-bias families: Eleven models span graph-convolutional, attention, hybrid/MLP, and frozen external-pretraining families.These families encode complementary priors including kinematic-chain locality, long-range co-occurrence, pooled statistics, and transferred representations.
  • B. Fusion: equal-weight logit-mean: The prediction is the argmax of the equal-weight mean of per-member raw logits.Each member supplies a 12-class logit vector before fusion.
  • B. Fusion: equal-weight logit-mean: +1.15 pp Macro-F1 favors logit-mean fusion over probability averaging at no cost.Raw-logit averaging is fixed as the canonical convention.

C. Pretraining and protocol discipline

The system prioritizes protocol discipline and heterogeneous ensemble diversity over unsupported single-model upgrades. Under canonical 10-fold LPO evaluation, the 11-way logit-mean ensemble improves substantially, with gains distributed across complementary members and external branches.

  • Pretraining and diversity: The eleven-member pool spans graph-convolutional, attention, hybrid/MLP, and frozen external-pretraining families with deliberately different inductive biases.The external branches add fewer than 0.01 M trainable parameters per branch, while total trainable parameters are 12.98 M.
  • Protocol discipline: Several tested approaches failed or regressed, including supervised contrastive learning, mixture-of-experts, scenario-text alignment, VLM methods, multi-crop inference, and calibration.C3D late fusion regressed by −0.59 pp and was associated with a 69.8% country-leakage signal.
  • Protocol and main result: 36.80 ± 4.00% per-fold Macro-F1, a protocol-matched +11.07 pp over the reproduced STGCN++ baseline, is achieved by the submitted 11-way ensemble.The baseline is 25.73 ± 4.03%, and the comparison uses the 74-performer split and per-fold mean ± SD convention.
  • Protocol and main result: The lift path progresses from 25.7 to 30.1 to 33.9 to 36.8% across baseline, best single, 7-way, and final 11-way logit-mean stages.Fig. 3 reports 10-fold LPO mean ± SD for these stages.
  • Pretraining and diversity: Pairwise per-sample error correlations remain within [0.15, 0.43], supporting logit averaging as an error-cancellation strategy across diverse model families.The lowest correlations involve the frozen external-pretraining column, while homogeneous additions provide smaller or negative returns.
  • Pretraining and diversity: The four frozen external-pretraining branches jointly contribute −2.91 pp when removed, supplying essentially the whole 7 →11-way lift.Their contributions are distributed across branches rather than carried by one member.

VI. EXPLAINABILITY RESULTS

The explanation suite tests whether attributions correspond to behaviorally meaningful body-region evidence. Part-masking confirms that the model’s important regions matter more than equally sized reverse-ranked regions, with positive effects at both resolutions.

  • Faithfulness: +0.124 ± 0.031 important−reverse AUC gap over 6 body parts confirms attribution faithfulness in every fold.At 25-joint resolution, the gap increases to +0.199.
  • Faithfulness: Masking attribution-ranked important parts degrades Macro-F1 more than masking the same number of unimportant parts, and the effect is positive in every fold.This behavioral test establishes that the attributions are not merely decorative.
  • Faithfulness: +0.199 important−reverse AUC gap at 25-joint resolution shows stronger separation when masking individual joints.The comparison is between masking attribution-ranked important parts and the same number of reverse-ranked parts.

B. Stability: the ranking survives input noise

Input-noise testing shows that the model’s body-region attribution ranking is highly stable, supporting the interpretation that its saliency is not merely a noise artifact.

  • B. Stability: the ranking survives input noise: Spearman ρ = +0.983 ± 0.039 at σ = 0.02, with the top part never flipping under input noise.This stability converts the masking result from a possible artifact into a property.
  • B. Stability: the ranking survives input noise: The LMA alignment is computed over 12 emotion × 4 region pairs, using emotion-block resampling rather than treating all pairs as independent.The 95% CI is [+0.150, +0.733], with permutation p < 0.001 under structure-preserving shuffling.
  • B. Stability: the ranking survives input noise: Region-level saliency aligns with LMA attributes more strongly than classical kinematics: Spearman ρ = +0.500 versus +0.033.The comparison is semantic alignment, while dependence is tested separately through masking and counterfactual perturbations.
  • B. Stability: the ranking survives input noise: Table III collects the explanation-suite values with confidence intervals or null results and plain-language readings.The suite’s metrics are recomputed on the same 10 LPO folds and out-of-fold predictions unless otherwise stated.

D. The finding holds for the submitted ensemble

The explanation finding extends from a strong ensemble member to the submitted 11-way ensemble itself, although the ensemble’s faithfulness gap is smaller.

  • D. The finding holds for the submitted ensemble: +8.5 pp partial-AUC at single-part resolution remains positive in 10/10 folds for the submitted ensemble.The ensemble test uses permutation-importance shuffling because zero-masking collapses members lacking per-part gates.
  • D. The finding holds for the submitted ensemble: Spearman ρ = +0.517 links submitted-ensemble region importance to rule-based LMA attributes.The emotion-block bootstrap 95% CI is [+0.333, +0.700], with permutation p = 0.001.
  • D. The finding holds for the submitted ensemble: The submitted ensemble’s per-emotion alignment is positive for 11 of 12 emotions and negative for none.The exact sign test is two-sided p = 0.001.
  • D. The finding holds for the submitted ensemble: The ensemble gap is smaller than the member’s +12.4 pp because six of seven skeleton members lack per-part gates.The consistently positive above-null result therefore describes the submitted system, not only one component.
  • D. The finding holds for the submitted ensemble: Pose-preserving motion edits provide convergent observational evidence: head-freeze shifts true-class probability by Δptrue = −0.0164 and flips 41% of predictions.Arm-amplify edits flip 23.1%, while saliency–disruption agreement is modest at ρ = +0.49 with n = 6 and p = 0.33.

F. What the explanations correctly refuse to claim

The explanation suite reports where evidence is spatially grounded while explicitly declining to claim localized within-window temporal evidence or deployable affect recognition.

  • F. What the explanations correctly refuse to claim: Within-window temporal saliency is diffuse: per-sample entropy is ≈98.7% of log T and the important−reverse AUC gap is only ≈+0.002.This negative does not claim that temporal order carries no signal.
  • F. What the explanations correctly refuse to claim: The narration is only a scoped interface to audited saliency and LMA channels, with no quantitative claim relying on it.A label-field audit of narration cards appears in the supplementary material.
  • F. What the explanations correctly refuse to claim: The system is a research probe under a fixed protocol, not a deployable affect recognizer.Its Macro-F1 is approximately 37% on 12-way body-only acted-emotion recognition, where chance is 8.3%.
  • F. What the explanations correctly refuse to claim: The paper’s conclusion ties performer-shift generalisation to both complementary predictions and faithful explanations of motion evidence.The reported gain is cross-validated on labeled training performers rather than established by a hidden-test leaderboard result.

ETHICAL IMPACT STATEMENT

The study uses released DIEM-A data under existing ethics approval and performer consent, without collecting or reidentifying additional personal data. Its findings are bounded by an East-Asian acted-affect distribution, and intercultural or spontaneous-affect generalization is untested.

  • DIEM-A was collected under the dataset holder’s ethics approval and explicit performer consent.
  • The work uses only challenge-released data and collected no new recordings, identifiers, or auxiliary biometric attributes.
  • The LPO split contains 40 Japanese and 34 Taiwanese performers, bounding findings to an East-Asian acted-affect distribution.
  • Intercultural generalization beyond these populations is not tested, and acted emotion may not transfer directly to spontaneous in-the-wild affect inference.

S9. MEMBER ERROR CORRELATION

The eleven-member ensemble benefits from complementary error patterns, while ablations show that removing any member reduces pooled-OOF Macro-F1. Supplementary analyses also characterize saliency, LMA alignment, temporal diffusion, and explanation-card grounding.

  • Member ablation: 36.94% pooled-OOF Macro-F1 is the 11-member ensemble base, with all leave-one-out member-removal deltas negative.
  • Stratification: The JP–TW Macro-F1 gap is 35.3 versus 38.9%, but it is a dataset stratum rather than an interpreted cultural effect.
  • Explanation audit: Region saliency aligns with LMA at ρ = +0.500 versus +0.033 for classical kinematics, while temporal saliency is near-uniform.
  • Counterfactual edits: Head-freeze is the most disruptive counterfactual edit, producing mean ∆ptrue = −0.0164 and a 41% flip rate.
  • Narration audit: All 50 explanation cards pass the grounding audit with zero hallucinated claims and grounded-ratio = 1.0, although per-card correctness is excluded.
  • Error correlation: Off-diagonal error correlations range from 0.15 to 0.43, with the most-redundant Conv1D+Tr ↔KP-MLP pair at ρ = 0.43.
Loading 2609.02510v1…