PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

Naoto Nishida, Yoshio Ishiguro

arXiv:2609.02510v1cs.CVcs.HCcs.LG

TL;DR

Body-only acted-emotion recognition across performers is underdetermined, and this paper combines diverse models while testing whether decisions rely on motion-grounded body-region evidence. In 10-fold leave-performer-out cross-validation, its ensemble reaches 36.80 ± 4.00% per-fold Macro-F1, +11.07 pp (+43% relative) over the same-split reproduced STGCN++ baseline.

  • Problem

    Body-only 12-class acted-emotion recognition across disjoint performers is underdetermined: related emotions share kinematic signatures, while performers have distinct movement idiolects.

    Source: §I · p. 1

  • Method

    The paper combines eleven models spanning four inductive-bias families and tests explanations with part-masking, counterfactual edits, and comparisons of region saliency with LMA attributes and classical kinematics.

    Source: §I · p. 1, A. Model pool: four inductive-bias… · p. 3

  • Results

    36.80 ± 4.00% per-fold Macro-F1, +11.07 pp (+43% relative) over the same-split reproduced STGCN++ baseline of 25.73 ± 4.03%; region saliency-LMA Spearman rho was +0.500 versus +0.033 for classical kinematics.

    Source: §I · p. 1, §V · p. 3

  • Takeaways & Limitations

    The authors conclude that performer-shift generalization calls for both complementary predictions and faithful explanations.

    Source: §VIII · p. 7

    The claims concern the official 64-frame short-window LPO setting, not full-sequence affect understanding.

    Source: §VII · p. 7

Abstract

from arXiv · show

We study body-only, 12-class acted-emotion classification from skeleton motion under leave-performer-out (LPO) evaluation, a hard, underdetermined setting: chance is 8.3%, and a protocol-matched reproduced STGCN++ baseline reaches only 25.73 +/- 4.03% Macro-F1. We show that reliable gains come not from a new architecture but from combining eleven models with orthogonal error modes: under 10-fold LPO cross-validation on the labeled training performers, an equal-weight logit-mean ensemble reaches 36.80 +/- 4.00% per-fold Macro-F1, a protocol-matched +11.07 pp (+43% relative) over the same-split reproduced baseline. Our central contribution is a tested explanation suite: for a strong ensemble member, part-masking and counterfactual edits show (rather than assert) that its decisions depend on motion-grounded body-region evidence, and this region saliency aligns with rule-based Laban Movement Analysis (LMA) attributes far more than with classical kinematics: region-level saliency-LMA Spearman rho = +0.500 versus +0.033, roughly 15x, and the alignment holds for the submitted 11-way ensemble itself at rho = +0.517; the audit is post hoc and needs no retraining. The same suite faithfully reports a negative: within-window temporal saliency is diffuse rather than localized.

I. INTRODUCTION

For underdetermined 12-class body-motion emotion recognition across disjoint performers, the paper finds that combining diverse models transfers better than intuitive single-model upgrades. It also tests model explanations, finding body-region evidence aligned with LMA attributes while reporting diffuse temporal saliency as a negative result.

I. INTRODUCTION

  • The DIEM-A task asks for 12-class acted-emotion recognition from body motion alone across disjoint performers; chance is 8.3%, and a strong STGCN++ baseline reaches only about 25% Macro-F1.Semantically related emotions share kinematic signatures, while performer-specific movement idiolect makes cross-performer within-class variation substantial.Source: §I · p. 1
  • Supervised-contrastive heads, VLM distillation, post-hoc calibration, and other single-model upgrades failed or regressed under the protocol, motivating a conservative system.The paper attributes transferable gains to diversity across graph, attention, hybrid/MLP, and externally pretrained models, whose different errors can cancel when logits are averaged.Source: §I · p. 1
  • Part-masking and counterfactual edits show that a strong ensemble member depends on motion-grounded body-region evidence, whose saliency correlates more with rule-based LMA attributes than with classical kinematics.The reported saliency–LMA Spearman ρ is +0.500 versus +0.033; the audit is post hoc and needs no retraining, and the same alignment holds for the submitted 11-way ensemble.Source: §I · p. 1
  • The explanation suite reports six explicit verdicts—five positive and one negative—including diffuse within-window temporal saliency and a deterministic, audited motion-to-rationale narrator.The contribution is the method and audit, not a dataset release; release is held pending consent and license review.Source: §I · pp. 1–2
  • 36.80 ± 4.00% per-fold Macro-F1 versus 25.73 ± 4.03% for reproduced STGCN++, a +11.07 pp (+43%) gain for the 11-way equal-weight logit-mean ensemble.The comparison uses 10-fold leave-performer-out evaluation on the 74-performer training split; the gain tracks measured error-space orthogonality.Source: §I · pp. 1–2, Figure 1 · p. 2

II. RELATED WORK · III. TASK, DATA AND EVALUATION

These sections position the work as a validated audit of motion explanations and define a 12-class acted body-motion task evaluated under performer-disjoint leave-performer-out (LPO).

II. RELATED WORK

  • Graph-convolutional and attention models – set the single-network starting point, while body-emotion datasets extend action recognition to acted affect –.These models are typically evaluated within-performer, unlike the harder LPO regime used here.Source: §II · p. 2
  • Masked-motion, contrastive, and heterogeneous-corpus pretraining – motivate frozen external pretraining as an ensemble family, although such transfer does not help in the reported comparison.The transfer result is reported in Table I.Source: §II · p. 2
  • Prior motion explanations use perturbation, saliency, counterfactuals, and semantic attributes –,, but few test whether evidence drives predictions under performer shift.The work contributes a validated audit protocol that screens probes for faithfulness and stability and reports negative findings as well.Source: §II · p. 2

III. TASK, DATA AND EVALUATION

  • DIEM-A contains 12 acted-emotion classes; models use 25-node skeleton tensors and fixed 64-frame windows, with training and test performers disjoint.The training split has 7,992 clips from 74 performers and the test split has 1,944 clips from 18 performers. Sequences are recorded at 120 Hz, with median length ≈845 frames; the 64-frame window is ≈0.53 s and preserves parity with the official baseline. Five earliest Japanese performers used a 41-marker OptiTrack rig, while remaining recordings use 57-marker Vicon; none of the five is in training, and capture-era effects are checked by excluding JP 06–JP 12.Source: §III · p. 2
  • Evaluation uses 10-fold LPO cross-validation on the 74-performer training split, with Macro-F1 as primary and Accuracy as secondary metrics.Scores use equal logit-mean aggregation and are reported as per-fold mean ± SD; pooled out-of-fold values are descriptive secondary results.Source: §III · p. 2
  • +11.07 pp is the headline gain over the 25.73 ± 4.03% protocol-matched reproduced STGCN++ baseline, using the same split and logit-mean/per-fold convention.The official 25.21 ± 4.49% STGCN++ result uses a different, 92-performer full-LPO protocol; the held-out test score is pending the challenge leaderboard.Source: §III · pp. 2–3

IV. METHOD · A. Model pool: four inductive-bias families · B. Fusion: equal-weight logit-mean

The method combines eleven models from four inductive-bias families using an equal-weight mean of raw logits. Logit-mean fusion is the canonical choice because it limits the influence of confidently wrong members and outperforms probability averaging in Macro-F1.

A. Model pool: four inductive-bias families

  • The eleven-member pool spans graph-convolutional, attention, hybrid/MLP, and frozen-pretraining families to reduce overlap in model errors.Graph-convolutional members include STGCN++ [4], CTR-GCN, ProtoGCN, and Region-Aware ConvTr; SkateFormer is the attention member.Source: A. Model pool: four inductive-bias… · p. 3
  • The pool also includes Conv1D+Transformer and Keypoint-Pool-MLP hybrids, plus MotionBERT-Lite, C3D-marker-stats, and MAMP NTU60-xsub and NTU120-xset.Region-Aware ConvTr and the hybrid/MLP family are part-aware variants of the Conv1D+Transformer template.Source: A. Model pool: four inductive-bias… · p. 3
  • The four families encode complementary priors: kinematic-chain locality, long-range position-insensitive co-occurrence, pooled spatiotemporal statistics, and transferred generic-motion representations.Source: A. Model pool: four inductive-bias… · p. 3
  • Region-Aware is both a pool member and the explained model in §VI, and is the strongest single model at 30.05 ± 3.48%.Source: A. Model pool: four inductive-bias… · p. 3

B. Fusion: equal-weight logit-mean

  • For M members with logits z_m ∈ R^12, prediction is the argmax of their equal-weight mean of raw logits.Source: B. Fusion: equal-weight logit-mean · p. 3
  • +1.15 pp Macro-F1: logit-mean fusion exceeds probability averaging, while raw-logit averaging before normalization limits confidently wrong members’ influence.The method fixes logit-mean as its canonical fusion convention.Source: B. Fusion: equal-weight logit-mean · p. 3

C. Pretraining and protocol discipline · D. Negatives as design constraints · V. PERFORMANCE RESULTS

Under strict 10-fold LPO evaluation, the 11-way logit-mean ensemble improves on the reproduced STGCN++ baseline. The sections describe the protocol, failed approaches, and evidence that diverse error patterns support the gain.

C. Pretraining and protocol discipline

  • Training and selection use fixed-seed 10-fold LPO, fold-wise OOF predictions, training-split-only model selection, and no test-label access or fitted ensemble weights.Two MAMP backbones pretrained on NTU60-xsub and NTU120-xset are transferred frozen alongside frozen MotionBERT-Lite and C3D-marker-statistics encoders.Source: C. Pretraining and protocol discipline · p. 3
  • The eleven-member system has 12.98 M trainable parameters and uses eleven forward passes plus a parameter-free logit mean; explanations run post hoc with no retraining.Seven members train in-domain, while the four external branches remain frozen with linear heads; deterministic regeneration uses stored per-member OOF arrays.Source: C. Pretraining and protocol discipline · p. 3

D. Negatives as design constraints

  • The search reports failed or regressing supervised-contrastive, mixture-of-experts, scenario-text alignment, VLM, multi-crop inference, and post-hoc calibration approaches rather than omitting them.C3D late fusion regressed −0.59 pp and was associated with a 69.8% country-leakage signal; the table frames lessons non-causally.Source: D. Negatives as design constraints · p. 3, Table I · p. 4

V. PERFORMANCE RESULTS

  • 36.80 ± 4.00% per-fold Macro-F1, +11.07 pp (+43% relative) over the same-split reproduced STGCN++ baseline at 25.73 ± 4.03%.The paired-fold improvement excludes zero in 10/10 folds, with a bootstrap 95% CI of [9.86, 12.33] pp; performer-level resampling gives [9.14, 12.17] pp.Source: §V · p. 3, Figure 3 · p. 4
  • The lift path is 25.7 → 30.1 → 33.9 → 36.8%; logit-mean beats probability-averaging by +1.15 pp, while z-scoring and rank-only Borda retain +0.99 pp and +0.69 pp.Ensembling beats the best single model, 30.05 ± 3.48%, by +6.8 pp; all 12 classes improve over baseline, with sadness improving by +6.1 pp.Source: §V · pp. 3–4, Figure 3 · p. 4
  • Pairwise per-sample error correlations span [0.15, 0.43], and all eleven leave-one-out deltas are negative, indicating contributions without a single dominant or redundant member.Removing the frozen external-pretraining block costs −2.91 pp; individual leave-one-out deltas range from −0.84 pp for MAMP-NTU60 to −0.09 pp for CTR-GCN.Source: §V · pp. 4–5
  • Residual errors include jealousy → contempt and guilt → sadness/shame, while the JP–TW macro gap is a dataset stratum rather than a cultural finding.The gap is entangled with capture era and performer idiolect; excluding the seven earliest-captured Japanese training performers shifts pooled-OOF Macro-F1 by only −0.44 pp (36.50 vs 36.94%).Source: Figure 2 · p. 4, §V · p. 5

VI. EXPLAINABILITY RESULTS · A. Faithfulness: masking the cited regions lowers accuracy

The explainability suite tests whether model attributions are behaviorally and semantically meaningful, rather than merely visualizing attention. Its faithfulness test finds that masking important regions degrades performance, while the suite also reports a negative result: within-window temporal saliency is diffuse.

VI. EXPLAINABILITY RESULTS

  • The post-hoc suite tests the Region-Aware ensemble member and reruns key tests on the submitted ensemble, adding no accuracy cost.It covers faithfulness, stability, semantic grounding, behavioral corroboration, grounded narration, and within-window temporal saliency.Source: §VI · p. 5
  • Attribution rankings remain stable under input noise.Source: §VI · p. 5
  • Region saliency aligns with the Laban vocabulary for both the Region-Aware member and the submitted ensemble.Source: §VI · p. 5
  • Motion edits shift predictions in the same part ranking, behaviorally corroborating the attributions.Source: §VI · p. 5
  • The 50 audited narration cards contain no unsupported claims.Source: §VI · p. 5
  • Within-window temporal saliency is diffuse, a reported negative finding.Source: §VI · p. 5

A. Faithfulness: masking the cited regions lowers accuracy

  • +0.124 ± 0.031 important−reverse AUC gap over 6 body parts (+0.199 at 25-joint resolution), positive in every fold.Masking parts ranked important degrades Macro-F1 more than masking the same number of unimportant parts.Source: A. Faithfulness: masking the cited… · p. 5

B. Stability: the ranking survives input noise · C. What the model reads: body motion in Laban terms

Input noise preserves the attribution ranking, while region-level saliency aligns substantially more with rule-based Laban Movement Analysis than with classical kinematics. The Laban alignment is semantic rather than causal evidence by itself.

B. Stability: the ranking survives input noise

  • Under input noise, the attribution ranking is preserved at Spearman ρ = +0.983 ± 0.039 for σ = 0.02, and the top part never flips.This supports interpreting the ranking as a property rather than an input-noise artifact.Source: B. Stability: the ranking survives… · p. 5

C. What the model reads: body motion in Laban terms

  • Region-level saliency correlates with Laban Movement Analysis at ρ = +0.500 versus +0.033 for classical kinematics, roughly 15× stronger.The correlation uses the 12 emotion × 4 region pairs; the comparison is computed under the identical pipeline.Source: C. What the model reads: body motion… · pp. 5–6
  • The LMA alignment’s 95% CI is [+0.150, +0.733] and its structure-respecting permutation test gives p < 0.001.The interval uses emotion-block bootstrap resampling over 12 emotion clusters; the permutation test shuffles the four region scores within each emotion.Source: C. What the model reads: body motion… · p. 5
  • Per-emotion LMA alignment is positive for 10 of 12 emotions, with median ρ = +0.70 and an exact two-sided sign-test p = 0.039.Emotions are the unit of this cluster-level check.Source: C. What the model reads: body motion… · p. 5
  • The LMA signature averages standardized values for 32 attributes across four axes, then groups them into head, arms, legs, and torso region scores.Each attribute is standardized against the full training pool; region importance is formed by averaging absolute standardized values within each region.Source: C. What the model reads: body motion… · p. 5
  • The rule-based LMA export requires no model retraining, so its Macro-F1 cost is 0 pp by construction.The passage identifies this as the bold row of Table III.Source: C. What the model reads: body motion… · p. 5
  • For sadness, salient regions track a bowed, sunken posture associated with head_bow+, head_height−, and trunk_lean+.The authors treat the LMA correlation as semantic alignment, not causal evidence by itself; dependence is tested separately through part-masking and counterfactual perturbations.Source: C. What the model reads: body motion… · pp. 5–6

D. The finding holds for the submitted ensemble · E. Counterfactual edits move predictions as predicted · F. What the explanations correctly refuse to claim

The submitted 11-way ensemble retains positive body-evidence faithfulness and LMA alignment, while counterfactual edits provide observational corroboration. Temporal saliency remains diffuse, so the explanations do not claim frame localization or that temporal order carries no signal.

D. The finding holds for the submitted ensemble

  • The submitted 11-way ensemble has a positive important−reverse faithfulness gap in 10/10 folds (+8.5 pp partial-AUC) and region-level LMA alignment of ρ = +0.517.The alignment exceeds the single-member +0.500; the ensemble gap is smaller than the member’s +12.4 pp because six of seven skeleton members lack per-part gates.Source: D. The finding holds for the submitted… · p. 6
  • At the cluster level, per-emotion LMA alignment is positive for 11 of 12 emotions and negative for none (exact sign test, two-sided p = 0.001).Region-level alignment has an emotion-block bootstrap 95% CI of [+0.333, +0.700] and permutation p = 0.001.Source: D. The finding holds for the submitted… · p. 6

E. Counterfactual edits move predictions as predicted

  • Pose-preserving, motion-perturbing edits shift predictions: head-freeze changes true-class probability by ∆ptrue = −0.0164 on average and flips 41% of predictions; arm-amplify flips 23.1%.These averages cover 864 validation clips × 10 folds and are observational evidence, not causal identification.Source: E. Counterfactual edits move… · p. 7
  • Per-part saliency-vs-disruption agreement is modest (ρ = +0.49, n = 6, p = 0.33), so the edits offer suggestive corroboration rather than the load-bearing cross-method result.The passage identifies that result as the ρ = +0.500 vs +0.033 contrast with its real n.Source: E. Counterfactual edits move… · p. 7

F. What the explanations correctly refuse to claim

  • Temporal saliency is near-uniform across all 12 emotions (per-sample entropy ≈98.7% of log T; class-mean 99.5%), with an important−reverse AUC gap of only ≈+0.002.The two entropy values are aggregations of the same near-uniform distribution; within-window frame importance is diffuse, with no single frame dominating.Source: F. What the explanations correctly… · p. 7
  • Diffuse frame importance does not show that temporal order carries no signal; the paper reports this frame-localization negative as a test of the faithfulness suite.The passage also links ensemble gains to inter-performer F1 variation of ∼4×, decomposed into roughly half intrinsic difficulty and half architecture-specific error, the disagreement structure exploited by the ensemble.Source: F. What the explanations correctly… · p. 7

G. Scoped qualitative cards (decoupled from the audit) … ETHICAL IMPACT STATEMENT

The paper presents ensemble gains and audited motion-grounded explanations as research findings bounded by the short-window, body-only acted-emotion protocol. It also limits qualitative narration, generalization, and artifact release to address uncertainty, bias, and re-identification risk.

G. Scoped qualitative cards (decoupled from the audit)

  • Qualitative narration serves only as a scoped interface to audited saliency and LMA channels; no quantitative claim relies on it.A label-field audit of the narration cards is in the supplementary material.Source: G. Scoped qualitative cards (decoupled… · p. 7

VIII. CONCLUSION

  • 36.80 ± 4.00% Macro-F1 for the ensemble versus 25.73 ± 4.03% for the same-split reproduced baseline yields a protocol-matched +11.07 pp lift.The ensemble result is 10-fold cross-validation on 74 performers; the held-out test score is pending.Source: §VII · p. 7, §VIII · p. 7

VII. DISCUSSION AND LIMITATIONS

  • The roughly 37% Macro-F1 result concerns 12-way body-only LPO acted-emotion recognition, not saturation; every class improves, with the weakest gaining +6.1 pp.Chance is 8.3%; the official baseline is 25%, and the contribution is a protocol-matched +43% relative gain.Source: §VII · p. 7
  • Per-fold SD is ≈4 pp, while bootstrap 95% CIs show the ensemble interval clearing the 7-way interval; maximum confidence is ∼62%.Models use a fixed 64-frame window for parity with the official baseline, and longer or multiwindow inference regressed; claims concern this short-window setting, not full-sequence affect understanding.Source: §VII · p. 7
  • ρ = +0.500 versus +0.033 for saliency–LMA versus saliency–kinematics over 48 emotion–region pairs, and the ensemble reproduces the LMA alignment at ρ = +0.517.The first contrast has p < 0.001 under emotion-block resampling; ensemble alignment has p = 0.001. The six-case saliency–counterfactual agreement is n = 6, p = 0.33; JP/TW differences are confounded, and excluding seven earliest-captured Japanese training performers changes pooled OOF Macro-F1 by −0.44 pp (36.50 vs 36.94%).Source: §VII · p. 7

VIII. CONCLUSION

  • Part-masking, stability, and counterfactual tests show that a strong member and the submitted ensemble rely on motion-grounded body-region evidence aligned with rule-based Laban attributes.The suite also reports diffuse within-window temporal saliency and a 0/50-audited narrator.Source: §VIII · p. 7

ETHICAL IMPACT STATEMENT

  • The work uses previously collected, consented data and releases research methods and audit artifacts rather than a deployable detector or the rationale dataset.Country classification reaches roughly 69.8% from the same inputs, exposing re-identification and over-fitting risks; rationale data remain withheld pending consent/license review because skeleton renders retain residual re-identifiability.Source: ETHICAL IMPACT STATEMENT · p. 8
  • The findings are bounded by an East-Asian acted-affect distribution and skeleton-only inputs, and do not establish intercultural or acted-to-spontaneous generalization.The LPO split includes 40 Japanese and 34 Taiwanese performers; the 64-frame window covers about 7.5% of the median sequence, and 36.80% Macro-F1 remains far below a plausible human-usable threshold.Source: ETHICAL IMPACT STATEMENT · p. 8

SUPPLEMENTARY MATERIAL … S9. MEMBER ERROR CORRELATION

The supplementary analyses show that the ensemble members contribute nonredundant errors and that region-level saliency aligns with LMA attributes, while temporal saliency remains near-uniform. Additional audits report performer-stratum results, counterfactual perturbations, and explanation-card grounding.

S9. MEMBER ERROR CORRELATION

  • 36.94% pooled-OOF Macro-F1 (per-fold headline: 36.80 ± 4.00%) is the 11-member ensemble base, and removing any one member lowers pooled-OOF Macro-F1.The leave-one-out ablation uses n = 7,992 clips on the 74-performer split; the last row removes the whole frozen external block.Source: Table S1 · p. 9
  • The JP–TW macro gap is 35.3 vs 38.9%, and excluding the seven earliest-captured Japanese training performers shifts pooled-OOF Macro-F1 by only −0.44 pp.The five OptiTrack-captured performers are absent from the training split; Fig. S2 also shows the row-normalised confusion matrix and per-class F1 by performer country.Source: Figure S2 · p. 10
  • The rule-based LMA schema defines 32 clip-level features across four Laban axes, computed from BVH-24 forward kinematics.The per-emotion z-score signature and region aggregation are defined in the main paper.Source: Table S3 · p. 11
  • Temporal saliency is near-uniform across emotions (per-sample entropy ≈98.7% of log(64); class-mean 99.5%), while Head is the top-1 joint for all 12 emotions.For surprise, a joint-by-frame burst retains local structure despite class-averaged temporal uniformity; the supplementary negative-results catalogue extends Table 2.Source: Figure S4 · p. 13, Table S4 · p. 15, S9. MEMBER ERROR CORRELATION · p. 15
  • Region saliency–LMA Spearman ρ is +0.500, versus +0.033 for classical kinematics.The figure also presents mean LMA z-scores per emotion across 32 attributes spanning Body, Effort, Shape, and Space.Source: Figure S5 · p. 14
  • Counterfactual edits show head-freeze is most disruptive (mean Δptrue −0.0164, 41% flip), while per-part saliency versus disruption has Spearman ρ = +0.49 (n = 6, p = 0.33).These are observational perturbations, not causal tests; left-arm amplification produced a 23.1% flip rate.Source: Figure S5 · p. 14
  • All 50 explanation cards pass the grounding audit (0 hallucinated claims; grounded-ratio = 1.0), while per-card correctness is excluded pending a card-generator fix.The figure shows three qualitative examples—correct, confident-error, and semantic-confusion—from 16 label-consistent cards.Source: Figure S6 · p. 16
  • Off-diagonal member error correlations range from 0.15 to 0.43, with the lowest correlations for C3D-stats and the highest for Conv1D+Tr ↔ KP-MLP.Pairwise Pearson (= ϕ) correlations use OOF n = 7,992; C3D-stats correlations to skeleton models are approximately 0.15–0.19.Source: Figure S7 · p. 16

Related papers

Loading 2609.02510v1…