Source-linked AI summary
GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition
Ting Zhu, Huizhi Liang, Boguslaw Obara, Shufei Duan, Varun Ojha
TL;DR
MER representations do not explicitly organize hierarchical emotional structures alongside cross-modal semantics. GeoMER combines Euclidean multimodal fusion with hyperbolic emotion-core routing. On IEMOCAP, it achieves 83.54% WA and 83.35% UA, improving over the MemoCMT baseline.
Problem
Euclidean MER representations capture cross-modal interactions but do not explicitly provide a geometric structure for organizing hierarchical emotional relationships.
Source: §1 · p. 2Method
GeoMER combines Euclidean multimodal fusion for cross-modal semantics with hyperbolic geometric core routing for structured emotion representations.
Source: §1 · pp. 2–3Results
GeoMER achieves 83.54% WA and 83.35% UA, improving over the MemoCMT baseline by 2.49% WA and 2.02% UA.
Source: §4.2 · p. 10Takeaways & Limitations
The results support a hybrid design in which hyperbolic structure complements rather than replaces Euclidean representation learning.
Source: §4.1 · p. 10
Abstract
from arXiv · showhide
Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating information from multiple modalities such as speech and text. Psychological studies suggest that emotions exhibit latent hierarchical structured relationships in their semantic representations. However, most MER approaches rely on Euclidean representations, which capture fine-grained cross-modal interactions but do not explicitly provide a geometric prior for organizing hierarchical emotional structures. To mitigate this gap, we propose GeoMER, a joint Euclidean-Hyperbolic modeling framework that introduces hyperbolic emotion-core routing to capture higher-level, abstract emotional representations for MER. The proposed framework jointly models token-level cross-modal semantic interactions and latent emotion representations in two complementary geometric spaces. Specifically, GeoMER consists of a Euclidean Multimodal Fusion Module (EuclFusion) for multimodal representation fusion and a Hyperbolic Geometric Core Routing (HypGeoRoute) module that performs geometry-aware core assignment and aggregation in a hyperbolic manifold to capture more abstract feature representations. The HypGeoRoute module complements the Euclidean token-level modeling. We experiment on the IEMOCAP multimodal MER benchmark and demonstrate that the proposed method achieves the best performance, with 83.54% weighted accuracy and 83.35% unweighted accuracy, outperforming Euclidean baselines, including MemoCMT.
1 Introduction
MER needs to represent both cross-modal semantics and hierarchical emotional relationships, but Euclidean representations do not explicitly organize those structures. GeoMER combines Euclidean fusion with hyperbolic emotion-core routing to model both kinds of representations.
1 Introduction
- Euclidean MER representations model cross-modal interactions but do not explicitly organize hierarchical emotional relationships described by psychological theories.Those theories describe emotion mixtures and intensity variations, motivating non-Euclidean representations for structured emotion.Source: §1 · p. 2
- Earlier hyperbolic affective methods focused on sample label hierarchies or general Euclidean–hyperbolic layer composition, whereas GeoMER is task-specific.Fine-grained emotion categories exhibit multilevel and intensity-dependent relationships [20], and HypEmo models category hierarchies with label-aware embeddings.Source: §1 · p. 2
- GeoMER jointly models discriminative cross-modal semantics in Euclidean space and abstract hierarchical emotion structure in hyperbolic space.The framework is presented as more than replacing Euclidean embeddings or reusing existing components for MER.Source: §1 · p. 2
- GeoMER jointly optimizes a cross-attention-transformer Euclidean branch and a hyperbolic branch, integrating them through unified fusion.This design aims to retain Euclidean discriminative representations alongside more abstract hyperbolic features.Source: §1 · pp. 2–3
- HypGeoRoute softly assigns token representations to learnable geometric cores in the Poincaré ball using distance-aware routing.It also applies core-wise hyperbolic self-attention to model relationships among cores while preserving geometric consistency.Source: §1 · p. 3
2 Related Work
Prior MER research includes deep attention and Transformer approaches as well as hyperbolic methods for emotion and multimodal representation. The paper identifies limited scope in existing hyperbolic approaches and a lack of unified joint optimization in hybrid geometric methods.
2 Related Work
- Deep attention methods and Transformer systems TDFNet and MemoCMT model dynamic interactions across modalities.Earlier MER studies used handcrafted features and simple fusion methods that were sensitive to noise.Source: §2 · p. 3
- Hyperbolic MER research spans conversational hypergraph fusion, heterogeneous speech fusion, label-aware fine-grained emotion embeddings, and hybrid multimodal learning.Emotion Collider [5], HypEmo, and HyperPoint address emotion mixture, label-aware classification, and multimodal representation learning, respectively.Source: §2 · p. 3
- Existing hybrid geometric methods typically combine Euclidean and hyperbolic representations loosely, without a unified framework for joint optimization.The paper also notes that most MER methods remain Euclidean, while hyperbolic approaches often focus on specific components such as embeddings or labels.Source: §2 · p. 3
3 Methodology
GeoMER combines Euclidean cross-modal interaction modeling with hyperbolic routing of tokens into abstract emotional representations. The two branches are combined for emotion classification.
3.1 Feature Embedding
- HuBERT audio and RoBERTa text encoders provide acoustic embeddings and contextual transcription tokens for the framework.Audio is sampled at 16 kHz.Source: §3.1 · p. 4
3.2 Euclidean Multimodal Fusion Module
- EuclFusion uses bidirectional cross-attention and a BiLSTM to align speech and text cues and aggregate their fused sequence into an utterance embedding.The resulting Euclidean representation captures multimodal interactions and complements the hyperbolic branch.Source: §3.2 · pp. 4–5
3.3 Hyperbolic Geometric Core Routing
- HypGeoRoute softly routes multimodal tokens by hyperbolic distance to shared learnable cores, representing latent emotional components in the Poincaré ball.Tokens can contribute to multiple cores, supporting abstract representations of utterances with multiple emotional cues.Source: §3.3 · p. 5, Feature Alignment and Hyperbolic… · pp. 5–6
- Routing-weighted tokens are aggregated into modality-specific core representations using the Einstein midpoint in the Poincaré ball.The procedure maps tokens to the tangent space, aggregates them there, and maps the result back to the manifold.Source: Feature Alignment and Hyperbolic… · p. 6
- Hyperbolic self-attention models semantic relationships among emotion cores, with a Möbius residual connection preserving manifold consistency during feature propagation.Attention-head outputs are concatenated, projected through a hyperbolic linear transformation, and combined with the input.Source: Feature Alignment and Hyperbolic… · pp. 6–7
- A gate based on cross-modal discrepancy fuses each pair of modality-specific cores, and learned core importance weights aggregate them into the hyperbolic branch representation.The Euclidean and hyperbolic representations are concatenated and passed to a linear classifier and softmax.Source: Cross-modal Gated Fusion · pp. 7–8
4 Experiments
On IEMOCAP’s four-class speech-and-text benchmark, GeoMER’s Euclidean–hyperbolic model achieves the strongest reported multimodal results against MemoCMT. The experiments also examine component contributions, per-class performance, and the number of hyperbolic emotion cores.
4 Experiments
- On IEMOCAP’s 5,531 utterances, GeoMER uses speech and text across four emotion classes; evaluation reports WA, UA, and Macro-F1 for class imbalance.The split contains 4,479 training, 498 validation, and 554 test samples; HuBERT-base and RoBERTa-base provide frozen features.Source: MER Benchmark · p. 8, Evaluation Metrics · p. 9, Experiment Setup · p. 9
4.1 Main Results
- 83.54 ± 0.77% WA and 83.35 ± 0.57% UA; GeoMER improves on the reported MemoCMT results by 1.69% WA and 2.02% UA.Table 2 compares representative methods on IEMOCAP’s four-class setting; reproduced MemoCMT comparisons are reported separately.Source: §4.1 · p. 9, Table 2 · p. 10
- Against reproduced MemoCMT, GeoMER improves WA by 2.49% and UA by 2.02%, with Euclidean fusion and hyperbolic modeling playing complementary roles.The Euclidean branch strengthens speech–text interaction, while hyperbolic routing organizes latent abstract representations.Source: §4.1 · p. 10
4.2 Ablation Study
- EuclFusion raises reproduced MemoCMT from 81.05% WA and 81.33% UA to 81.95% WA and 82.38% UA; full GeoMER reaches 83.54% WA and 83.35% UA.Macro-F1 increases from 80.84% to 83.17% for MemoCMT and full GeoMER, respectively.Source: §4.2 · p. 10, Table 3 · p. 11
- Angry has the highest recall at 86.20%, while neutral is most challenging at 78.70% and is misclassified as sad (9.00%) or happy (8.40%).The figure’s confusion matrix places ground-truth labels in rows and predicted labels in columns.Source: Figure 2 · p. 11, §4.2 · pp. 11–12
- The ablations characterize Euclidean fusion and hyperbolic geometry as complementary: one strengthens cross-modal interaction, the other structures abstract representations.Source: §4.2 · p. 11
- WA rises from 82.23% at Nc=2 to 83.54% at Nc=4; Nc=6 and 8 remain at 83.30% and 83.10%, respectively.The authors use Nc=4; soft routing distributes tokens across cores, which act as latent anchors rather than mapping one-to-one to classes.Source: Figure 3 · p. 12, §4.2 · p. 12
5 Conclusion
GeoMER combines Euclidean multimodal fusion with hyperbolic routing to model cross-modal interactions and latent emotional hierarchies. On IEMOCAP, it outperforms MemoCMT, and ablations support the proposed components.
5 Conclusion
- 83.54% WA and 83.35% UA, outperforming MemoCMT by 2.49% WA and 2.02% UA on IEMOCAP.The ablation study further validates the effectiveness of the proposed components.Source: §5 · p. 13
- GeoMER combines EuclFusion for cross-modal interactions with HypGeoRoute for latent hierarchical emotional structures.HypGeoRoute uses distance-aware assignment, manifold-consistent aggregation, core-wise hyperbolic self-attention, and cross-modal gated fusion on the Poincaré ball.Source: §5 · p. 13
- The ablation study further validates the effectiveness of the proposed components.Source: §5 · p. 13