PaperlayerView exact revision on arXiv

Source-linked AI summary

LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation

Ahmed Shehab Khan, Zhiyuan Li, Yan Tong

arXiv:2608.23880v1cs.CV

TL;DR

GER must combine spatially distributed and potentially contradictory cues, while existing methods use detector-driven multi-stream pipelines with only image-level supervision. LG-GER uses an MLLM offline to generate structured spatial evidence and distills it into a single VLM through four losses. It achieves the highest reported accuracy on GAF 3.0 and competitive results on GroupEmoW with detection-free inference.

  • Problem

    GER requires integrating faces, gestures, objects, interactions, and scene context, but image-level supervision does not indicate where cues occur or how strongly they contribute.

  • Method

    LG-GER uses an MLLM to generate spatial evidence offline and distills bounding boxes, emotion signals, and confidence scores into a single VLM through four complementary losses.

  • Results

    LG-GER achieves the highest reported accuracy on GAF 3.0 and competitive results on GroupEmoW with detection-free inference.

  • Takeaways & Limitations

    Spatial supervision changes model behavior toward selective, evidence-driven attention while auxiliary components are discarded at inference.

  • Takeaways & Limitations

    LG-GER excludes diffuse evidence types such as pose, interaction, and environment because their large regions provide little spatial discrimination.

Abstract

from arXiv · show

Inferring the collective emotional state of a group of people from a single image, a task known as group emotion recognition (GER), requires integrating spatially distributed cues such as faces, poses, interactions, and scene context. Current methods rely on detector-driven multi-stream pipelines. These are trained with only image-level supervision that lacks guidance on which regions matter or how strongly each contributes. We propose LG-GER, a language-guided distillation framework that uses a multimodal large language model (MLLM) to generate dense, spatially grounded evidence, i.e., bounding boxes paired with emotion signals and confidence scores, for the training images. This structured evidence is distilled into a single vision-language model (VLM) backbone through four complementary losses: classification, region-text grounding, spatial emotion, and spatial confidence regression. At inference, LG-GER requires no detectors, no MLLM, and no multi-stream fusion, making GER practical for real-time and resource-constrained deployment. LG-GER has been evaluated on two benchmark GER datasets (GroupEmoW and GAF~3.0) and achieves competitive or superior results compared to state-of-the-art methods that require detection and multi-stream processing at inference.

1 Introduction

GER must integrate heterogeneous, context-dependent cues, yet existing detector-driven pipelines rely on undifferentiated image-level supervision. LG-GER uses offline MLLM annotations to distill spatial evidence into a single VLM while retaining competitive or superior benchmark performance.

  • Motivation: Group emotion depends on faces, body language, objects, interactions, and scene context, because isolated cues can signal opposite emotions.The same visual element may indicate different emotions across contexts, making compositional cue integration necessary.
  • Limitations of prior methods: Existing GER pipelines use dedicated detectors, separate encoders, and late-stage fusion, creating propagated detector errors and additional inference cost.These systems also incur engineering overhead and multiple potential failure points.
  • Training gap: Image-level labels provide no region-level guidance about where emotional cues occur or how strongly they contribute.This leaves the image treated as a single undifferentiated unit during training.
  • Proposed remedy: MLLMs can act as offline annotators that generate bounding boxes, emotion signals, and confidence scores without being deployed at inference.This uses MLLM capabilities for supervision while avoiding billion-parameter inference and downstream fine-tuning costs.
  • LG-GER: LG-GER distills structured MLLM evidence into one VLM through four complementary losses and an Emotion Adapter for emotion-discriminative region-text grounding.The evidence pipeline supplies dense, spatially grounded region-level supervision for training images.
  • Evaluation: Across GroupEmoW and GAF 3.0, LG-GER achieves competitive or superior results against state-of-the-art multi-stream methods.The framework is evaluated on two GER benchmark datasets.

2 Related Work

Prior GER methods combine detected regions, separate feature streams, graphs, attention, transformers, and language inputs, but remain detection-dependent at inference. LG-GER instead distills spatially grounded MLLM evidence into a single VLM backbone for detection-free GER.

  • Individual emotion recognition: Individual emotion recognition integrates facial, pose, and scene cues, but these methods address single individuals rather than multiperson group emotion.Context can sometimes be more informative than facial expression alone.
  • Group emotion recognition: GER benchmarks introduced collective emotion classification over images labeled positive, neutral, or negative, with GroupEmoW expanding scale and diversity.The EmotiW challenge series catalyzed the field.
  • Detection and fusion: The dominant GER paradigm detects faces, bodies, objects, or scene regions, encodes streams separately, and fuses their features.Graph-based approaches extend this paradigm by modeling interactions among detected entities.
  • Attention and transformers: Attention and transformer methods dynamically model feature importance and interactions across detected faces, objects, and global scene representations.Examples include regional attention, context-aware fusion, and bidirectional cross-patch attention.
  • Language-based GER: The closest prior language-based GER method adds LLM-generated descriptions and emotion labels to a multi-stream model but still requires detection at inference.It uses language as an additional inference-time input feature.
  • LG-GER distinction: Top-performing GER methods require explicit detection and multi-stream fusion at inference, whereas LG-GER departs from this paradigm.LG-GER uses a single VLM backbone and transfers structured spatial evidence during training.
  • Distillation: MLLMs have increasingly served as offline annotators, but prior emotion-recognition distillation transferred semantic prompts or class-level signals rather than spatial evidence.LG-GER distills bounding boxes with region-level emotion signals, evidence types, and continuous confidence scores.

3 Methodology

LG-GER generates structured spatial evidence offline with an MLLM and distills it into a single VLM backbone using holistic and region-level supervision. Auxiliary spatial heads are used only during training, enabling detection-free inference without an MLLM or multi-stream fusion.

  • LG-GER generates MLLM evidence offline, distills it into a VLM backbone during training, and discards auxiliary heads and MLLM-derived data at inference.Inference uses only the backbone and classification head.
  • Each evidence region contains a normalized bounding box, evidence type, cue description, emotion signal, and signal strength.Signal strength ranges from 0 to 1 and indicates how strongly the cue signals its emotion.
  • The pooled representation drives classification, while patch features within evidence boxes are mean-pooled into region representations for localized supervision.This routing supports holistic and localized representations without gradient competition between classification and spatial objectives.
  • The Emotion Adapter remaps VLM text embeddings into an emotion-discriminative space, allowing region features to align with cue descriptions by cosine similarity.It is trained offline with triplet loss because standard VLM text embeddings are not optimized for emotional discrimination.
  • Four complementary objectives combine image-level classification with region-text grounding, spatial emotion, and spatial confidence supervision.The three spatial heads operate on evidence-region features during training only.

4.1 Experimental Setup

LG-GER is evaluated on two three-class GER benchmarks using dataset-specific train, validation, and test arrangements. Accuracy is reported on the GroupEmoW test set and the GAF 3.0 validation split.

  • GroupEmoW contains 11,127 training, 3,178 validation, and 1,589 test images, while GAF 3.0 contains 9,815 training and 4,346 validation images.GAF 3.0 test labels are not publicly available, so post-competition evaluation uses validation.
  • Both GroupEmoW and GAF 3.0 use positive, neutral, and negative emotion classes.
  • Training uses AdamW with lr = 1 × 10−4 and weight decay = 0.01, cosine annealing, a 2-epoch warmup, 30 epochs, batch size 32, bf16 precision, and gradient clipping.Augmentation includes random resized crops, horizontal flips, color jitter, and random augmentation.
  • Evaluation reports GER accuracy on the GroupEmoW test set and the GAF 3.0 validation split.Wide images use averaged logits over multiple crops; other images use a single center crop.

4.2 Implementation Details

The implementation uses a mostly frozen SigLIP 2 vision backbone with separate pooled and patch representations, lightweight classification and spatial heads, and an offline Emotion Adapter. Evidence selection, loss balancing, and Grad-CAM support the training and qualitative analysis setup.

  • SigLIP 2 ViT-L/16-256 produces a pooled representation and a 16 × 16 patch grid, with only the final two transformer blocks unfrozen.The unfrozen blocks use a 0.1× learning rate while earlier layers remain frozen.
  • A 2-layer MLP maps pooled features to three emotion logits using focal loss with γ = 2.0 and class-balanced α.
  • Spatial emotion and confidence heads classify per-region emotion signals and regress signal strength through BCE.The Emotion Adapter is a separate 2-layer MLP trained offline with triplets from MLLM-generated cue descriptions.
  • Gemini 3.0 Flash supplies evidence annotations, while localized face, gesture, object, and sign regions are retained for spatial supervision.Each image retains 3–8 regions covering approximately 12–15% of the patch grid.
  • GradNorm balances the losses with classification and confidence multipliers, and a 2-epoch curriculum introduces spatial emotion and confidence losses after classification and grounding.Grad-CAM maps are extracted from the first LayerNorm of the final transformer block for qualitative analysis.

4.3 Results

LG-GER achieves strong benchmark performance while removing detection and multi-stream requirements at inference. Ablations, teacher comparisons, and cross-dataset tests support the value of distilled spatial evidence and emotion-adapted representations.

  • Benchmark comparison: 84.08% validation accuracy on GAF 3.0 is the highest reported result, surpassing BridgeFormer by +0.50%.On GroupEmoW, LG-GER reaches 92.39% test accuracy, second only to BridgeFormer’s 94.77%.
  • Benchmark comparison: LG-GER is the only compared method requiring neither detection nor multi-stream fusion at inference.All prior methods use multi-component inference pipelines with explicit detection.
  • Ablation study: +1.07% on GroupEmoW and +1.02% on GAF 3.0 are the full method’s improvements over the baseline.These gains are reported in the component ablation.
  • Teacher comparison: +7.7% on GroupEmoW and +9.1% on GAF 3.0 are LG-GER’s improvements over Gemini zero-shot classification.Gemini 3.0 Flash scores 84.7% and 75.0%, respectively, as a zero-shot classifier.
  • Ablation study: The Emotion Adapter increases class-separation margin from 0.023 to 0.405, an 18× improvement.Inter-class cosine similarity falls from 0.556 to 0.063, while raw embeddings hurt GroupEmoW accuracy by −0.07%.
  • Cross-dataset validation: Cross-dataset accuracy remains 88.23% for GAF→GEW and 79.54% for GEW→GAF without fine-tuning.The reported gaps from in-domain performance are −4.16% and −4.54%, respectively.

4.4 Analysis

Analysis shows that spatial supervision changes where LG-GER attends, improving selected emotion cues while exposing a failure mode on neutral scenes with mixed expressions. Neutral remains the bottleneck class across configurations.

  • Per-class analysis: Neutral remains the bottleneck class across all configurations, while GAF 3.0 shows Positive and Negative improvements but a slight Neutral decrease.The comparison includes the baseline, LG-GER, and Gemini teacher.
  • Qualitative analysis: LG-GER concentrates on emotionally informative regions, whereas the baseline distributes attention broadly across images.The attention comparison uses Grad-CAM maps on GroupEmoW test images.
  • Qualitative analysis: In a celebratory interaction, LG-GER shifts attention toward the central serving interaction and correctly predicts Positive instead of Neutral.The difference map shows gained attention on contextual interaction and lost attention in the uninformative periphery.
  • Qualitative analysis: LG-GER amplifies the most expressive face and suppresses background, correctly predicting Positive where the baseline misses the emotional tone.The model does not attend to all faces equally.
  • Qualitative analysis: For a neutral crowd with mixed expressions, focused attention misclassifies the scene as Negative while broad attention correctly predicts Neutral.The example exposes a tension between selective and diffuse attention strategies.

5 Conclusion

LG-GER distills MLLM-generated spatial evidence into a single-stream VLM, using spatial training signals while discarding auxiliary components at inference. It combines competitive benchmark performance with detection-free inference and evidence of generalized emotional patterns.

  • Conclusion: LG-GER distills MLLM-generated spatial evidence into a single-stream VLM backbone for group emotion recognition.Training uses region-text alignment, spatial emotion classification, and spatial confidence regression.
  • Conclusion: The framework achieves the highest reported accuracy on GAF 3.0 and competitive results on GroupEmoW with detection-free inference.Auxiliary components are discarded at inference.
  • Conclusion: Spatial supervision shifts attention from diffuse scanning to selective, evidence-driven focus, while cross-dataset validation indicates generalized emotional patterns.Future work includes annotation-quality auditing and extension to video-based GER.
Loading 2608.23880v1…