SoftRerank: Hierarchical Soft Fusion with Candidate-Label Reranking for Long-Tailed Micro-Action Recognition

• Micro-actions have weak motion, sparse temporal evidence, and small inter-class variations, making brief localized patterns difficult to distinguish.MA-52 organizes 52 fine-grained categories into seven coarse body-motion groups, with similar actions sometimes sharing motion regions and trajectories.Source: §1 · pp. 1–2
• Long-tailed labels, sparse discriminative frames, and similar actions challenge MA-52 recognition; hard coarse-to-fine cascades can propagate errors and exclude plausible labels.Source: §1 · p. 2
• End-to-end InternVideo2.5 fine-tuning shares video representations across coarse and group-conditional fine heads, while soft fusion globally searches all 52 labels.Training also uses class-balanced sampling, inverse-frequency reweighting, rare-class augmentation, and hierarchy-aware Mixup.Source: §1 · p. 2
• Selective reranking activates for low-confidence or small-margin predictions and compares video features with a compact set of likely confused labels.Candidates include high-scoring incorrect predictions, labels from the same body-motion group, and frequently confused labels.Source: §1 · p. 2
• 79.99% F1-mean on MA-52 earned the framework first place in the 3rd Micro-Action Analysis Grand Challenge at ACM Multimedia 2026.Source: §1 · p. 2
• Figure 1 presents class-balanced sampling, inverse-frequency reweighting, and stronger rare-class augmentation as preprocessing strategies for MA-52.Source: Figure 1 · p. 2

• Skeleton-based methods model joint relations and motion over time; they are robust to background and lighting shifts but depend on pose quality and struggle with occlusion, fast motion, and fine appearance details.Source: §2 · p. 2
• Figure 2 depicts the proposed framework integrating imbalance-aware preprocessing and InternVideo2.5 representations with coarse-to-fine soft fusion and candidate-label reranking.Source: Figure 2 · p. 3
• Multimodal methods fuse complementary signals, while MAR systems such as MANet and PCAN address competition and ambiguity among micro-actions.Recent MA-52 systems commonly pair body-level context with fine-motion cues.Source: §2 · pp. 2–3
• Video foundation models are increasingly adapted to fine-grained motion, with fine-tuning efficiency and inherited long-tail bias identified as open questions.Source: §2 · p. 3

3.1 Data Preprocessing

• Tail classes receive stronger augmentation, while class-balanced sampling and within-group inverse-frequency reweighting address the long-tailed label distribution.Sampling interpolates between instance-balanced and class-balanced distributions; augmentation intensity also scales with class frequency.Source: §3.1 · p. 3
• Hierarchical Mixup/CutMix interpolates fine and coarse labels together, preserving parent assignments and supervising both heads.Cross-group mixing places soft samples on group boundaries, where soft fusion performs its arbitration.Source: §3.1 · p. 4

3.2 Backbone and Full Fine-tuning

• The authors fully fine-tune InternVideo2.5 so its shared representation adapts to subtle action differences, without an external temporal contextualization module.They pair full fine-tuning with data-side rebalancing and augmentation because tuning a billion-scale backbone on this dataset is prone to overfitting.Source: §3.2 · p. 4

3.3 Hierarchical Soft Fusion

• Hierarchical soft fusion combines coarse-group and group-conditional fine-label probabilities, searching all fine labels rather than only the coarse top-1 group.For a flat 52-way fine head, the authors instead describe log-linear product-of-experts fusion, with the coarse-signal weight selected on validation macro-F1.Source: §3.3 · p. 4

3.4 Lightweight Candidate-Label Reranker

• Out-of-fold predictions provide reranker training examples, and each candidate set combines the true label with likely confusions from model predictions, the same body-motion group, and validation confusion statistics.Out-of-fold predictions come from models that did not train on the corresponding samples; the candidate sets target hard negatives rather than random labels.Source: §3.4 · pp. 4–5
• The lightweight matching network compares video and candidate-label features using their product and absolute difference, then scores and ranks candidates with a candidate-level cross-entropy objective.The objective supervises the ground-truth label’s relative ranking within the candidate set.Source: §3.4 · p. 5
• At inference, the reranker refines low-confidence or small-margin predictions, while retaining the base model’s top-1 for high-confidence samples.Training also includes some low-confidence correct examples alongside incorrect samples, helping it learn when to preserve the base prediction.Source: §3.4 · p. 5

4.1 Datasets and Metrics

• MA-52 contains 22,422 videos from 205 subjects; F1mean is the final metric for unified coarse- and fine-grained assessment.The challenge rankings use 1,138 samples from the 5,586-sample MA-52 test set.Source: §4.1 · p. 5

4.2 Implementation Details

• Implementation uses mixed precision and activation checkpointing; videos are center-cropped, resized to 224 × 224, and processed on 8 A100 GPUs.The backbone learning rate was grid-searched from 3e-6 to 3e-7 to keep optimization stable.Source: §4.2 · pp. 5–6

4.4 Ablation Analysis

• Soft fusion raises F1mean from 75.25 to 79.09; preprocessing raises it to 79.74, reranking to 80.44, and their combination to 81.29.Reranking raises action-level macro-F1 from 69.50 to 71.51; preprocessing mainly improves micro-F1, while macro-F1 scores remain nearly unchanged.Source: Table 3 · p. 6, §4.4 · p. 6, Table 2 · p. 6

4.3 Comparison with State-of-the-Art Methods

• The method achieves the best body-level, action-level, and overall performance among the compared micro-action recognition approaches.The final comparison uses the MA-52 test set of 1,138 samples.Source: §4.3 · p. 6

5 Conclusion

• The framework fine-tunes InternVideo2.5 end to end and combines imbalance-aware training, hierarchical soft fusion, and candidate-label reranking.Class-balanced sampling and inverse-frequency reweighting address MA-52’s long-tailed label distribution.Source: §5 · p. 6
• Coarse and group-conditional fine-grained heads share a video representation; soft fusion preserves global competition among fine-grained classes.The reranker uses hard samples and candidate-level video-label matching to target easily confused actions.Source: §5 · p. 6
• 79.99% F1-mean on MA-52 surpasses the previous state of the art by 2.24 F1 points and ranks first in the MAR challenge.Source: §5 · p. 6