Source-linked AI summary
Asymmetric Cross-Modal Fine-Grained Visual Categorization: ACF-Net and the BirdPro Benchmark
Bohan Deng, Shuo Ye, Zitong Yu
TL;DR
Asymmetric audio-visual FGVC lacks strict instance- and time-level correspondence, complicating representation learning and modality alignment. The paper proposes ACF-Net and BirdPro, with experiments reporting state-of-the-art performance and improved robustness in realistic non-ideal settings.
Problem
Asymmetric audio-visual FGVC often lacks strict instance- or time-level correspondence, while existing FGVC methods commonly assume unimodal inputs or reliable cross-modal correspondence.
Method
ACF-Net uses Optical Flow-Guided Motion for motion-sensitive visual representations and reliability-aware adaptive fusion for weakly matched audio-video pairs.
Results
ACF-Net achieves state-of-the-art performance, with extensive experiments demonstrating effectiveness and robustness in realistic non-ideal cross-modal settings.
Takeaways & Limitations
BirdPro provides a bird-oriented benchmark for asymmetric audio-visual FGVC, while ACF-Net supports category-level recognition under weakly matched pairs.
Takeaways & Limitations
The two modalities are only category-aligned and may come from different temporal segments, recording conditions, or instances of the same bird species.
Abstract
from arXiv · showhide
Audio-visual cross-modal Fine-Grained Visual Categorization (FGVC) aims to identify fine-grained categories by jointly leveraging visual and auditory information. However, FGVC under asymmetric cross-modal scenarios has received limited attention, where paired video and audio are not strictly synchronized and may not even correspond to the same individual or moment. Such weak and ambiguous cross-modal correspondence poses substantial challenges to effective representation learning and modality alignment. To address these issues, we propose ACF-Net, a novel optical flow-guided framework for asymmetric audio-visual fine-grained learning. ACF-Net consists of two key modules: Optical Flow-Guided Motion (OFGM) and Asymmetric CrossModal Adaptive Fusion (ACAF). OFGM captures motion-sensitive visual cues and suppresses irrelevant background interference, thereby enhancing discriminative dynamic representations in videos. ACAF estimates modality reliability under weakly matched audio-video pairs and performs uncertainty-aware adaptive fusion to improve category-level recognition robustness. To support research on asymmetric cross-modal FGVC, we further construct BirdPro, a new bird-oriented audio-visual benchmark, since existing datasets often lack large-scale category-level audio-video associations under non-strict temporal and instance correspondence. BirdPro contains 1,919 audio recordings and 11,965 videos covering 194 bird species. Extensive experiments show that ACF-Net achieves the best results compared with representative baseline methods, outperforming the strongest baselines by 2.97% and 1.92% in the fused and mismatched settings, respectively.
1 Introduction
The paper studies asymmetric audio-visual FGVC with only category-level correspondence and introduces ACF-Net and BirdPro to address weak alignment and heterogeneous modalities.
- ACF-Net combines Optical Flow-Guided Motion and Asymmetric Cross-Modal Adaptive Fusion for asymmetric audio-visual FGVC.OFGM emphasizes category-relevant dynamics and suppresses static background distractions, while ACAF estimates modality reliability for adaptive fusion.
- Asymmetric audio-visual FGVC permits audio and video to differ in instance, time, or context while retaining category-level semantic consistency.
- Reliability-aware fusion lets the model rely more on the informative modality under weakly associated audio-video pairs.The design targets robust category-level recognition in asymmetric settings.
- BirdPro provides a bird-oriented benchmark for asymmetric audio-visual FGVC.The benchmark addresses the need for category-level audio-video associations without strict temporal and instance correspondence.
- ACF-Net achieves state-of-the-art performance in the reported experiments.
2 Related Work
Prior FGVC research largely uses unimodal inputs or reliable cross-modal correspondence, while asymmetric settings require complementary cues despite noisy or weak modality relationships.
- FGVC distinguishes visually similar categories by capturing subtle discriminative cues through part-based, regional, multiscale, attention, and structured representations.
- Video FGVC exploits temporal dynamics, and auditory fine-grained recognition provides complementary semantic cues for bird species recognition and bioacoustic monitoring.
- Most existing FGVC methods use unimodal settings or assume reliable cross-modal correspondence.
- Asymmetric audio-visual scenarios require exploiting complementary cues while suppressing noisy or weakly related modal information.
- Cross-modal alignment learns semantically consistent representations using shared embeddings, contrastive learning, attention, or collaborative encoding.
3 Proposed Method
ACF-Net addresses asymmetric audio-visual FGVC by enhancing visual motion cues and adaptively fusing audio-visual predictions according to modality reliability. Its training combines fused and modality-specific classification with prototype and concept-decoupling regularization.
- Overall framework: ACF-Net processes category-aligned audio-video pairs that may come from different temporal segments, recording conditions, or bird instances.The modalities share a category label but lack strict instance- or time-level correspondence.
- Optical Flow-Guided Motion: OFGM computes optical-flow motion masks to emphasize dynamic visual regions while preserving appearance information and weakening less relevant regions.The mask is normalized, applied element-wise to RGB frames, and retains a minimum appearance intensity controlled by α.
- Optical Flow-Guided Motion: Unlike RGB-flow feature concatenation, OFGM uses optical flow only as a spatial prior, avoiding an additional motion encoder.This formulation guides the visual encoder toward fine-grained foreground dynamics.
- Asymmetric Cross-Modal Adaptive Fusion: ACAF estimates uncertainty from modality prediction distributions and converts it into sample-adaptive fusion weights.Lower entropy indicates a more confident modality, while higher entropy indicates greater uncertainty; a learnable audio scaling factor is included.
- Training objective: The training objective combines fused, visual, and audio classification losses with audio prototype and concept-decoupling regularization.Coefficients λv, λa, λp, and λd control the contributions of the corresponding loss terms.
4 Experiments
Experiments establish BirdPro as a semantically consistent bird-oriented benchmark and evaluate ACF-Net under unimodal, fused, mismatched, ablation, and visualization settings. Results show benefits from reliability-aware fusion, balanced regularization, and motion-aware representations.
- Dataset Construction: BirdPro uses CUB species-level annotations and corresponding scientific names to retrieve semantically consistent videos and audio data.Candidate videos are manually screened for visual identifiability and dominant target regions, while key frames cover diverse viewpoints, poses, and environments.
- Evaluation Settings: Class-level derangement reassigns training audio from class c to another class π(c), while validation remains clean to test mismatch robustness.The experiment evaluates whether fusion strategies tolerate corrupted audio-visual correspondence during training.
- Comparison Results: 87.23% fusion accuracy makes ACF the best method, exceeding the strongest baseline by 2.97% in the matched comparison.Uncertainty-aware fusion reduces the influence of unreliable audio cues, whereas direct fusion variants do not consistently improve performance.
- Comparison Results: 74.26% accuracy makes ACF the strongest method under mismatched correspondence, outperforming the strongest baseline by 1.92%.Compact Bilinear Pooling falls to 52.55%, while uncertainty-aware methods better tolerate corrupted cross-modal supervision.
- Ablation Study: ACAF raises fusion accuracy from 82.98% to 84.89% and audio accuracy from 45.96% to 57.66%, while the full model reaches 87.23% fusion accuracy.OFGM improves video accuracy from 71.06% to 71.92%; together, OFGM and ACAF provide complementary benefits.
- Visualization: ACF-enhanced features form more compact audio clusters, refine video discriminability, and focus attention on bird regions such as the head, neck, and body.Raw audio features are scattered and entangled across categories, while attention suppresses irrelevant background areas.
5 Conclusion
The paper studies asymmetric audio-visual FGVC without strict instance- or time-level correspondence, proposing ACF-Net and the BirdPro benchmark. Experiments demonstrate ACF-Net’s effectiveness and robustness in realistic non-ideal cross-modal settings.
- Conclusion: ACF-Net enhances motion-sensitive visual cues and performs reliability-aware adaptive fusion for weakly matched audio-video pairs.BirdPro is constructed as a bird-oriented benchmark for asymmetric audio-visual FGVC.
- Conclusion: Extensive experiments demonstrate ACF-Net’s effectiveness and robustness for fine-grained recognition in realistic non-ideal cross-modal settings.The conclusion frames these results within asymmetric audio-visual FGVC lacking strict correspondence.