Source-linked AI summary
Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion
Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant
TL;DR
OV-AVEL’s fixed scalar fusion collapses visual–text and audio–text similarities, and standard audio inputs discard phase. The paper learns complex-valued fusion with paired modality companions; on AVE-OV, its 4-stream model is best across all twelve reported cells.
Problem
Existing OV-AVEL pipelines collapse visual–text and audio–text similarities with a fixed scalar rule, while standard audio input discards phase cues.
Source: §1 · p. 2Method
Results
On AVE-OV, the 4-stream complex model gains +8.9/+5.9/+8.4 Avg on seen/unseen/total versus the fine-tuned baseline and is best on all twelve cells.
Source: §4.4 · p. 11Takeaways & Limitations
The 2-stream complex fusion supplies the dominant gain, while auxiliary streams add a smaller complementary gain.
Source: §5 · p. 14
Abstract
from arXiv · showhide
Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex architecture sets a new state of the art on both OV-AVEL benchmarks. On the open (unseen-class) split of OV-AVEBench we reach 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 (+1.6/+4.1/+6.6 over the previously reported fine-tuned baseline), with consistent gains for seen classes as well. We also modify AVE dataset for this task and observe that our architecture reaches 60.7/51.9/50.4% Acc/Seg-F1/Event-F1, achieving state-of-the-art OV-AVEL results on it as well. We also propose a two-stream alternative, which also sees great improvements over the baseline.
1 Introduction
OV-AVEL pipelines reduce visual–text and audio–text similarities to a fixed scalar, while also discarding audio phase. The paper replaces scalar fusion with complex-valued fusion and adds paired visual and audio companion streams.
1 Introduction
- Existing OV-AVEL methods combine visual–text and audio–text similarities with a fixed scalar rule, discarding a degree of freedom that could vary across classes and segments.The cited rules include weighted averaging and geometric mean.Source: §1 · p. 2
- On OV-AVEBench’s unseen split, the 4-stream complex model improves baseline Segment-F1/Event-F1 by +4.1/+6.6.Figure 1 also depicts the 2- and 4-stream pipelines.Source: Figure 1 · p. 2
- The paper also identifies discarded audio phase as a limitation because phase carries onset, harmonic, and transient cues absent from magnitude alone.The phase-derived input is shaped to mel before being passed to the frozen encoder.Source: §1 · p. 2
- The paper treats visual–text and audio–text similarities as the real and imaginary parts of a complex input to a five-layer ComplexCNN.This replaces the scalar geometric-mean fusion of.Source: §1 · p. 2
- The four-stream extension pairs visual iHSV and CycleGAN-translated audio-phase companions with their modalities and feeds both complex pairs to the same ComplexCNN.The auxiliary streams add a smaller, complementary gain on top of the two-stream fusion.Source: §1 · pp. 2–3
- The two-stream approach combines visual content as the real part and audio as the imaginary part of one complex channel, while the four-stream approach uses a separate complex channel per modality.The paper characterizes the two-stream representation as simpler, more efficient, and redundancy-reduced.Source: §1 · p. 3
2 Related Work
Prior work spans closed-vocabulary AVEL, video-level zero-shot learning, and open-set AVE, but OV-AVEL requires segment-level labels for specific unseen classes. This paper places its contribution in complex-valued fusion and complementary audio-phase and visual-structure inputs.
2 Related Work
- Closed-vocabulary AVEL requires test classes to overlap training classes, while generalized zero-shot AV learning predicts unseen categories at the whole-video level and OpenAVE only adds an unknown class.OV-AVEL instead assigns explicit segment-level labels to unseen categories.Source: §2 · p. 3
- Open-vocabulary AVEL uses text-aligned foundation-model embeddings, but prior OV-AVEL methods combine visual–text and audio–text similarities with a fixed rule before classification.The paper uses frozen ImageBind like the baseline and changes fusion rather than the representation stage.Source: §2 · p. 3
- Complex-valued networks tie real weights into structured cross-channel mixing, while audio phase and the iHSV visual companion supply complementary inputs for the proposed fusion.The cited background describes phase cues and the mel-manifold translation used to make phase-derived audio usable by a frozen encoder.Source: §2 · p. 4
3 Methodology
The method keeps ImageBind’s multimodal encoders frozen and replaces fixed scalar fusion with complex-valued, segment-level classification. Paired visual and audio streams form two complex channels that a ComplexCNN fuses, while unseen classes are supplied through text prompts at inference.
3.1 Frozen backbone and notation
- Unseen test classes are introduced by appending their names to the text prompts, while the ImageBind encoders remain frozen.The model classifies each one-second segment of a 10-second video, including a background category.Source: §3.1 · p. 4, §3 · p. 4
3.2 Visual stream as a complex pair: RGB +i· imaginary component of iHSV
- The visual complex stream pairs RGB content with an iHSV-derived colour-structure companion, both processed by the frozen vision encoder.The paired companion preserves cyclic hue and chromatic structure in a synthetic HSV image.Source: §3.2 · p. 5
3.3 Audio stream as a complex pair: mel +i· CycleGAN-phase
- The audio complex stream pairs log-mel magnitude with phase translated into a mel-compatible input for the frozen audio encoder.Temporal attention produces text-aligned similarities for the mel and translated-phase streams.Source: §3.3 · p. 6, §3.3.1 · p. 8
3.3.1 Imaginary part: CycleGAN-translated phase.
- An offline, unpaired CycleGAN maps instantaneous-frequency phase to mel; at OV-AVEL time, only its IF-to-mel generator is used.Its training includes adversarial, cycle-consistency, identity, and distribution-matching losses; distribution matching anchors translated mel statistics to real mel for the frozen encoder.Source: §3.3.1 · pp. 6, 8, Figure 3 · p. 7
3.4 ComplexCNN fusion
- The full model stacks visual and audio complex similarities as two channels and fuses them with a five-layer ComplexCNN.Temporal-only kernels avoid mixing class indices, and the magnitude of the complex output is rescaled into per-segment class scores.Source: §3.4 · pp. 8–9
3.5 Training objective and trainable parameters
- Training uses per-segment cross-entropy; the trainable modules are the temporal-attention blocks and ComplexCNN, while ImageBind and the phase translator are frozen.At inference, the predicted class is the highest-scoring class, with unseen classes added through text prompts and no per-class head.Source: §3.5 · pp. 9–10
4 Experiments and Results
The experiments evaluate the approach on OV-AVEBench and AVE-OV, using segment-level metrics and ablations to test its benchmark performance and design choices. The four-stream model leads on AVE-OV, while cross-dataset transfer and ablations reveal differences between the two- and four-stream variants and between complex and real-valued fusion.
4.1 Datasets and Splits
- OV-AVEBench has 46 seen and 21 unseen classes; AVE-OV recasts AVE with 20 seen and 8 unseen classes and is roughly 6× smaller, serving as an out-of-distribution stress test.Both use T=10 frame labels; their reported splits include seen, unseen, and total test partitions.Source: §4.1 · p. 10
4.2 Evaluation Metrics
- Acc averages frame correctness, Seg-F1 macro-averages per-clip frame-wise F1, and Event-F1 scores matching contiguous predicted events at IoU ≥0.5.Event-F1 also penalizes boundary jitter, label flicker, and class confusion.Source: §4.2 · p. 10
4.3 Implementation
- Frozen ImageBind encoders feed trained temporal-attention blocks and a 5-layer ComplexCNN; the 2-stream adds negligible latency, while 4-stream adds roughly +135 ms.The encoders remain frozen; the temporal-attention blocks and fusion network are trained.Source: §4.3 · p. 11
4.4.1 Cross-dataset transfer
- In both transfer directions, 2-stream fusion retains a positive margin over the fine-tuned baseline (+3.6/+8.5 Avg), while 4-stream transfer varies by direction.Relative to 2-stream, 4-stream drops −4.8 Acc on OV→AVE but gains +2.1 Acc on AVE→OV.Source: §4.4.1 · p. 11
4.4 Experimental Results
- On AVE-OV, the 4-stream model wins all 12 cells and improves unseen-split scores over the re-implemented fine-tuned baseline by +2.7/+5.2/+9.6/+5.9 points (Acc/Seg/Eve/Avg).On two unseen-class OV-AVEBench clips, the 4-stream model predicts every segment correctly; the baseline confuses “female singing” and misses helicopter segments as background.Source: §4.4 · p. 11, Table 2 · p. 12, Figure 6 · p. 13
4.5 Ablation Studies
- Ablations compare complex fusion with a parameter-matched RealCNN and test stream count; swapping the real/imaginary assignments performs worse than the chosen assignment.The parameter-matched comparison uses unseen categories, where generalization is most sensitive to architectural choices.Source: §4.5 · p. 12, §4.5.1 · p. 12
4.5.1 Ablation: ComplexCNN vs Real CNN fusion
- Complex fusion beats matched real fusion by +7.9/+2.0 Event-F1 at 2/4 streams on OV-AVEBench; auxiliary streams lift complex unseen Event-F1 +0.8/+2.3 on OV-AVEBench/AVE-OV.On AVE-OV, real-to-complex fusion yields a +15.9 Event-F1 gap at 2-stream and a tie at 4-stream; on OV-AVEBench, 4-stream real fusion still trails 2-stream complex fusion.Source: §4.5.1 · p. 12
- Replacing CycleGAN-translated phase with raw instantaneous-frequency input drops Event-F1 by 2.2/1.9 points on OV-AVEBench/AVE-OV.The comparison holds the 4-stream complex-pair architecture fixed; the audio imaginary stream is the only change.Source: §4.5.2 · p. 13, Table 6 · p. 14
5 Conclusion
The conclusion presents complex-valued fusion as the dominant contribution and a four-stream extension as a complementary addition. The full architecture improves OV-AVEBench results, though one cross-dataset transfer experiment favors the two-stream approach.
5 Conclusion
- On OV-AVEBench unseen, the full architecture improves the fine-tuned baseline by +6.6 Event-F1 (+1.6/+4.1 Acc/Seg); seen and total partitions gain +9.1/+7.5 Event-F1.The four-stream approach outperformed the two-stream approach on both datasets.Source: §5 · p. 14
- Two-stream complex fusion treats visual–text and audio–text similarities as a complex value; paired iHSV visual and CycleGAN-translated phase audio streams add a smaller complementary gain.Both contributions feed the resulting complex values or content–structure pairs to a five-layer complex-valued CNN.Source: §5 · p. 14
- At two streams, complex-over-real fusion yields +7.9 Event-F1 on OV-AVEBench and +15.9 on AVE-OV; at four streams, parameter-matched gains are +2.2 Avg and +1.0 Avg, respectively.Source: §5 · p. 14
- Adding iHSV and CycleGAN phase to the two-stream complex model adds +0.8/+2.3 Event-F1; translated phase adds +2.2 Event-F1 over raw IF.The auxiliary-stream comparisons are reported across OV-AVEBench and AVE-OV, respectively.Source: §5 · p. 14
- The four-stream model performs worse than the two-stream model in one of two cross-dataset transfer experiments; the authors suggest possible overfitting on the larger training dataset.The overfitting explanation is presented as a possibility.Source: §5 · p. 14