When Fusion Order Matters: Insights on Progressive Multimodal Representation Learning for Cross-modal Retrieval

1 Introduction

• Retrieval must combine complementary visual, acoustic, and spoken cues into representations suitable for matching, but the same inputs do not determine their fusion order.The paper frames the unresolved question as how order affects retrieval when supporting components are held fixed.Source: §1 · p. 1
• The tested one-by-one schedules outperform stacked fusion, and audio-before-speech reaches 53.9 text-to-video R@1 versus 53.2 for the reverse order.UniPRO compares these paths with matched encoders, components, and training settings.Source: §1 · p. 1
• Complete-modality controls and frozen-model tests show that missing inputs alone cannot explain fusion-order sensitivity.The order difference persists with all three modalities present, and a learned difference can remain when inputs are removed only at inference.Source: §1 · p. 1
• For audio queries on complete MSR-VTT samples, introducing audio last yields higher R@1 in both retrieval directions, but not at every recall cutoff.This supports examining whether later fusion preserves information relevant to the query.Source: §1 · p. 1
• 97.9 R@1 on MSR-VTT-9k makes the predicted visual-last schedule the highest-performing evaluated order for image queries.The prediction revisits the original visual representation after visual, acoustic, and speech fusion.Source: §1 · p. 2
• Speech missingness in training and testing reverses the preferred order in audio-to-video R@1 and video-to-audio SumR, with metric-dependent crossover points.The findings motivate balancing input-pattern fit with later fusion of information matching the query.Source: §1 · p. 2
• UniPRO is introduced as a common framework for comparing fusion organization and modality order, and its pipeline is evaluated on four text-to-video benchmarks with component ablations.The contribution also uses the resulting principles to predict an order for image queries.Source: §1 · p. 2

2.1 Cross-Modal Video Retrieval

• Text-to-video methods improve matching between text queries and visual content, while audio and image query approaches extend retrieval beyond language.The cited work includes CLIP-based video retrieval, audiovisual representations, and image- or language-centered cross-modal alignment.Source: §2.1 · p. 2

2.2 Multimodal Video Representation Learning

• Multimodal video methods add acoustic and speech-semantic cues, improve alignment, and regulate audio contributions to video retrieval.UniPRO examines how different fusion paths combine the same inputs and interaction components.Source: §2.2 · p. 2

2.3 Multimodal Interaction and Progressive Fusion

• Prior interaction architectures vary information-exchange direction and communication paths; UniPRO studies modality-input order within a fixed fusion structure.This differs from refinement across network depth, such as feeding late-fusion information back to earlier unimodal feature generators.Source: §2.3 · p. 2

3.1 Modality Representations

• UniPRO encodes video frames, soundtrack, and available transcripts as visual, acoustic, and speech-semantic features, then matches their fused representation to an independently encoded query.CLIP encodes visual frames and text queries, AST encodes audio, and Whisper transcripts are encoded with CLIP’s text encoder.Source: §3 · p. 2, §3.1 · pp. 2–3

3.2 Component Design

• POSR prioritizes retrieval-relevant transcript words under the CLIP input budget, while REM uses bidirectional attention to strengthen audio–speech relations and MFSA aligns all three modality pairs.POSR favors word categories likely to carry entities, actions, and attributes; REM enhances audio and speech streams before progressive fusion.Source: §3.2.1 · p. 3, §3.2.2 · p. 3, §3.2.3 · p. 3

3.3 A Unified View of Progressive Fusion

• One-by-one fusion passes each updated representation to the next gated stage, and the text-retrieval comparison varies whether audio or speech is introduced second.The method distinguishes schedules by modality order and, for stacked fusion, by which modality supplies the first attention query.Source: §3.3 · p. 4

3.4 Training and Transfer Across Query Modalities

• For image queries, UniPRO adds a third gated stage that revisits the original visual tokens, comparing schedules that vary the preceding audio and speech order.The four three-stage schedules either end with visual features or introduce the repeated visual representation earlier; VAS is the two-stage reference.Source: §3.4 · p. 4

4.1 Datasets

• The four benchmarks are MSR-VTT-9k, MSR-VTT-7k, VATEX, and Charades, using their stated train, validation, and test splits.The 6,000 official VATEX test videos are excluded because their annotations are private.Source: §4.1 · p. 4

4.2 Text-to-Video Experimental Configuration

• The main text-retrieval experiments use five training epochs, a 512-dimensional embedding space, and four-layer gated fusion and query-resampler modules.Frame and token sampling, batch sizes, learning rates, and contrastive margins vary by dataset or parameter group.Source: §4.2 · p. 4

4.3 Evaluation Metrics

• Evaluation reports R@1, R@5, and R@10 for both retrieval directions, plus SumR, the sum of those three recall values within each direction.Scores and differences are displayed to one decimal place, so rounding may cause small discrepancies.Source: §4.3 · pp. 4–5

4.5 Performance

• UniPRO improves aggregate recall on all four benchmarks, with SumR gains of 4.7 on MSR-VTT-9k, 1.9 on MSR-VTT-7k, 1.4 on VATEX, and 2.6 on Charades.On MSR-VTT-9k, it reaches 53.9 R@1, 79.6 R@5, 87.4 R@10, and 220.9 SumR.Source: Table 1 · p. 5, §4.5 · p. 5

4.4 Comparison Methods

• UniPRO is compared with vision-only methods and multimodal methods that additionally use acoustic or speech information.The comparison includes four benchmarks and representative methods such as CLIP4Clip, X-Pool, ECLIPSE, TEFAL, AVIGATE, and SAVE.Source: §4.4 · p. 5

4.5 Performance

• With ViT-B/16, UniPRO reaches 55.0 R@1 and 224.1 SumR for text-to-video retrieval, and 51.9 R@1 and 220.0 SumR for video-to-text retrieval.These SumR scores exceed SAVE by 1.9 and 1.5 points, respectively.Source: §4.5 · p. 5
• UniPRO reaches 50.5 R@1 and 217.2 SumR on MSR-VTT-9k video-to-text retrieval, exceeding AVIGATE by 0.8 R@1 points and SAVE by 1.6 R@1 and 3.8 SumR points.It also improves on SAVE at all three recall cutoffs.Source: §4.5 · p. 5, Table 2 · p. 6

Core Component Ablation

• Removing REM reduces SumR by 3.0 points for text-to-video and 0.9 points for video-to-text retrieval; removing POSR reduces SumR by 4.8 and 2.1 points, respectively.Removing POSR also reduces R@1 by 2.2 and 1.1 points; these changes are consistent with transcript condensation retaining semantics that fixed truncation may discard.Source: §5 · p. 6
• Adding the original visual representation after progressive fusion reduces SumR by 2.4 points for text-to-video and 0.4 points for video-to-text retrieval.Reintroducing visual features at the output does not automatically improve the learned representation; its effect depends on how the information enters fusion.Source: §5 · p. 6

Multi-Feature Soft Alignment Ablation

• Removing vision–speech alignment reduces SumR by 2.3 points for text-to-video and 1.0 point for video-to-text retrieval; removing audio–speech alignment reduces it by 1.9 and 1.6 points.Both alignment paths contribute beyond audiovisual alignment alone.Source: §5 · p. 6

6.1 Fusion Organization and Order

• Both one-by-one schedules outperform stacked fusion at all three recall cutoffs, and VAS exceeds VSA by 0.7 R@1 points.The comparison uses matched encoders, components, and training settings.Source: §6.1 · pp. 6–7, Figure 3 · p. 6

6.2 Principle I

• Complete-input controls leave VAS ahead of VSA, including with test-time input removal, so missing inputs alone do not explain order sensitivity.On CM-Pool, VAS has T2V R@1 of 53.5 versus 52.7 and V2T R@1 of 52.0 versus 50.5; its T2V lead also persists across test-input conditions after complete-input training.Source: §6.2 · p. 7

6.3 Principle II

• For audio queries on complete MSR-VTT samples, VSA raises A2V R@1 from 55.2 to 56.6 and V2A R@1 from 57.2 to 58.5, though the advantage does not hold at every cutoff.The text and audio experiments use different datasets, so their comparison does not isolate query modality alone.Source: §6.3 · pp. 7–8

6.4 Balancing the Principles

• Speech missingness reverses the preferred order in both A2V R@1 and V2A SumR, with crossover points that differ by retrieval direction and metric.VSA leads A2V R@1 through 60% missingness, while VAS leads at 70%–90%; V2A SumR favors VSA at 0%–10% and VAS from 20% onward.Source: §6.4 · p. 8

6.5 Predicting an Order for Image Queries

• 97.9 I2V R@1: VASV exceeds VAS by 0.9 points, VSAV by 0.4, and VVAS/VAVS by 0.5 on MSR-Frame.Its R@5 and R@10 are also at least as high as the tested alternatives; broader robustness remains to be established.Source: Figure 6 · p. 8, §6.5 · p. 8

• UniPRO makes fusion order an explicit choice in a common framework for progressive multimodal representation learning.Source: §7 · p. 8
• The evaluated one-by-one schedules outperform stacked fusion, and complete-input controls show that missingness alone cannot explain the remaining order differences.Source: §7 · p. 8
• Learning difficulty and preservation of query-relevant information offer complementary guidance; speech-missingness reversals motivate balancing them as availability changes.Source: §7 · p. 8
• Repeated-run evaluation and broader query protocols can further test the generality of these empirical principles.Source: §7 · p. 8

A.1 CM-Pool

• CM-Pool independently filters the MSR-VTT, VATEX, and Charades training and test splits for valid visual, audio, speech, and text data, then concatenates retained portions into 17,002 training and 1,436 test videos.The source filtering retains 5,679/638 MSR-VTT-9k, 10,178/612 VATEX, and 1,145/186 Charades training/test videos.Source: §A.1 · p. 10, A Construction of the Analysis Datasets · p. 10
• CM-Pool keeps training and test identities disjoint, excludes the VATEX validation split, and uses one deterministically selected training query per video per epoch and one fixed test query per video.The test protocol yields 1,436 queries and 1,436 video candidates; VAS and VSA share videos and query selection.Source: §A.1 · p. 10

A.2 MSR-Avail

• MSR-Avail partitions test videos into V+A+S, V+A without S, and V-only groups, totaling 1,000 videos in MSR-VTT-9k and 2,990 in MSR-VTT-7k.The groups contain 638/307/55 and 1,872/963/155 videos, respectively.Source: §A.2 · p. 10, Figure 8 · p. 10

A.3 MSR-Complete and MSR-SDrop

• MSR-Complete retains 5,679 training and 638 evaluation videos with valid V, A, and S; a fixed random soundtrack crop of length L/20 serves as the audio query.A2V ranks the 638 videos, while V2A ranks their associated audio crops.Source: §A.3 · p. 10
• MSR-SDrop removes S from the same proportion of training and test samples at rates from 0% to 100%, while retaining V and A and keeping samples in the pools.VAS and VSA are separately trained and evaluated at each rate; audio query crops and candidate identities stay fixed, and each configuration uses one run.Source: §A.3 · p. 11

A.4 MSR-Frame

• MSR-Frame uses one fixed random source frame per video as an image query, with candidates retaining their natural A and S availability.It evaluates source-frame matching; repeated-V schedules use three learned gated stages, while VAS uses two.Source: §A.4 · p. 11

B Further Analysis Results

• The appendix retains complete numerical results for plotted order comparisons, while natural modality groups and fixed-checkpoint input removal are reported in main-paper Tables 7 and 6.Table 8 complements Figure 3.Source: B Further Analysis Results · p. 11, §B.1 · p. 11

B.1 Fusion Organization and Complete Inputs

• Tables 8–10 cover fusion organization and order on MSR-VTT-9k and audio–video retrieval in both directions on CM-Pool and MSR-Complete.The CM-Pool and MSR-Complete constructions use different source datasets.Source: §B.1 · p. 11, Table 8 · p. 11, Table 9 · p. 11

B.2 Controlled Speech Removal

• Tables 11 and 12 provide the values underlying the controlled speech-removal comparisons, with training and testing missingness matched at each rate.Audio crops and candidate identities remain fixed across rates; Appendix A.3 gives the protocol.Source: §B.2 · p. 11, Table 11 · p. 12, Table 12 · p. 12

B.3 Image Retrieval

• 5.5 and 13.3 points: VASV exceeds the listed CLIP4Clip and ViCLIP I2V R@1 values by these amounts.Table 13 reports the available image-retrieval metrics; reverse scores remain unspecified.Source: Table 13 · p. 12, §B.3 · p. 12