PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

Jingxue Huang, Xiaosong Li, Yi Lin

arXiv:2610.34765v1cs.CV

TL;DR

Cross-frame TTT-based sequence modeling for multi-modal video fusion remains underexplored, while flow alignment is costly and sensitive to estimation accuracy. This paper combines dynamic-region temporal interaction with MAE-based temporal consistency and reports superior fusion quality and higher efficiency on an edge device.

  • Problem

    Cross-frame use of TTT sequence modeling for multi-modal video fusion remains underexplored, despite the need for efficient temporal-dependency modeling.

    Source: §I · p. 1

  • Method

    The method uses linear-complexity TTT interaction over dynamic regions and an MAE-based constraint aligning fused and source-video temporal variations.

    Source: §I · p. 2, C. Loss Function · p. 6

  • Results

    The method achieves superior fusion quality on multiple benchmarks and higher computational efficiency than compared methods on Jetson AGX Orin.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors report high-quality fusion on multiple public datasets and higher inference efficiency when evaluated on an edge device.

    Source: §VI · p. 11

    Online inference buffers fixed-length clips before joint processing, adding latency.

    Source: §V · p. 11

Abstract

from arXiv · show

Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.

I. INTRODUCTION

The introduction motivates efficient multi-modal video fusion by the complementary strengths of visible and infrared sensors and the cost and error sensitivity of flow-based temporal alignment. It presents dynamic-region-guided TTT interaction, MAE-based temporal consistency, and FD3; the authors report superior fusion quality and higher computational efficiency on an edge device.

I. INTRODUCTION

  • Visible sensors preserve texture, color, and structure but are vulnerable to poor illumination and weather, whereas infrared sensors perceive thermal targets but often lack texture and spatial resolution.Source: §I · p. 1
  • Flow-based alignment is computationally costly and sensitive to optical-flow errors, while global spatio-temporal attention has quadratic complexity that hinders scaling to long sequences.Flow errors can accumulate over long sequences, limiting practical deployment on resource-constrained edge devices.Source: §I · p. 1
  • STIM uses linear-complexity TTT temporal modeling focused on dynamic regions, while an MAE-based feature-level constraint aligns fused and source-video variations to suppress flickering.Dynamic regions include moving targets, changing boundaries, and areas with distinct cross-modal response differences; static backgrounds provide limited temporal cues.Source: §I · p. 2
  • FD3 evaluates the directional distance of temporal variations within dynamic regions in the pretrained MAE feature space.Source: §I · p. 2
  • The method achieves superior fusion quality on multiple benchmark datasets and higher computational efficiency than compared methods on Jetson AGX Orin.The reported metrics in Fig. 1 are evaluated on VTMOT, but the supplied caption does not specify their values.Source: §I · p. 2

A. Multi-modal Image Fusion

B. Multi-modal Video Fusion

Existing multi-modal video fusion methods use temporal interaction or optical-flow warping, which can be computationally costly. The proposed framework combines efficient sequence modeling with dynamic-region-guided interaction and spatial refinement.

B. Multi-modal Video Fusion

  • Prior video-fusion methods use heavy temporal aggregation or optical-flow warping, which can incur computational overhead and inference latency for edge deployment,,.Heavy aggregation may also over-couple features under complex motion or large temporal variations, causing detail loss and inter-frame artifacts.Source: B. Multi-modal Video Fusion · p. 3

C. Efficient Visual Sequence Modeling

  • Test-Time Training adapts a compact nonlinear inner model online from key–value pairs, enabling adaptive token–context interactions with O(N) complexity.This offers a linear-complexity alternative to quadratic Softmax attention while retaining nonlinear relation modeling.Source: C. Efficient Visual Sequence Modeling · p. 3

A. Overall Framework

  • The framework extracts and enhances shallow features, models intra-frame context with VITTT and inter-frame dependencies with STIM, then fuses and reconstructs the features.TokenFusion integrates selected VITTT and STIM block outputs; the decoder also uses enhanced shallow features through a residual connection.Source: A. Overall Framework · p. 3

B. Spatio-Temporal Interaction Module

  • STIM selects dynamic patches for sparse temporal interaction while refining spatial features over the full grid in a parallel dense branch.The selected-patch count is K = max(1, round(ρN)); the temporal branch uses TTT-based adaptation, and the dense branch uses a convolutional inner model.Source: B. Spatio-Temporal Interaction Module · pp. 4–5

C. Loss Function

The loss combines pixel-level intensity and gradient supervision with feature-level temporal constraints. Frozen MAE features provide structural guidance and are used to align fused-video temporal variations with those of the source videos.

Pixel-level Supervision

  • Pixel-level supervision combines intensity and gradient constraints to preserve frame-wise intensity and structural information.The gradient term uses the Sobel operator, and the loss terms are weighted by hyperparameters.Source: C. Loss Function · p. 5

Feature-level Supervision

  • Frozen pretrained MAE features provide a shared representation space for aligning fused-video temporal transition directions with those of both source videos.The paper motivates these features as responsive to structural patterns, object boundaries, and regional content.Source: C. Loss Function · pp. 5–6
  • Time-reversal consistency regularizes the fused sequence by requiring its restored reversed-input feature variations to match those of the original fusion.The stated aim is to suppress temporal-direction bias and promote more stable temporal modeling.Source: C. Loss Function · p. 6
  • A distillation loss aligns projected features from all three VITTT blocks for each modality with the final MAE output features.The projection heads are modality- and layer-specific.Source: C. Loss Function · p. 6

A. Experiment Settings

Experiments use three datasets for training and testing, with a fourth reserved for generalization testing. Evaluation combines image-quality and temporal metrics, including a feature-level measure of dynamic-region variation.

A. Experiment Settings

  • HDO, M3SVD, and VTUAV provide training and test sequences, while eight VTMOT sequences are reserved for generalization testing.Training uses 22 HDO, 50 M3SVD, and 38 VTUAV sequences; testing uses five, eight, and eight sequences, respectively.Source: A. Experiment Settings · p. 6
  • Training runs for 50,000 iterations in two stages, increasing clips from five 256 × 256 frames to seven 384 × 384 frames.The batch size decreases from five to three in the second stage.Source: A. Experiment Settings · p. 6
  • Evaluation covers image quality with QG, QM, FSIM, and VIF, and temporal quality with flowD and feaCD; FD3 measures feature-variation direction differences.FD3 is lower when fused and source sequences have smaller directional discrepancies in region-level feature variations.Source: A. Experiment Settings · pp. 6–7

B. Multi-modal Video Fusion Experiment

Across four datasets, the proposed method leads most image-quality metrics and all-dataset feaCD and FD3 rankings. It also shows strong qualitative fusion, temporal stability, downstream segmentation, and edge-device efficiency.

B. Multi-modal Video Fusion Experiment

C. Analysis of the MAE Feature Space

The analysis finds that MAE features characterize structural- and region-level video variations more effectively than the compared pretrained feature spaces.

C. Analysis of the MAE Feature Space

  • MAE has the highest spatial consistency with pixel-level responses qualitatively and the highest IoU across the tested ρ settings.The comparisons concern high-response regions and support MAE’s characterization of structural- and region-level video variations.Source: C. Analysis of the MAE Feature Space · p. 9

D. Analysis of the Hyperparameter ρ

Dynamic-region guidance improves objective performance over full-region temporal modeling, and ρ = 0.25 performs best overall among the tested ratios.

D. Analysis of the Hyperparameter ρ

  • Dynamic-region-guided variants outperform full-region temporal modeling (ρ = 1.00) on objective evaluation, with ρ = 0.25 best overall.The analysis attributes the improvement to focusing STIM interactions on tokens with more informative variations.Source: D. Analysis of the Hyperparameter ρ · p. 9

E. Analysis of the FD3 Metric

The section introduces a stress benchmark for temporal metrics and presents comparisons of FD3 sensitivity across degradation types.

E. Analysis of the FD3 Metric

  • The stress benchmark evaluates temporal metrics using three types of perturbations, including temporal misalignment and local content lag.Source: E. Analysis of the FD3 Metric · p. 9
  • Table III reports quantitative comparisons using different ρ values on the HDO dataset.Source: Table III · p. 10
  • Table IV compares the sensitivity of temporal metrics under different degradation types.Source: Table IV · p. 10

THE MOST DESIRABLE RESPONSE IN EACH ROW, JOINTLY CONSIDERING

The FD3 stress-benchmark results compare metric responses to temporal misalignment, local content lag, and temporal smoothing. FD3 shows stronger severity-sensitive and monotonic responses across these perturbations than the compared metrics.

THE MOST DESIRABLE RESPONSE IN EACH ROW, JOINTLY CONSIDERING

  • FD3’s relative increase under temporal misalignment rises from 31.1% to 53.6% as offset severity increases, while feaCD’s response falls from 19.0% to 16.0%.Source: THE MOST DESIRABLE RESPONSE IN EACH… · p. 10
  • FD3’s relative increase under local content lag rises from 24.2% to 34.0% as the affected-region ratio grows from 0.4 to 0.6.flowD and feaCD vary only marginally with affected-region area.Source: THE MOST DESIRABLE RESPONSE IN EACH… · p. 10
  • FD3’s relative increase under temporal smoothing rises from 12.5% to 26.1%, while flowD assigns more severe oversmoothing a better temporal-quality score.feaCD’s response rises from 12.1% to 15.2% and gradually saturates.Source: THE MOST DESIRABLE RESPONSE IN EACH… · p. 10
  • MAE-based FD3 shows the largest relative changes across the tested feature extractors and a monotonic decline from 34% to 24% as the affected-region ratio decreases.The comparison uses CLIP, DINO, and MAE feature extractors under local content lag.Source: THE MOST DESIRABLE RESPONSE IN EACH… · p. 10, Table V · p. 10
  • Across the three perturbations, FD3 provides higher sensitivity, more reliable response directions, and clearer monotonic trends than the compared temporal metrics.The authors present it as an interpretable diagnostic perspective aligned with multi-modal video fusion, not as a simple replacement for existing metrics.Source: THE MOST DESIRABLE RESPONSE IN EACH… · pp. 10–11

F. Ablation Experiment

The ablations report that STIM, the temporal constraints, MAE features, and Lrev contribute to the proposed configuration’s performance. The paper also notes a sequence-buffering latency limitation and summarizes the method’s results and FD3 metric.

F. Ablation Experiment

  • Replacing STIM with ViT slightly degrades quantitative performance, while ViT’s computational cost rises rapidly with more frames or higher spatial resolution.STIM’s linear-complexity sequence modeling maintains relatively stable efficiency across input scales.Source: F. Ablation Experiment · p. 11
  • Removing STIM or both temporal constraints performs considerably worse than the proposed model across the overall metrics.The visual comparisons also show blurring and abrupt dark variations associated with temporal flickering in those variants.Source: F. Ablation Experiment · p. 11, Figure 10 · p. 11
  • CLIP- and DINO-based temporal-loss variants lead on a few individual metrics, but their overall performance remains below the MAE-based configuration.Source: F. Ablation Experiment · p. 11
  • Adding Lrev yields consistent improvements across the evaluation metrics compared with the variant without Lrev.Source: F. Ablation Experiment · p. 11

V. LIMITATION AND FUTURE WORK

  • Sequence-based inference must buffer fixed-length clips before joint processing, adding latency; future work proposes streaming fusion with temporal state propagation.Source: §V · p. 11

VI. CONCLUSION

  • The conclusion describes a TTT-based method for temporal interaction with dynamic regions, MAE-feature temporal consistency constraints, high-quality fusion on multiple public datasets, and higher edge-device inference efficiency.It also introduces FD3 to measure the directional distance of temporal variations within dynamic regions in pretrained MAE feature space.Source: §VI · p. 11
Loading 2610.34765v1…