PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion

Meiqi Gong, Hao Zhang, Jiayi Ma

arXiv:2610.39322v1cs.CV

TL;DR

Existing multimodal video fusion methods underuse long-term temporal context, while inconsistent representations can produce flickering. MemoryFusion combines dual memory banks with progressive temporal smoothing and refinement, and reports superior fusion quality and temporal consistency over existing frame-based and video-based methods.

  • Problem

    Existing video fusion methods focus mainly on short-range frame correlations, leaving long-term temporal context underused.

    Source: §1 · p. 2

  • Method

    MemoryFusion uses short- and long-term memory banks for local and historical cues, followed by coarse temporal smoothing and fine temporal refinement.

    Source: §1 · p. 3

  • Results

    MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MemoryFusion jointly targets visual quality and temporal consistency in multimodal video fusion.

    Source: §1 · p. 3

    Robustness may decrease under severely unseen degradations because the method is not degradation-specific and depends partly on training-data degradation coverage.

    Source: §A.4 · p. 14

Abstract

from arXiv · show

Multimodal videos contain rich temporal cues that can facilitate faithful fusion. However, existing video fusion methods mainly rely on short-range temporal interactions, leaving valuable long-term information underexplored. Moreover, inconsistent fusion representations can cause flickering artifacts, undermining temporal coherence. To address these coupled challenges, we propose MemoryFusion, a two-stage framework integrating cross-temporal memory learning with progressive temporal refinement. Considering the complementary roles of adjacent and distant frames, the first stage combines a Short-Term Memory Bank for local temporal guidance with a Long-Term Memory Bank that preserves reliable historical information through a fusion-oriented update strategy. To mitigate temporal fluctuations, a Temporal Smoothing Module leverages soft motion cues for coarse background stabilization. Building upon the resulting fusion prior, The second stage further employs a lightweight 3D residual Temporal Refinement Module to correct remaining temporal inconsistencies while enhancing structural details. Extensive experiments demonstrate that MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based fusion methods. The source code is publicly available at https://github.com/Meiqi-Gong/MemoryFusion.

1 Introduction

Multimodal fusion combines complementary information, but frame-based methods do not model inter-frame dependencies and existing video methods underuse long-term context. MemoryFusion addresses these gaps with dual temporal memory and progressive smoothing and refinement.

  • Multimodal image fusion integrates complementary information from different modalities to produce more comprehensive and reliable representations.Source: §1 · p. 1
  • Existing video fusion methods mainly use short-range correlations, while frame-based methods cannot effectively model inter-frame dependencies and may produce temporal flickering.Existing consistency modeling also relies on complex networks with simple temporal constraints and no temporal or spatial prior.Source: §1 · p. 2
  • MemoryFusion is a two-stage approach that uses cross-temporal learning to address underused temporal context and inadequate temporal consistency in multimodal video fusion.Source: §1 · pp. 2–3
  • The Short-Term Memory Bank guides fusion with recent cues, while the Long-Term Memory Bank preserves diverse, reliable historical information through redundancy- and fusion-quality-aware updates.The banks provide local and global temporal guidance, respectively.Source: §1 · p. 3
  • The Temporal Smoothing Module mitigates coarse background transitions, and the Temporal Refinement Module enforces fine consistency while refining details.The first-stage module selectively preserves preceding-frame background information; the second uses a 3D convolutional network to model temporal constraints.Source: §1 · p. 3
  • MemoryFusion achieves superior fusion quality and temporal consistency over existing frame-based and video-based methods.Source: §1 · p. 3

2 Method

MemoryFusion uses a two-stage pipeline: memory-guided fusion with coarse temporal consistency modeling, followed by refinement based on the first-stage fusion prior. The stages combine multi-scale feature processing and temporal information to produce fused frames and improve video-level consistency.

2.1 Notation and Overview

  • The first stage extracts hierarchical visible and infrared features, fuses them, retrieves memory, applies temporal smoothing, and decodes memory-enhanced features into fused frames.After the first frame, the same processing pipeline uses previous-frame features; the second stage processes the fused sequence together with both original input sequences.Source: §2.1 · pp. 3–4

2.2 Memory Banks

  • STMB retrieves recent-frame cues, while LTMB retrieves representative distant-frame cues and prunes entries based on redundancy and fusion quality.STMB refreshes every frame; LTMB fills to its capacity before selective updates, retaining entries that are unique and beneficial to fusion quality.Source: §2.2 · pp. 4–5

2.3 Temporal Smoothing Module (TSM)

  • TSM estimates soft motion from visible-frame feature differences and adaptively balances current and previous features across motion and static regions.The visible features are used for motion estimation because infrared noise can be mistaken for motion; the smoothed feature serves as the previous-frame feature in the next step.Source: §2.3 · pp. 5–6

2.4 Temporal Refinement Module (TRM)

  • TRM uses stacked 3D separable convolutions and residual learning to refine temporal consistency from the fused video and original visible and infrared sequences.Its objective combines gradient loss for fine spatial textures with temporal loss for inter-frame consistency.Source: §2.4 · p. 6

3 Experiments

Experiments evaluate MemoryFusion on two registered infrared-visible video datasets, with training on M3SVD and testing on both. The results report strong fusion quality and temporal consistency, component-specific ablation findings, plug-and-play gains from TRM, and moderate computational cost.

3.1 Implementation Details

  • Experiments use pre-registered M3SVD and HDO video pairs, training on M3SVD and testing on both with four fusion-quality and two temporal-consistency metrics.Comparisons include five image-fusion and three video-fusion methods; selected methods receive retrained deblurring and denoising preprocessing for fair comparison.Source: §3.1 · p. 6

3.2 Video Fusion

  • MemoryFusion achieves the best reported image-quality and temporal-consistency metrics on M3SVD, with superior fusion results also reported across all metrics on degraded HDO.Qualitatively, it restores detail and contrast; its first stage already surpasses image-based methods, while the second stage restores temporal coherence and further improves image quality.Source: Table 1 · p. 7, §3.2 · pp. 7–8

3.3 Ablation Studies

  • Removing TSM has little effect on image quality but produces temporal flickering, indicating its main contribution is inter-frame stability.The ablation study evaluates most variants against the stage-1 baseline; only the two second-stage variants are compared with complete MemoryFusion.Source: §3.3 · p. 8
  • Without STMB, the model cannot effectively reuse recent-frame cues, and ablation results show blurred textures and weakened local contrast.The reported findings associate recent temporal information with preserving local structures and fine textures.Source: §3.3 · pp. 8–9
  • Removing LTMB limits fusion to short-range cues and degrades structural integrity and detail, especially in complex or spatially diverse regions.Random LTMB replacement also destabilizes global guidance, indicating that its benefit depends on selective updating as well as memory storage.Source: §3.3 · p. 9

3.4 Plug-and-Play Spatiotemporal Refinement with TRM

  • Applied to fixed MMIF-EMMA and UniVF outputs, TRM reduces flowD by 36.6% and 30.4%, respectively, while improving CC and SCD for both methods.Other fusion metrics remain comparable, and the reported gains do not require additional training or finetuning.Source: §3.4 · pp. 9–10

3.5 Computation Efficiency Comparison

  • MemoryFusion's computational cost remains moderate: its first stage is competitive with video-fusion methods, and its lightweight second stage adds negligible overhead.The comparison reports fusion-network costs for preprocessing-based methods, excluding their additional pretrained deblurring or denoising modules.Source: §3.5 · p. 10, Table 4 · p. 10

4 Conclusion

MemoryFusion combines historical-frame memory and temporal smoothing in its first stage, then applies lightweight residual refinement. Experiments on multiple public datasets report state-of-the-art fusion quality and temporal consistency, with generalization across diverse scenes.

4 Conclusion

  • MemoryFusion uses dynamically updated dual memory banks to retrieve historical cues for current-frame fusion.A Temporal Smoothing Module mitigates coarse frame variations to establish a stable temporal foundation.Source: §4 · p. 10
  • A lightweight Temporal Refinement Module uses the first-stage fusion prior for fine-grained temporal coherence and spatial-fidelity refinement.Source: §4 · p. 10
  • Experiments on multiple public datasets report state-of-the-art fusion quality and temporal consistency, alongside generalization across diverse scenes.Source: §4 · p. 10

A.1 Experimental Details

MemoryFusion takes raw infrared-visible video sequences as input without external de-degradation preprocessing. Its reported efficiency comparison is favorable to the baselines because their preprocessing costs are excluded.

A.1 Experimental Details

  • Baseline efficiency figures exclude DSTNet/MDIVDNet preprocessing costs, whereas MemoryFusion uses raw sequences without external de-degradation preprocessing.This makes the efficiency comparison favorable to the baselines, while MemoryFusion remains competitive without external preprocessing.Source: §A.1 · p. 13

A.2 Extension to Medical Scenes

The medical-sequence evaluation treats consecutive MRI or PET slices as video frames because adjacent slices have strong anatomical correlation. MemoryFusion's qualitative results preserve structural edges and texture while retaining PET intensity patterns.

A.2 Extension to Medical Scenes

  • The evaluation uses MRI and PET sequences from the Harvard medical imaging dataset, treating consecutive slices as video frames because of their anatomical correlation.The experiment focuses primarily on fusion performance because medical scenes are relatively homogeneous.Source: §A.2 · p. 13
  • Compared methods show blurred boundaries, over-smoothed details, structural-metabolic inconsistencies, or PET color distortion in the reported examples.Source: §A.2 · p. 13
  • MemoryFusion combines MRI structural information with PET metabolic patterns, preserving sharp anatomical edges, gray-matter texture, and PET intensity distributions in the reported results.Source: §A.2 · p. 13

A.3 Results of Non-degraded Scenes

In non-degraded scenes, MemoryFusion produces clearer human boundaries and sharper, more consistent structures, while some methods over-enhance, over-smooth, or distort details.

  • MemoryFusion incorporates infrared thermal cues for clearer human boundaries while preserving texture continuity.Source: §A.3 · p. 13
  • MemoryFusion delivers sharper, more consistent structures with higher contrast around the window region.Source: §A.3 · p. 14
  • The results indicate preserved global scene semantics and improved fine-detail reconstruction even on non-degraded scenes, where other methods may over-smooth or distort details.Source: §A.3 · p. 14
  • In non-degraded scenes, MemoryFusion preserves clearer human boundaries and sharper, more consistent window structures, while TemCoCo and VideoFusion show boundary artifacts from over-enhancement.Source: §A.3 · pp. 13–14

A.4 Limitation

MemoryFusion is not designed specifically for degradation restoration, and its robustness may decrease under severely unseen degradations.

  • MemoryFusion is not explicitly designed as a degradation-specific restoration framework, so robustness partly depends on training-data degradation coverage and may decrease for severely unseen degradations.Source: §A.4 · p. 14
  • When motion is mainly visible in infrared frames, TSM's visible-frame-difference cue may provide limited smoothing guidance, placing greater pressure on the second-stage refinement module.Source: §A.4 · p. 14
  • The authors identify degradation-aware modeling and reliability-aware bi-modal motion estimation as possible future additions.Source: §A.4 · p. 14
Loading 2610.39322v1…