PaperlayerView exact revision on arXiv

Source-linked AI summary

From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

Yuchen Xian, Yunqiu Xu, Yang He, Yi Yang

arXiv:2606.12303v1cs.CV

TL;DR

Multimodal image fusion relies on 2D shared feature grids that model local structure well but provide limited control over global appearance. This paper introduces a compact 1D token interface with selective token editing alongside the 2D pathway, achieving state-of-the-art performance with consistent gains in global coherence and local fidelity.

  • Problem

    Existing 2D-centric shared representations entangle image-level appearance factors with spatial details, limiting stable and compact modeling of global appearance.

  • Method

    The framework uses a frozen pretrained 1D tokenizer as a global appearance interface and selectively edits a small subset of appearance-sensitive tokens while retaining 2D local-detail modeling.

  • Results

    Across four IVIF benchmarks and downstream applications, the method achieves state-of-the-art performance with consistent gains over prior methods.

  • Takeaways & Limitations

    A compact 1D token space can complement 2D fusion by improving global appearance coherence while preserving local structural details.

  • Takeaways & Limitations

    STE depends on tokenizer-specific appearance factors and selected token positions, so new tokenizer configurations may require task-specific probing.

Abstract

from arXiv · show

Multimodal image fusion aims to integrate complementary information from different modalities into a fused image that preserves rich local details while maintaining globally consistent appearance. Existing approaches build shared representations on 2D feature grids, which excel at modeling local structures but offer limited leverage over image-level global appearance factors. To balance these objectives, we introduce a compact 1D token interface based on a frozen pretrained image tokenizer for modeling non-local appearance/base factors. Rather than using the tokenizer as a reconstruction backbone, our design uses the 1D token space as a global carrier while retaining the 2D spatial pathway for local structure restoration. Specifically, we introduce Selective Token Editing (STE), which sparsely updates/replaces a small set of critical tokens, providing a lightweight mechanism to steer global appearance coherence while keeping the fusion backbone unchanged and avoiding extra losses. Experiments on four commonly used benchmarks show that our method achieves the best overall performance, with consistent, multi-metric improvements in both global coherence and local fidelity. Project page: https://zju-xyc.github.io/1D-Fusion-Project-Page/

1. Introduction

The paper argues that dense 2D shared grids entangle global appearance with local details, limiting controllable appearance alignment in multimodal image fusion. It introduces a hybrid interface using frozen 1D tokens for global appearance guidance and a 2D pathway for local structure, with selective token editing improving fusion quality without complex losses.

  • Motivation: Dense 2D shared grids spatially entangle global appearance factors with local textures, modality-specific cues, and residual noise, hindering appearance alignment.This mismatch can produce brightness inconsistency, blurred details, or amplified artifacts.
  • Method: The proposed framework uses a compact 1D token space as a non-spatial carrier for global appearance while retaining 2D fusion for localized textures, edges, and structural details.Tokens are mapped back into the spatial domain through a lightweight token-to-map interface and injected as global guidance.
  • Method: A frozen pretrained 1D tokenizer provides a controllable global interface that complements the conventional 2D fusion backbone.The tokenizer is used for appearance/base representation rather than as a universally superior reconstruction backbone or strongest semantic encoder.
  • Results: Experiments across multiple fusion benchmarks and downstream applications show consistent improvements in illumination consistency, detail sharpness, and artifact reduction.The evaluation covers infrared-visible and medical image fusion, object detection, and semantic segmentation.
  • Method: Selective token editing updates only a small subset of appearance-sensitive token dimensions, improving fusion quality without introducing complex loss designs.This intervention targets illumination, contrast, and perceptual tone while preserving local detail modeling.

2. Related Work

Prior work primarily relies on dense 2D representations for image reconstruction and multimodal fusion, which favor local structures but make global appearance control costly or limited. This work instead explores compact 1D shared spaces with selective token editing for controllable global appearance regulation alongside standard 2D fusion.

  • Compressed Tokenizers: 2D-grid image tokenizers quantize VAE-style latents and decode images pixel-wise, but their locality bias and many tokens make global appearance editing expensive.Examples include VQ-VAE, VQGAN, and related latent-tokenizer approaches.
  • Multimodal Image Fusion: Early multimodal image fusion methods operate on dense 2D feature maps with CNN-style encoders and decoders to preserve local textures.Later methods add transformer-style interactions or auxiliary guidance to capture long-range dependencies, while fusion remains based on dense 2D representations.
  • Multimodal Image Fusion: Compact 1D shared-space fusion remains less explored but offers global appearance control with few tokens while remaining compatible with standard 2D fusion.The proposed method couples a 1D token-based shared representation with Selective Token Editing for controllable global appearance regulation.

3. Rethinking 2D Shared Representations for Multimodal Image Fusion

Existing fusion pipelines encode aligned modalities into dense 2D shared feature grids, where fusion is performed spatially before decoding. This representation suits local detail but structurally mismatches image-level base factors, making global estimation and adjustment difficult.

  • Existing 2D Fusion Pipeline: Encoder–fusion–decoder methods map visible and infrared inputs into shared dense 2D feature grids, aggregate them in grid space, and reconstruct the fused image.The shared representation is explicitly parameterized as a dense 2D grid, with interactions primarily location-wise or neighborhood-wise.
  • Base–Detail Mismatch: Base captures image-level brightness, contrast, and global tone, whereas detail consists of spatially varying high-frequency structures tied to specific locations.Base is a low-dimensional factor shared across the image and is not naturally associated with any coordinate.
  • Base–Detail Mismatch: In 2D grids, base-related information is distributed implicitly across multiple spatial positions rather than represented as a compact independent variable.This creates a structural mismatch when fusion must preserve detail while aligning or adjusting base.
  • Latent-Factor Analysis: A latent-factor abstraction shows that 2D features entangle base with detail and location-dependent residuals through spatial broadcasting.The shared operator maps the base factor into feature space and broadcasts it across spatial locations, while local encoding captures detail.
  • Consequences: Distributed base parameterization makes global estimation statistically inefficient and global adjustment ill-conditioned because both require coordinated processing across high-dimensional spatial fields.Base estimation is sensitive to location-dependent residuals, while adjustment induces a many-to-one inverse problem.

4. Methodology

The method replaces the conventional shared 2D feature grid with a compact 1D token interface for global appearance/base modeling while retaining a 2D pathway for local detail reconstruction. It combines Selective Token Editing, token-to-map adaptation, factorized base/detail fusion, residual decoding, and two-stage training with a frozen tokenizer.

  • Shared 1D Representation: The framework uses a compact 1D token sequence as the shared representation carrier, confining the redesign to the representation layer while preserving 2D backbones for detail modeling.The 1D space provides a non-spatial carrier for accessing and regulating global factors with fewer degrees of freedom.
  • Selective Token Editing: Selective Token Editing sparsely modifies token entries to regulate appearance while preserving core semantics and compatibility with arbitrary 2D fusion backbones.Under TiTok-32, configuration-specific positions 12 and 18 are edited through channels {6, 7, 8} using a learnable bias.
  • Token-to-Map Interface: A token-to-map interface adapts tokens into 2D feature maps so global semantics remain concentrated in token space while local processing uses a spatial substrate.The hierarchical mapping lifts token dimensions from 12 to 64, forms a 32×32 coarse map, and upsamples it to 256×256 with residual local aggregation and multiscale detail features.
  • Factorized Fusion: Private encoders decompose each modality into base and detail subspaces, enabling separate fusion that decouples base alignment from detail preservation.Base captures low-frequency globally consistent appearance, whereas detail preserves high-frequency spatially localized structures.
  • Two-Stage Training: Two-stage training stabilizes factorization through intra-modality reconstruction before fusion, while the pretrained tokenizer remains frozen throughout optimization.Freezing prevents representation drift and keeps base consistently carried by the compact token space.

5. Experiments

Experiments across IVIF and MIF benchmarks show that the method consistently improves fusion quality, downstream recognition, and global–local balance. Ablations further support sparse joint token editing, compact token sequences, and a lightweight frozen-tokenizer design.

  • Setup and Metrics: Evaluation covers IVIF and MIF tasks, using three IVIF fusion benchmarks plus object detection and semantic segmentation downstream evaluations.IVIF training uses MSRS, with M3FD, RoadScene, and TNO for fusion evaluation; MIF uses Harvard Medical Image Dataset.
  • Quantitative Comparison: The method achieves the best or second-best fusion performance across all three IVIF benchmarks, improving information preservation, appearance coordination, cross-modal integration, and structural fidelity.Metrics include EN, SD, SCD, EI, SF, AG, and SSIM.
  • Downstream Applications: The method obtains the highest mAP50:95 on M3FD and the best mIoU on FMB among all fusion methods.The downstream evaluations use YOLOv8s for object detection and SegFormer-B1 for semantic segmentation.
  • Ablation on Token Position: Jointly editing token positions 12 and 18 achieves the best overall balance, whereas editing either position alone causes ghosting effects.Position 12 mainly improves edge-related quality, while position 18 contributes more to appearance smoothing; their effects are complementary.
  • Ablation on Token Numbers: The 32-token TiTok variant achieves the best performance because its global semantics are more concentrated, making specific-token adjustment more effective.The comparison includes 32, 64, and 128 latent tokens.
  • Efficiency and Representation Choice: Only 1.325M parameters are trainable, while the method reduces FLOPs and latency versus Text-DiFuse and outperforms recognition-oriented DINOv3 and CLIP interfaces overall.The total parameter count is larger because the introduced tokenizer is frozen; TiTok performs more strongly across M3FD, RoadScene, TNO, and MIF.

6. Conclusion

The framework replaces dense 2D shared feature grids with a compact 1D token space to improve global semantic coherence while preserving local structural details. Targeted token modification further improves fused image quality, yielding consistent gains and state-of-the-art performance.

  • Conclusion: The framework replaces dense 2D shared feature grids with a compact 1D token space built from a pretrained tokenizer.This design enhances global semantic coherence while preserving local structural details.
  • Conclusion: Targeted token modification further improves the quality of fused images.
  • Conclusion: Extensive experiments on IVIF benchmarks and downstream applications show consistent gains over prior methods and state-of-the-art performance.

Impact Statement

The work improves global appearance consistency while preserving local structures in multimodal image fusion. Better fusion may benefit perception with complementary sensors in adverse conditions, though the passage notes a potential downside without specifying it.

  • Technical impact: The proposed 1D token-based shared representation and selective token editing improve global appearance consistency while preserving local structures.The passage gives illumination and sharpness as examples of global appearance factors.
  • Applications: Improved fusion can benefit nighttime driving, robotics, remote sensing, and medical imaging by increasing robustness under adverse conditions.These applications use complementary sensors whose information can be combined through fusion.
  • Limitations: The passage flags a potential downside of improved fusion quality but does not specify its nature.

A. Additional Theoretical Analysis … A.3. Appearance Sensitivity vs. Semantic Invariance

The analysis motivates compact 1D tokens as a controllable carrier for non-local appearance factors, complementing a 2D pathway that preserves local structures. It contrasts this design with broadcasted 2D factors and recognition-oriented invariant features, which are less direct or less suitable for appearance-sensitive multimodal fusion.

  • A. Additional Theoretical Analysis: Compact 1D tokens provide a controllable interface for non-local appearance factors rather than universally replacing reconstruction backbones.The analysis specifically frames the 1D space as an appearance/base carrier, not as a universally superior reconstruction backbone.
  • A.1. Broadcasted Base Factors in 2D Grids: 2D shared features broadcast global appearance factors across spatial positions, mixing illumination, contrast, and tone with local structures.This representation does not encode global appearance factors as independent variables.
  • A.1. Broadcasted Base Factors in 2D Grids: Recovering a global base factor from a high-dimensional spatial field becomes sensitive to noise, high-frequency edges, and modality-specific artifacts when residuals are structured or correlated.The aggregation process must extract a low-dimensional factor from spatially distributed features.
  • A.1. Broadcasted Base Factors in 2D Grids: 2D fusion backbones preserve local detail effectively but regulate global appearance less directly because many spatial positions must coordinate around one image-level factor.This coordination can make appearance regulation unstable.
  • A.2. 1D Tokens as a Compact Control Interface: A compact 1D token representation places global information into fewer token variables, enabling sparse token editing to influence appearance without modifying every spatial location.The implementation uses K = 32 tokens with dimension C = 12.
  • A.2. 1D Tokens as a Compact Control Interface: The spatial pathway restores local structures while the token branch supplies global appearance guidance, distinguishing this interface from dense grids and recognition-oriented semantic encoders.Table 8 summarizes the functional roles of these representation forms for multimodal image fusion.
  • A.3. Appearance Sensitivity vs. Semantic Invariance: Recognition-oriented invariance benefits recognition, retrieval, and classification, but fusion must remain sensitive to illumination, contrast, thermal saliency, and modality-specific structures.These low-level appearance changes are information to preserve or regulate rather than nuisance factors.

B. Detailed Architecture and Implementation … D.3. Implication for Fusion

The framework uses frozen compact 1D tokens for global appearance guidance while trainable 2D modules restore local structure. Extended analyses show that two effect-based slots balance sharpening and background smoothing, while STE requires re-identification across tokenizer configurations and remains useful despite domain-dependent reconstruction quality.

  • B. Detailed Architecture and Implementation; B.1. Module-Level Design: The frozen tokenizer supplies compact 1D tokens, while trainable modules adapt them into the 2D fusion pipeline through token-to-map conversion, factorized base/detail fusion, and residual reconstruction.The tokenizer is frozen, and the remaining trainable modules connect token representations to spatial fusion and reconstruction.
  • B.2. Residual Reconstruction: Residual reconstruction predicts ∆I and adds it to the reference input, reducing decoder burden and focusing learning on complementary corrections.The decoder avoids generating the fused image entirely from scratch.
  • B.3. Implementation Notes: During training and evaluation, inputs are resized to 256 × 256, while only the token-to-map interface, factorization modules, fusion layers, and decoder are optimized.Keeping the tokenizer frozen prevents drift in the pretrained token space and stabilizes appearance control.
  • C. Extended Token Editing Evidence; C.1. Full Selector Distribution: With two slots, the final setting, the selector identifies complementary positions; additional slots mainly absorb weaker residual effects, indicating diminishing returns beyond two.The first two slots capture the dominant sparse manipulation structure.
  • C.2. Slot-Wise Visual Interpretation: Slot 0 mainly sharpens local edges and contours, Slot 1 mainly suppresses background residue, and jointly editing both improves the balance between local detail and global coherence.The two slots therefore exhibit complementary observed effects rather than identical editing behavior.
  • C.3. Interpretation of the Selected Slots: The selected slots are empirical effect-based slots under TiTok-32, not universal semantic labels, and their fixed indices should not be treated as tokenizer-independent meanings.STE transfers through selection and editing, not through fixed positions or token identities.
  • C.4. Generalization of STE: When the tokenizer, token count, or token dimension changes, applying STE requires rerunning lightweight selection or probing to re-identify sensitive positions.The transferable component is the selection-and-editing mechanism rather than fixed slot indices.
  • D. Tokenizer Domain Gap and Robustness; D.1. Reconstruction Quality on Infrared and Medical Images; D.2. Alternative Tokenizers; D.3. Implication for Fusion: The frozen tokenizer reconstructs infrared images with high SSIM but shows lower SSIM for medical images, while alternative compact tokenizers remain usable despite varying reconstruction behavior.Fusion does not require standalone high-fidelity tokenizer decoding: tokens provide compact global guidance, and the 2D branch preserves local structure.

D.4. Limitations and Future Directions

The framework’s effectiveness depends on whether a frozen tokenizer exposes appearance-sensitive factors useful for fusion. Since tokenizer families may organize appearance information differently, task-specific probing may be needed before selective token editing.

  • Limitations and Future Directions: STE depends on the frozen tokenizer exposing appearance-sensitive factors that are useful for fusion.The TiTok-based interface remains effective on infrared-visible and medical fusion tasks, but this dependency limits portability across tokenizer families.

E. Additional Qualitative Results

Additional qualitative comparisons show that the method improves global appearance coherence while preserving local structures across challenging infrared-visible and medical image fusion scenarios. It maintains salient modality-specific information, clearer boundaries, and fewer artifacts under low illumination, clutter, and cross-modality conditions.

  • Infrared-visible fusion: In urban night, road, and low-light surveillance scenes, the method enhances salient infrared responses while preserving visible structures and maintaining coherent global appearance.Existing methods are described as suffering from over-smoothed textures, unstable brightness, or residual artifacts.
  • Infrared-visible fusion: Across additional M3FD cases, the method preserves thermal targets, visible structural cues, global brightness, and clearer object boundaries.Relevant visible cues include lane regions, building contours, vehicle boundaries, and background layout.
  • Medical image fusion: Across Harvard MRI-CT, MRI-PET, and MRI-SPECT fusion settings, the method preserves complementary anatomical and functional information with more coherent fused appearances.The approach balances MRI structural clarity with modality-specific intensity information from CT, PET, or SPECT.
  • Infrared-visible fusion: On challenging M3FD road scenes, the method reduces visual artifacts and better balances thermal saliency with visible-scene structure.Competing methods are characterized by unstable brightness, over-smoothed textures, and local artifacts around vehicles, pedestrians, and road boundaries.
Loading 2606.12303v1…