Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

Siyuan Liu; Xuze Zhang; Yongshun Wang; Licong Pan; Hang Liu; Huihui Li

Source-linked AI summary

Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, Huihui Li

arXiv:2609.04726v1Sep 4, 2026cs.CV

TL;DR

BMT addresses the limited semantic awareness of SAR-to-optical translation by jointly optimizing translation and segmentation with a shared hierarchical ViT and complementary conditioning modules. Across paired and unpaired datasets, it achieves competitive translation quality and improves downstream segmentation performance, while retaining a trade-off between the two objectives.

Problem
Existing S2O methods can produce visually plausible images but often lack semantic awareness, while translation and segmentation are commonly modeled separately despite their potential complementarity.
Source: §I · p. 1
Method
BMT jointly optimizes S2O translation and semantic segmentation through a shared hierarchical ViT with LocalViTBlock, multiscale ControlNet-style conditioning, enhanced color output processing, and bounded uncertainty weighting.
Source: §I · pp. 1–2, B. Cooperative Hierarchical Encoding… · p. 3, 1) Enhanced Multi-Scale Output Module · p. 4
Results
BMT achieves competitive generation quality and improves downstream segmentation performance on the WHU-OPT-SAR and HRSID-DIOR datasets.
Source: B. Results of WHU-SAR-OPT · p. 8, C. Results of HRSID-DIOR · p. 8, §V · p. 11
Takeaways & Limitations
The results support collaborative cross-task learning for SAR-to-optical translation and semantic segmentation, although generation and segmentation objectives remain in trade-off.
Source: §V · p. 11

Abstract

from arXiv · show

Synthetic Aperture Radar (SAR) images have all-weather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images. Generative models for SAR-to-optical (S2O) conversion can improve visual interpretability, but existing methods often ignore the constraints on semantic structure, which are necessary for downstream tasks, for the sake of visual effects. We propose a unified collaborative dual-task learning framework, termed BMT (Bridging Modalities and Tasks), that jointly optimizes S2O image translation and semantic segmentation through a shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale refinement processing, color correction and anti-aliasing, which calibrates channel-level color statistics through feature fusion; (3) a ControlNet-style conditional injection mechanism that encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolution; (4) a bounded Kendall uncertainty weighting scheme that prevents either task from dominating the shared representation. We evaluate the framework under both paired and unpaired translation settings, on the public WHU-OPT-SAR paired dataset and a self-constructed unpaired ship dataset built from HRSID and DIOR, respectively. The experimental results show that the proposed method achieves competitive S2O translation quality and semantic segmentation performance. The dataset and source code have been publicly released at https://github.com/Lewisyuaner/BMT-S2O-main.

I. INTRODUCTION

BMT addresses the separation of SAR-to-optical translation and semantic segmentation by jointly learning both tasks with a shared hierarchical ViT. It combines local-global feature modeling, multiscale conditioning, refined generation, and dual-task decoding to preserve semantic structure during translation.

SAR provides all-weather, day-and-night imagery, but speckle noise, geometric distortions, and non-intuitive scattering hinder interpretation and downstream visual tasks.Source: §I · p. 1
Existing S2O translation and SAR segmentation methods are largely studied independently, leaving semantic awareness and bidirectional task complementarity insufficiently addressed.Source: §I · p. 1
BMT uses a shared hierarchical ViT encoder connected to image-generation and semantic-segmentation decoders for unified collaborative learning.Source: §I · p. 1, Figure 1 · p. 3
LocalViTBlock combines global self-attention with spatial depthwise convolution through learnable gating to capture long-range dependencies and fine-grained local features.Source: §I · pp. 1–2
ControlNet-style conditioning injects SAR wavelet features and segmentation labels into every encoder layer through zero-initialized convolutions, while the output module combines refinement, color correction, and anti-aliasing.Source: §I · p. 2, Figure 1 · p. 3

III. METHOD

The method uses a multistage hierarchical ViT encoder that progressively builds multiscale representations for two decoding tasks. LocalViTBlock handles high-resolution local-global modeling, while deeper stages use standard global attention and preserve skip features.

The shared backbone uses a hierarchical ViT encoder and two independent decoders for image generation and semantic segmentation.Source: A. Overview · p. 3
The encoder progressively downsamples spatial resolution while increasing feature dimensions, producing multiscale representations for accurate boundary recovery.Source: B. Cooperative Hierarchical Encoding… · p. 3
LocalViTBlock addresses high-resolution modeling by combining global self-attention with local depthwise-convolution features through a learnable gate.Source: B. Cooperative Hierarchical Encoding… · p. 3
The gate begins with approximately equal branch contributions because its scalar is initialized at zero and passed through a sigmoid.Source: B. Cooperative Hierarchical Encoding… · p. 3
Deeper stages use standard global self-attention after downsampling, with more Transformer blocks and skip connections supplying semantic and boundary cues to the decoders.Source: B. Cooperative Hierarchical Encoding… · pp. 3–4

C. Generation Branch

The generation branch restores full-resolution color images from bottleneck features using symmetric upsampling and skip connections. Its output module adds multiscale color-statistics conditioning and anti-aliasing, while multiscale discrimination targets both texture and geometry.

The generator decoder symmetrically upsamples bottleneck features and reuses encoder skip connections to restore positional, textural, and boundary details.Source: C. Generation Branch · p. 4
The enhanced output module extracts global, regional, and local color statistics using pooling scales of 1×1, 4×4, and 16×16.Source: 1) Enhanced Multi-Scale Output Module · p. 4
Color-conditioning vectors are combined with locally transformed features and passed to scale and bias heads for statistically guided color reproduction.Source: 1) Enhanced Multi-Scale Output Module · p. 4
A Gaussian-style anti-aliasing convolution suppresses checkerboard artifacts produced by the upsampling decoder.Source: 1) Enhanced Multi-Scale Output Module · p. 4
The multiscale cascaded discriminator uses PatchGAN subdiscriminators to enforce both high-frequency texture detail and large-scale geometric consistency.Source: 2) Multi-Scale Discriminator · p. 4

D. ControlNet-Style Multi-Scale Condition Injection Module

BMT replaces single-point conditioning with multiscale spatial injection throughout the encoder. SAR wavelet features and segmentation labels are aligned into a feature pyramid, then introduced through zero-initialized convolutions for stable conditioning.

The ControlNet-style mechanism injects conditional signals, SAR wavelet features, and segmentation labels at every encoder layer rather than only at the bottleneck.Source: D. ControlNet-Style Multi-Scale… · p. 4
A multilevel db2 DWT decomposes SAR images into multiscale frequency features, including LL, LH, HL, and HH subbands.Source: 1) Wavelet Feature Extraction · pp. 4–5
The wavelet features are fused into a pixel-aligned multiscale feature map and concatenated with one-hot segmentation labels for joint texture-semantic conditioning.Source: 1) Wavelet Feature Extraction · p. 5, 2) Condition Encoder · p. 5
A lightweight hierarchical convolutional encoder converts the concatenated conditions into a feature pyramid whose resolutions and channel dimensions match the ViT stages.Source: 2) Condition Encoder · p. 5
Zero-initialized 1×1 convolutions make conditioning an identity mapping at initialization, then gradually enable conditional influence during training.Source: D. ControlNet-Style Multi-Scale… · p. 4, 3) Zero Convolution Injection · p. 5

E. Segmentation Branch Decoder

The segmentation branch shares the encoder and skip connections with the generative branch while producing pixel-level predictions and enforcing semantic consistency through direct and cyclic supervision. Auxiliary generation losses support photorealism, sharpness, and color fidelity in paired and unpaired settings.

Segmentation decoder

The segmentation decoder shares the encoder and skip connections with the generative decoder but outputs pixel-level logprobabilities through a 1 × 1 convolution.Its loss is applied to both direct and recurrent segmentation paths.Source: E. Segmentation Branch Decoder · p. 5
Cyclic supervision feeds the synthetic image through a second generator and segmentation decoder to preserve semantic structure under cycle transformations.Source: E. Segmentation Branch Decoder · p. 5

Auxiliary losses

Three auxiliary losses improve perceptual quality through VGG-16 features, spatial-gradient matching, and color-histogram matching.The VGG-16 loss uses features from four layers with weights {1.0, 1.0, 0.5, 0.25}.Source: 3) Auxiliary loss · p. 5
Color-histogram matching penalizes distribution drift, including background color bleeding into ship targets, while total unpaired generation loss combines adversarial, cycle, gradient, and histogram terms.Source: 3) Auxiliary loss · p. 5

4) Segmentation Loss:

The segmentation objective combines Focal and Dice losses, with class weighting that emphasizes ships and additional supervision applied to cycle-reconstructed images.

Segmentation loss

The segmentation loss combines Focal loss and Dice loss for pixel-level classification.Source: 4) Segmentation Loss · p. 6
The class-weight vector ω = [1, 3] up-weights the ship class, while γ ≥0 controls focal-loss focusing.Source: 4) Segmentation Loss · p. 6
A cycle segmentation loss supervises predictions on cycle-reconstructed images.Source: 4) Segmentation Loss · p. 6

5) Bounded Multi-Task Weighting:

BMT models multi-task learning through uncertainty weighting while bounding task weights to prevent either task from starving the shared encoder. The formulation retains adaptive modulation of task importance during training.

Uncertainty weighting

The generator loss is formulated as a weighted sum over K subtasks using learnable uncertainty parameters.The formulation follows Kendall et al.’s uncertainty-minimization approach.Source: 5) Bounded Multi-Task Weighting · p. 6

Bounded weighting

The bounded formulation ensures that neither task starves the shared encoder while adaptively modulating their relative importance throughout training.Source: 5) Bounded Multi-Task Weighting · p. 6

IV. EXPERIMENTS

Experiments evaluate BMT on paired WHU-OPT-SAR data and an unpaired ship dataset, using visual, generative-quality, and downstream segmentation comparisons. BMT produces sharper, more structurally consistent translations and improves collaborative segmentation metrics over the compared references.

Experimental setup

The evaluation uses WHU-OPT-SAR paired data and a self-constructed unpaired ship dataset from HRSID and DIOR, with established generation and segmentation baselines.WHU-OPT-SAR yields 1,098 cleaned SAR-optical-label tiles split into 900 training and 198 test samples.Source: 1) Dataset · p. 6, 2) Baselines · p. 6

Translation results

BMT generates more accurately positioned water regions with clearer boundaries and more accurate colors than several compared translation methods.Pix2pix and Pix2pixHD produce blurrier water edges, while WFLM-GAN and HVTC-GAN show weaker boundary and color results.Source: B. Results of WHU-SAR-OPT · pp. 6–7
156.31 FID and 45.54 SSIM are reported as BMT’s strongest WHU-OPT-SAR generation results, outperforming all compared generative models on the reported metrics.The study evaluates SSIM, PSNR, FID, and KID; SSIM and PSNR favor higher values, while FID and KID measure feature-distribution differences.Source: 3) Evaluation Metrics · p. 6, B. Results of WHU-SAR-OPT · p. 7

Segmentation results

BMT produces more complete and continuous semantic segmentation over large water areas, whereas most baselines show truncated water boundaries.The comparison uses a unified pretrained FCN-32s model applied to optical images generated by the different S2O methods.Source: Figure 3 · p. 7, B. Results of WHU-SAR-OPT · p. 8
More than 10% improvements in mIoU and mPA are reported for BMT* versus baseline, source SAR, and optical-image segmentation results.The authors interpret this as evidence that the collaborative framework preserves semantic integrity while generating optical images closer to the real distribution.Source: B. Results of WHU-SAR-OPT · p. 8

C. Results of HRSID-DIOR

On the unpaired HRSID-DIOR ship dataset, BMT combines competitive distribution metrics with stronger visual structural fidelity and segmentation performance. Its results also expose a trade-off: style-diversity metrics do not always reflect ship and water accuracy.

S2O translation

BMT achieves FID 161.814, ranking behind UNSB at 142.734 and CUT at 145.782 while improving nearly 8.5% over CycleGAN.Its KID (×100) score is 5.809, second only to UNSB.Source: C. Results of HRSID-DIOR · p. 8
BMT produces clearer ship boundaries, sharper structural details, and more coherent nearshore-water separation than competing unpaired translation methods.CUT and UNSB rank highly on FID/KID but can confuse ship textures with water backgrounds.Source: C. Results of HRSID-DIOR · p. 8

Semantic segmentation

BMT reaches 88.51% mIoU and 93.07% mPA, the highest reported segmentation metrics among the compared methods.For ships, it achieves 77.63% IoU and 86.42% pixel accuracy.Source: C. Results of HRSID-DIOR · p. 8
BMT improves ship IoU by 0.260 percentage points and ship pixel accuracy by 1.084 percentage points over DeepLabv3+.The comparison is made under the condition that the generated images are more realistic.Source: C. Results of HRSID-DIOR · p. 8
Qualitative segmentation results show cleaner boundaries and fewer false positives, although closely moored vessels and weakly backscattering ships remain challenging.These difficult cases are consistent with the quantitative evaluation.Source: C. Results of HRSID-DIOR · p. 9

D. Ablation Studies

Ablations show that collaborative training improves both task branches, while individual modules and losses create non-linear trade-offs between segmentation accuracy and generated-image distribution quality.

Task collaboration

Collaborative training improves both generation and segmentation indicators compared with using either task branch alone.This supports cross-task gains within the unified framework.Source: D. Ablation Studies · p. 10

Module ablations

On HRSID-DIOR, adding LocalViTBlock increases mean IoU/mPA from 84.54/90.44 to 88.01/93.07.The result is presented as evidence for its role in cross-modal feature fusion and spatial-detail modeling.Source: D. Ablation Studies · p. 11
The conditional module is scenario- and metric-dependent: WHU-OPT-SAR mIoU/mPA reaches 44.09/54.73 under the full configuration, while the best KID, 5.21, occurs with only LocalViTBlock.The metrics therefore do not improve synchronously across configurations.Source: D. Ablation Studies · p. 11

Loss ablations

Adding cyclic segmentation loss raises mIoU from 87.62 to 88.32 but worsens FID/KID from 164.21/6.23 to 168.20/6.58.The loss ablation indicates that strengthening segmentation consistency can reduce generation quality.Source: D. Ablation Studies · p. 11
The complete loss combination reaches FID/KID 161.81/5.81 and mPA 93.07, while mIoU settles at 88.51.The authors describe this as the best trade-off, with remaining local fluctuations indicating unresolved multi-task conflicts.Source: D. Ablation Studies · p. 11

V. CONCLUSION

BMT jointly optimizes SAR-to-optical translation and semantic segmentation through a shared hierarchical ViT, combining local-global features, multiscale conditioning, and bounded task weighting. Results on WHU-OPT-SAR and HRSID-DIOR show competitive generation and improved downstream segmentation, while a trade-off between objectives remains.

Framework and findings

BMT uses a shared hierarchical ViT encoder to collaboratively optimize S2O translation and semantic segmentation.Its design integrates LocalViTBlock, multiscale conditional injection, and bounded uncertainty weighting.Source: §V · p. 11
On WHU-OPT-SAR and HRSID-DIOR, BMT achieves competitive generation quality and improves downstream segmentation performance.The conclusion attributes this result to positive cross-task complementarity.Source: §V · p. 11

Limitations and future work

A trade-off between generation and segmentation objectives remains, motivating future fine-grained coordination mechanisms.This is identified as an open direction for further performance improvement.Source: §V · p. 11