PaperlayerView exact revision on arXiv

Source-linked AI summary

StaX-Net: Stackelberg Interleaved Asymmetric Network for Cross-Modality Medical Image Translation

Amin Golnari, Saeid Sanei

arXiv:2610.34214v1cs.CVcs.AI

TL;DR

Anatomically inconsistent medical image translation can compromise clinical diagnosis. StaX-Net combines interleaved shared blocks, asymmetric leader-follower updates, and FAST target-prior distillation for CT-to-MRI synthesis. The authors report preservation of global geometry and fine structural detail relative to standard adversarial models.

  • Problem

    Traditional GANs can produce anatomically implausible hallucinations in medical image translation, where these errors can compromise downstream clinical tasks.

    Source: §1 · p. 1

  • Method

    StaX-Net combines interleaved shared generator-discriminator blocks, asymmetric leader-follower optimization, and a FAST layer that distills target MRI priors into a source-only transform.

    Source: §3.2.2 · p. 3, §6 · pp. 7, 9

  • Results

    StaX-Net reports a superior balance of global anatomical geometry and fine-grained high-frequency detail relative to standard adversarial models.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors report that target-informed representations mitigate the risk of anatomically implausible hallucinations relative to standard adversarial models.

    Source: §6 · p. 9

    StaX-Net does not claim formal analytical convergence to a strict Stackelberg equilibrium because exact-best-response hypergradients are computationally prohibitive for deep networks.

    Source: §3.2.2 · p. 3, §4.2 · p. 5

Abstract

from arXiv · show

Cross-modality medical image translation is challenged by anatomically inconsistent synthesis that can compromise clinical diagnosis. We propose StaX-Net, a Stackelberg interleaved asymmetric network that incorporates a partially coupled encoder-discriminator structure, and a 2D feature aggregation and spatial transformation (FAST) layer that distills target-supervised priors under the learning using privileged information paradigm. The shared blocks between the generator and discriminator enhance architectural feature alignment. To stabilize the leader-follower training, a Stackelberg-inspired asymmetric optimization scheme employs explicit shared-parameter freezing and asymmetric learning rates. The FAST layer distills target-domain priors during training and applies a source-only transform at inference. This constrains the synthesis process through target-informed feature representations relative to standard generative architectures. The code is available on GitHub for public use.

1. Introduction

StaX-Net targets anatomically implausible CT-to-MRI synthesis, a concern because hallucinations can compromise clinical tasks. It combines interleaved shared network blocks, asymmetric training, and target-supervised FAST features.

1. Introduction

  • Anatomically implausible hallucinations from traditional GANs can compromise downstream tasks such as radiation therapy planning or fracture diagnosis.CT-to-MRI translation is motivated by MRI’s stronger soft-tissue contrast and the potential to derive MRI-informed priors from CT.Source: §1 · p. 1
  • StaX-Net interleaves shared multi-scale blocks across the generator and discriminator while retaining pathway-specific activations and dedicated layers.The shared structure is designed to address feature misalignment and gradient conflicts in standard coupled networks.Source: §1 · p. 1
  • Its asymmetric updates freeze shared blocks during generator training, while FAST uses projected MRI priors during training to guide a source-only transform.The paper describes FAST as a structural regularizer that guides synthesis away from anatomically implausible representations.Source: §1 · p. 1

2. Related Work

Prior work explores parameter coupling and game-theoretic training, but does not specify how shared transformations should be arranged across pathways and assigned to competing objectives. StaX-Net positions its stage-wise sharing and asymmetric updates against these approaches.

2.1. Architectural Evolution: From Disjoint to Coupled GANs

  • Coupled GANs use parameter sharing to impose structural constraints and semantic alignment, while interleaving parameters across G and D without gradient conflict remains less explored.Earlier disjoint GAN parameter spaces are associated with divergent features and mode collapse.Source: §2.1 · pp. 1–2

2.2. Game-Theoretic Training Dynamics

  • Stackelberg formulations model a leader committing before a follower responds; selective discriminator-layer freezing provides an empirical precedent for StaX-Net’s strategy.Related work also describes batch policy learning, generalized Nash equilibrium problems, and unrolled adversarial regularization.Source: §2.2 · p. 2

2.3. Weight Sharing and Feature Alignment

  • Prior medical-imaging work uses shared backbones or weights, but sharing alone does not specify their arrangement or assignment to competing objectives.StaX-Net reuses shared transformations stage by stage, lets the discriminator control their optimization, and has the generator adapt dedicated synthesis modules.Source: §2.3 · p. 2

2.4. Stability, Lipschitz Continuity, and Convergence

  • Shared backbones can constrain G and D to a common feature representation, limiting feature divergence and potentially contributing to training stability.Other stability approaches cited include WGAN-GP gradient penalties and spectral normalization.Source: §2.4 · p. 2

3. Proposed Methodology

StaX-Net interleaves shared computational blocks across generator and discriminator stages while preserving pathway-specific processing. Its generator combines FAST-guided training, skip connections, and dedicated losses to reconstruct MRI representations.

3.1. Coupled Encoder-Discriminator Structure

  • At corresponding stages, the generator and discriminator reuse the same parameterized blocks but process distinct feature representations and retain separate task-specific transformations.Synthesis uses generator-dedicated parameters, while classification uses discriminator-dedicated parameters.Source: §3.1 · p. 2
  • During generator updates, shared blocks remain active as fixed feature extractors; only the generator’s dedicated parameters are updated.The discriminator-controlled update adjusts the shared-block parameters.Source: §3.1 · p. 2
  • Multi-scale target MRI guides FAST layers during training, while the target is omitted at inference and the generator reconstructs MRI through progressive upsampling.FAST outputs connect to the decoding pathway through skip connections.Source: §3.1 · p. 2, Figure 1 · p. 3
  • The discriminator alternates dedicated layers with the generator’s shared blocks before a fully connected network performs final classification.The paper describes this sharing as enforcing semantic feature alignment across the two domains.Source: §3.1 · p. 2
  • The generator’s follower loss combines adversarial, L1 geometric-anchoring, and SSIM structural-similarity terms with positive balancing hyperparameters.The total FAST-layer count is integrated across different spatial scales within the generator.Source: §3.1 · pp. 2–3
  • The leader loss uses an MSE objective and is optimized independently during the leader phase.The FAST prior-distillation loss is computed at each FAST layer.Source: §3.1 · p. 3

3.2. Stackelberg-Inspired Asymmetric Optimization

StaX-Net assigns shared and discriminator-dedicated parameters to a leader strategy, and generator-dedicated and FAST parameters to a follower strategy. Sequential updates approximate leader-follower dynamics using parameter freezing and asymmetric learning rates.

3.2.1. Parameter Space Decomposition

  • The leader strategy owns shared and discriminator-dedicated parameters, while the follower owns generator-dedicated and FAST parameters.This allocation replaces simultaneous updates to shared parameters, which the paper says could drive competing objectives in opposing directions.Source: §3.2.1 · p. 3

3.2.2. Sequential Leader-Follower Dynamics

  • The sequential scheme approximates leader-follower dynamics with asymmetric freezing and learning rates, with ηG > ηD.Exact best-response analytical hypergradients are computationally prohibitive for deep networks.Source: §3.2.2 · p. 3
  • The discriminator updates shared and dedicated parameters first; the generator then updates only its dedicated parameters with shared parameters frozen.The generator adapts its synthesis pathway to the representation established by the preceding discriminator update.Source: §3.2.2 · p. 3

3.3. Feature Aggregation and Spatial Transformation Layer

FAST transforms source CT features into a target-informed representation while keeping target MRI out of the forward input. Target supervision shapes its priors during training; inference uses only the source input.

3.3. Feature Aggregation and Spatial Transformation Layer

  • FAST’s internal loss is computed during the forward pass and injected into the total optimization objective.Figure 2 depicts the FAST layer architecture and this loss connection.Source: Figure 2 · p. 3

3.3.1. Formal Notation and LUPI Paradigm

  • The FAST layer expands source content, extracts spatial priors, applies content-driven FiLM modulation and prior-driven AdaIN styling, then fuses the representations.The spatial prior uses standard and dilated convolutional branches followed by squeeze-and-excitation.Source: §3.3.1 · p. 4

3.3.2. Phase 1 - Target-Supervised Prior Distillation

  • The target MRI is projected to align with the prior bank and supervises it through an auxiliary feature-space loss, without entering the forward input.The loss combines an L1 intensity anchor with structural alignment and is added to the generator objective during its update.Source: §3.3.2 · p. 4

3.3.3. Phase 2 - Inference without Target Supervision

  • At inference, FAST uses only the source CT slice and its learned parameters, with no target modality or target-dependent adaptation.The dynamic prior bank and content representations are generated from the source input.Source: §3.3.3 · p. 4

3.4. Implicit Feature Matching

StaX-Net shares computational blocks between generator and discriminator pathways to make feature alignment an architectural constraint. Its asymmetric training updates the discriminator first, then updates the generator while holding the discriminator fixed.

3.4. Implicit Feature Matching

  • The generator uses the discriminator’s shared feature extractor, making implicit feature matching an architectural constraint rather than a separate statistical penalty.The generator decoder receives features from the same shared computational graph, weights, and nonlinear activations used by the discriminator.Source: §3.4 · p. 4
  • The discriminator leads training by updating shared and dedicated parameters while the generator parameters are held fixed.The algorithm samples source and target data, synthesizes a fake image, and updates the discriminator using its loss.Source: §3.4 · p. 5
  • The generator then updates its dedicated parameters and FAST using the updated shared representation while discriminator parameters remain fixed.The generator objective includes the FAST loss.Source: §3.4 · p. 5

3.5. Cooperative-Competitive Hybrid Loss

The follower loss combines adversarial and deterministic constraints to balance target-like texture with anatomical and structural fidelity.

3.5. Cooperative-Competitive Hybrid Loss

  • The follower loss combines adversarial feedback with L1 and SSIM constraints to guide synthesis toward the paired target.L1 anchors global anatomy, SSIM preserves local structure, and the adversarial term encourages target-like high-frequency texture.Source: §3.5 · p. 5
  • L1 anchors global anatomy, while SSIM preserves local structure and supports precise boundary delineation.The passage describes these as deterministic constraints in the follower loss.Source: §3.5 · p. 5
  • The adversarial term encourages target-like high-frequency texture alongside the deterministic constraints.The passage characterizes the combined losses as balancing optimization and constraining synthesis toward the paired target.Source: §3.5 · p. 5

4. Theoretical Analysis

The analysis frames StaX-Net’s synthesis as target-informed and structurally regularized, with shared-block freezing intended to reduce gradient conflict. It presents this as a heuristic, not a formal Stackelberg-equilibrium guarantee.

4.1. Target-Informed Feature Distillation

  • FAST uses target-informed feature transformation as a structural regularizer that encourages anatomically consistent target-modality representations and mitigates hallucinations in empirical evaluations.The authors characterize the transformation as a strong inductive bias, not a strict mathematical constraint on the output space.Source: §4.1 · p. 5

4.2. Optimization Stability via Asymmetric Updates

  • Freezing shared blocks during the generator phase makes the follower adapt to the leader’s representation, and the authors report reduced training volatility and feature-space divergence.They do not claim formal analytical convergence to a strict Stackelberg equilibrium, which would require computationally prohibitive hypergradient evaluations.Source: §4.2 · p. 5

4.3. Design Justification

  • Freezing only shared blocks preserves the discriminator’s adversarial signal while letting the generator’s dedicated blocks map fixed shared features to the target domain.The asymmetric learning rate ηG > ηD compensates for the generator’s restricted update space without exact multistep unrolling or formal best-response calculations.Source: §4.3 · p. 5

5. Experiments

Experiments evaluate StaX-Net on paired SynthRAD 2025 CT–MRI subsets using fidelity, perceptual, distributional, frequency, and volumetric measures. Results report competitive fidelity, preservation of fine-structure frequency characteristics, and competitive volumetric consistency from a 2D framework.

5.1. Dataset and Preprocessing

  • Experiments use paired CT–MRI volumes from SynthRAD 2025 Abdomen/Pelvis, Head and Neck, and Thorax, split by subject into 80% training, 10% validation, and 10% testing.Volumes are cropped to remove empty boundary slices, resampled in-plane to 256 × 256, normalized to [−1, 1], and augmented with rotations, shifts, and flips.Source: §5.1 · pp. 5–6

5.2. Implementation Details and Baselines

  • StaX-Net is compared with Pix2Pix, Vanilla cGAN, UNet-AE, CycleGAN, and an adapted MedViT-cGAN under a shared 150-epoch training setup.The generator learning rate is 1.25 × 10−4 and the discriminator learning rate is 1 × 10−4; the FAST loss uses balanced L1 and gradient weights.Source: §5.2 · p. 6

5.3. Evaluation Metrics

  • The evaluation spans classical fidelity, VGG16 feature distance, Fréchet feature distance, power-spectrum analysis, and volumetric metrics including Z-MAE.PSNR and SSIM assess pixel fidelity and local structure; 3D metrics are calculated on reconstructed volumes rather than averaged slice-wise.Source: §5.3 · p. 6

5.4. Results and Evaluation

  • Despite relying entirely on an efficient 2D architecture, StaX-Net achieves superior volumetric fidelity compared with computationally intensive 3D diffusion framework MD-DGA, evaluated on the earlier SynthRAD 2023 dataset.Source: §5.4 · p. 7

5.4.1. Classical Fidelity and Statistical Stability

  • StaX-Net reports SSIM of 0.748 in Thorax and 0.750 in Head & Neck, with 3D MAE/RMSE of 0.1012/0.2492 and 0.0915/0.2318, respectively.UNet-AE has higher Head & Neck PSNR (19.84) and Abdomen/Pelvis 2D MAE (0.1281), but its smoother outputs can attenuate high-frequency micro-textures.Source: §5.4.1 · p. 7

5.4.2. Qualitative Evaluation and Error Localization

  • Synthesis grids show preservation of trabecular bone, soft-tissue boundaries, and cardiac margins, while CycleGAN shows localized structural deviations and UNet-AE smoother approximations.Source: §5.4.2 · p. 7

5.4.3. Frequency Domain Realism

  • StaX-Net follows the ground-truth power-spectrum trajectory more closely than the evaluated alternatives, while UNet-AE has reduced high-frequency content and Vanilla cGAN deviates in selected high-frequency regions.Source: §5.4.3 · p. 7

5.4.4. Volumetric (3D) Stress Test and SOTA Comparison

  • StaX-Net shows competitive cross-slice consistency in volumetric metrics; sequential 2D outputs processed through shared blocks and the FAST prior preserve volumetric characteristics without explicit 3D convolutions.Source: §5.4.4 · p. 7

6. Conclusion

StaX-Net combines interleaved shared generator–discriminator blocks and stabilized leader-follower updates with a FAST layer that distills target-supervised priors into a source-only inference mapping. Evaluations report a balance of anatomical geometry and fine structural detail, with reduced risk of anatomically implausible hallucinations relative to standard adversarial models.

6. Conclusion

  • Interleaved shared generator–discriminator blocks enhance feature alignment, while parameter freezing and asymmetric learning rates stabilize leader-follower updates.Source: §6 · p. 7
  • Figure 3 compares input CT, ground-truth MRI, and predicted MRI across three anatomical regions, while Figure 4 plots radially averaged power spectra for those regions.Figure 4 uses spatial frequency on the x-axis and logarithmic power magnitude on the y-axis.Source: Figure 3 · p. 8, Figure 4 · p. 9
  • The FAST layer distills target-supervised spatial priors into a dynamic source-only mapping for inference.Source: §6 · p. 9
  • StaX-Net preserves global anatomical geometry and fine-grained high-frequency details while mitigating anatomically implausible hallucination risk relative to standard adversarial models.The conclusion also cautions that classical pixel-wise metrics alone are inadequate for evaluating medical image translation.Source: §6 · p. 9

Declaration of Generative AI in Scientific Writing

The authors used Gemini Pro for English-language editing and reviewed and edited the resulting text.

Declaration of Generative AI in Scientific Writing

  • Gemini Pro was used to improve English, correct grammar, and enhance flow; the authors reviewed and edited the text and take responsibility for the final publication.Source: Declaration of Generative AI in… · p. 9

Related papers

Loading 2610.34214v1…