Source-linked AI summary
Cross-scale Aligned Supervision for Training GANs
Sangeek Hyun, MinKyu Lee, Jae-Pil Heo
TL;DR
Standard scale-wise adversarial supervision can produce realistic intermediate images without ensuring that stages follow the same sample trajectory or form a proper coarse-to-fine hierarchy. CAT preserves scale-wise discriminator feedback and adds generator-side consistency regularization; on ImageNet-256, CAT-H/2 achieves FID-50K 1.56 with one-step inference after 60 training epochs.
Problem
Standard scale-wise supervision matches intermediate outputs to per-resolution real distributions independently, so realism does not ensure cross-stage sample correspondence.
Method
CAT retains scale-wise discriminator feedback and adds generator-side consistency regularization aligning intermediate outputs with the final output.
Results
1.56 FID-50K is achieved by CAT-H/2 on ImageNet-256 with one-step inference after 60 training epochs.
Takeaways & Limitations
Generator-side cross-scale alignment is presented as an effective principle for scaling transformer-based GANs.
Takeaways & Limitations
CAT relies on a manually specified scale hierarchy, while adaptive scale selection and systematic discriminator-scaling analysis remain future work.
Abstract
from arXiv · showhide
Modern GANs often introduce adversarial supervision on intermediate generator outputs and interpret the resulting multi-stage synthesis as coarse-to-fine hierarchical generation. In this work, we challenge this interpretation. We argue that standard scale-wise adversarial supervision does not construct a proper coarse-to-fine hierarchy: each intermediate image is independently pushed toward the real distribution at its own resolution, but this scale-wise realism does not ensure that outputs across stages represent the identical generated sample. Moreover, the scale-specific image produced at each stage is not used as an explicit refinement target for the subsequent stage. Therefore, its adversarial loss can improve a scale-specific output without constraining later stages to preserve the same sample trajectory, allowing them to move toward a different sample rather than refine the previous output. We refer to this problem as a cross-scale trajectory misalignment problem. To resolve it, we propose CAT, a Cross-scale Aligned Transformer for multi-scale adversarial generation. CAT keeps the discriminator scale-wise, so each intermediate output is evaluated at its own resolution, while adding a simple generator-side consistency regularization that aligns intermediate outputs with the final output. On class-conditional ImageNet-256, CAT-H/2 achieves an FID-50K of 1.56 with one-step inference after only 60 training epochs, outperforming strong one-step GAN and diffusion/flow baselines.
1 Introduction
The paper argues that scale-wise adversarial supervision does not guarantee a coherent coarse-to-fine hierarchy, and proposes CAT to align intermediate outputs with the final output. On ImageNet-256, CAT-H/2 reaches FID-50K 1.56 after 60 training epochs with one-step inference.
- Scale-wise supervision can make intermediate images realistic at their resolutions without making them progressively refined states of the same sample.
- Because later stages continue from generator features rather than previous images, they can follow a different sample trajectory instead of refining xk.
- CAT retains a scale-wise discriminator while adding generator-side consistency regularization that aligns intermediate outputs with the final output.
- 1.56 FID-50K is achieved by CAT-H/2 on ImageNet-256 with one-step inference after 60 training epochs.
2 Preliminary
This section defines GAN generation and describes multi-stage adversarial supervision, where scale-specific intermediate images receive independent discriminator feedback while generator features carry synthesis between stages.
- GANs formulate image generation as an adversarial game between a generator and discriminator conditioned on noise and class information.
- Multi-stage GANs expose intermediate images and apply adversarial feedback at multiple generator stages, often motivating a coarse-to-fine interpretation.
- xk denotes the image supervised at stage k, while fk denotes hidden generator features passed into subsequent synthesis stages.
- Scale-wise supervision evaluates each intermediate image independently at its corresponding resolution, providing direct realism feedback without discriminator cross-scale information exchange.
3 Proposed Method
Standard scale-wise supervision can produce realistic intermediate images that do not follow a coherent trajectory toward the final sample. CAT preserves scale-wise discrimination while adding generator-side consistency alignment to reduce cross-scale rewriting and misalignment.
- Cross-scale trajectory misalignment: Cross-scale trajectory misalignment occurs because independently realistic intermediate outputs need not represent progressively refined states of the same generated sample.Scale-wise adversarial feedback matches per-scale distributions without imposing sample-wise correspondence across stages.
- Cross-scale aligned supervision: The discriminator evaluates each resized scale-specific image independently, using a block-diagonal attention mask to prevent cross-scale information exchange.The generator objective combines scale-wise adversarial loss with consistency regularization, while the discriminator objective remains unchanged.
- Cross-scale trajectory misalignment: The analysis measures final-output discrepancy, inter-stage rewrite magnitude, and update-direction alignment after resizing outputs to the highest resolution.These metrics assess whether intermediate outputs accumulate coherently toward the final sample.
- Cross-scale trajectory misalignment: Under standard scale-wise supervision, δk and Rk often exceed 0.8 while Ak remains low, indicating large rewrites that are weakly aligned with the final direction.The discrepancy and rewrite magnitude tend to increase during training rather than diminish at finer stages.
- Cross-scale aligned supervision: CAT aligns intermediate generator outputs with the final-stage output using a consistency loss while retaining direct scale-wise discriminator feedback.Lower-resolution stages receive weaker consistency weights because coarse outputs are inherently ambiguous.
4 Experiments
Experiments evaluate CAT on ImageNet-256 through benchmark comparisons, training dynamics, consistency ablations, compute analysis, and discriminator diagnostics. CAT-H/2 reaches strong one-step generation quality while consistency regularization improves cross-scale alignment and CAT outperforms comparable baselines.
- Comparison with prior work: 1.56 FID-50K is achieved with 60 training epochs, versus 1.72 after 800 epochs for iMF-XL/2.CAT-H/2 also uses lower training and inference GFLOPs than iMF-XL/2.
- Training dynamics: CAT-H/2 steadily improves to FID-50K 1.56 at 150K iterations, while CAT-M/2 reaches 1.93 at 100K iterations.The training curves indicate stable scaling with longer training.
- Consistency ablations: Consistency regularization improves FID from 5.43 to 4.06 for G-B/2 at 20 epochs and from 2.34 to 1.93 for G-M/2 at 40 epochs.The larger model shows stronger gains with longer training.
- Capacity and discriminator analysis: CAT-H/2 significantly outperforms GAT-XL/2 with FID 2.552 versus 4.021 despite comparable total parameters.CAT-B/2 achieves FID 4.063 versus 4.021 for GAT-XL/2 at a Base-scale configuration.
- Consistency ablations: Adding Lcons reduces average δk by 39% and 45% and Rk by 43% and 46% for Base and Medium models, respectively.It also improves Ak by 46% and 66% for Base and Medium models.
- Discriminator analysis: A scale-aggregated discriminator performs severely worse than CAT's scale-wise counterpart and exhibits strong cross-scale token interaction.The diagnostic suggests joint discriminator access does not itself provide the intended scale-wise realism assessment.
5 Related Work
Prior generative models decompose image synthesis into intermediate prediction stages, while GANs commonly apply adversarial supervision to intermediate outputs at multiple resolutions. CAT challenges interpreting scale-wise realism alone as a coherent coarse-to-fine refinement trajectory.
- Generative modeling paradigms: Diffusion, autoregressive, and masked prediction models generate images through sequences of intermediate prediction steps.These paradigms decompose generation into simpler subproblems rather than synthesizing the complete image at once.
- Multi-scale GANs: Multi-scale GANs apply adversarial losses to intermediate outputs at multiple resolutions, often interpreting this design as hierarchical generation.CAT argues that scale-wise realism alone does not ensure a coherent refinement trajectory.
6 Limitations and broader impact
CAT remains constrained by a manually specified scale hierarchy, and adaptive scale selection is left for future work.
- Scope limitations: CAT relies on manually specified numbers of stages and scale resolutions, leaving adaptive scale selection for future work.The paper also notes limited discriminator-scaling analysis and potential dataset bias or misuse.
7 Conclusion
The analysis finds that standard scale-wise supervision can leave intermediate outputs misaligned with the final image. CAT addresses this with generator-side consistency regularization and reaches strong one-step ImageNet-256 performance.
- Scale-wise supervision can produce realistic intermediate images that remain misaligned with the final image, with large discrepancy and inter-stage rewriting.
- CAT preserves scale-wise discriminator feedback while adding generator-side consistency regularization to align intermediate outputs.
- 1.56 FID-50K is achieved by CAT-H/2 on ImageNet-256 with single-step inference and 60 training epochs.
- The results suggest that generator-side cross-scale alignment is effective for scaling transformer-based GANs.
A.1 Implementation details
CAT uses fixed-resolution transformer synthesis and constructs multi-scale discriminator inputs by resizing accumulated intermediate latent outputs. Scale-wise discrimination is enforced while generator-side consistency aligns the outputs.
- CAT trains transformer generator and discriminator networks in the SD-VAE latent space, representing ImageNet-256 images as 32 × 32 × 4 latents.
- The generator produces accumulated outputs at a fixed token-grid resolution rather than naturally generating progressively different spatial resolutions.
- Each intermediate latent output hk is resized by a scale-specific operator rk to form the discriminator input at scale k.
- The discriminator uses scale embeddings, separate [cls] tokens, and block-diagonal attention to provide scale-wise feedback without cross-scale token exchange.
A.2 Details of GFLOPs computation
The GFLOPs estimates count generator and discriminator computation separately for training and inference. CAT keeps discriminator cost lower than a design that scales both networks.
- Training compute is reported per sample per iteration, while inference compute is reported per generated sample using analytical GFLOPs estimates.
- The estimate includes discriminator and generator steps, adversarial discrimination, gradient penalties, and frozen-discriminator evaluation for generator training.
- A CAT training iteration is approximated as 4FG + 10.5FD under the stated accounting rules.
- CAT-H/2 has a smaller discriminator forward cost than GAT-XL/2 because CAT keeps the discriminator at the Base scale while scaling the generator.
- GAT-XL/2 is estimated at 4FG + 15FD, producing substantially larger per-iteration training cost despite lower generator-only inference cost.
A.3 Preliminary Pixel-space Experiment
A preliminary pixel-space experiment applies the method with adapted patches and a multi-resolution hierarchy. The method achieves competitive FID with fewer training epochs, while broader tuning and controlled comparison remain future work.
- The pixel-space experiment uses G-B/2 and D-B/2 with an adapted patch size and resolution hierarchy [16, 32, 64, 256].
- The experiment additionally uses a ConvNeXtV2-based vision-aided loss following pMF.
- 3.54 FID-50K is obtained after 40 epochs, comparable to the pMF baseline trained for 160 epochs.
- The result suggests compatibility with pixel-space training and large-patch image-space generation.
- The pixel-space experiment is less extensively tuned, and more extensive tuning with a fully controlled comparison is left to future work.