PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Salient Factor Discovery and Generation with Visual Foundation Representations

Shuang Liang, Lejun Liao, Shiyuan Zhang, Max C. Zhang, Xiaolong Luo, Han Wang, Stefano Anzellotti, Yuan Yuan

arXiv:2609.39635v1cs.CVcs.LG

TL;DR

Contrastive analysis aims to separate target-specific detail from content shared across target and background datasets, including patterns that labels do not describe. SAGE factorizes a frozen representation-autoencoder latent and conditions a diffusion transformer on a reference image’s salient representation. It combines reconstruction, subtype discovery, and generation results across Digits-ImageNet and FFHQ, and separates three retinal diseases using normal/disease labels.

  • Problem

    Target/background labels can identify groups without describing their subtle, unknown image differences, limiting discovery of potential subtypes from those differences.

    Source: §1 · pp. 1–2

  • Method

    SAGE factorizes a frozen representation-autoencoder spatial latent into common and salient factors, then conditions a diffusion transformer on a reference image’s salient representation.

    Source: §1 · p. 2

  • Results

    SAGE reconstructs Digits-ImageNet and FFHQ with rFID below 2, discovers unlabeled subtypes, and raises Digits-ImageNet subtype accuracy to 90.5% vs. 27.7% for the unfactorized latent.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The salient space recovers digits, eyewear types, and retinal diseases without subtype labels, and supports generation that preserves a discovered subtype while common content varies.

    Source: §5 · p. 10

    On OCT-Kermany, reconstruction is less faithful than on natural images (SSIM 0.448 versus 0.571 on Digits-ImageNet and 0.723 on FFHQ) because the frozen autoencoder is pretrained on natural images.

    Source: E LIMITATIONS · p. 28

Abstract

from arXiv · show

Given a target dataset, such as faces with eyeglasses, and a background dataset, such as faces without, contrastive analysis separates \textit{salient} factors specific to the target from \textit{common} content shared by both. We aim for salient representations that capture target-specific detail in each image, such as the shape, color, and position of the glasses, so that they reveal subtypes without subtype labels and guide the generation of new examples of a discovered subtype, even one with no name or text description. We introduce SAGE, which learns both factors directly in the high-dimensional spatial latent of a frozen representation autoencoder and conditions a diffusion transformer on the learned salient representation of a reference image. On Digits-ImageNet and FFHQ eyeglasses, SAGE combines high-fidelity \textit{reconstruction} (rFID below $2$) with unsupervised \textit{subtype discovery}, recovering the digits better than baselines (probe accuracy $0.950$ vs.\ at most $0.281$) and revealing eyewear types, finer sunglasses styles, and mislabeled images; salient-conditioned \textit{generation} raises Digits-ImageNet subtype accuracy over the unfactorized latent ($90.5\%$ vs.\ $27.7\%$) and diversity on both datasets. On retinal OCT, SAGE's salient space separates three diseases using only normal/disease labels.

1 INTRODUCTION

Contrastive analysis seeks target-specific factors that distinguish a target dataset from shared content, including subtle patterns that labels alone do not describe. SAGE factorizes a frozen representation-autoencoder latent to discover those factors and use them for reconstruction, subtype discovery, and generation.

1 INTRODUCTION

  • Clinical group labels can identify who is affected without revealing how their brains differ, motivating methods that uncover subtle imaging patterns and potential subtypes.The introduction gives autism MRI as an example of this gap.Source: §1 · pp. 1–2
  • SAGE factorizes a frozen RAE spatial latent into common and salient factors that sum in the decoder’s latent space, retaining image-specific target detail.The spatial representation can preserve details such as the shape, color, and position of a particular pair of glasses.Source: §1 · p. 2
  • Existing generative contrastive-analysis methods decode factors but reconstruct complex natural images poorly, while contrastive methods learn semantic factors but lack a decoder.SAGE is designed to combine semantic richness with image decoding.Source: §1 · p. 2
  • SAGE’s salient factors reveal unlabeled subtypes and support reference-conditioned generation; on Digits-ImageNet, subtype accuracy is 90.5% vs. 27.7% for the unfactorized latent.The paper also reports rFID below 2 on Digits-ImageNet and FFHQ, and subtype discovery on FFHQ and retinal OCT.Source: §1 · p. 2

2 RELATED WORK

Prior contrastive-analysis methods trade off decoding, semantic factor quality, and the representation used for factorization. SAGE instead factorizes a frozen autoencoder’s full spatial latent for discovery and generation.

2 RELATED WORK

  • Generative contrastive-analysis models learn factors with a generator trained from scratch, while SepCLR learns salient representations but has no decoder and loses separation when reconstruction is added.CS-StyleGAN uses an inverted StyleGAN latent, and Diff-CA decomposes compact diffusion-generator conditioning tokens.Source: §2 · p. 2
  • Prior methods often represent salient factors as low-dimensional vectors or global tokens, whereas SAGE factorizes a frozen RAE’s full spatial latent for discovery and generation.SAGE conditions on a discovered salient representation rather than a full self-supervised representation or a prompt-specified attribute.Source: §2 · pp. 2–3
  • Figure 2 depicts Stage 1: trainable common and salient encoders split frozen RAEv2 latents, and its frozen decoder maps the factors back to images.The figure’s top row includes reconstruction and swap adversarial training; its lower row shows swapped-image re-encoding and factor-level objectives.Source: Figure 2 · p. 3, §2 · p. 3

3 METHODOLOGY

SAGE uses dataset labels to split a frozen representation-autoencoder latent into common and salient factors, then freezes the factorization to train a generator conditioned on salient content. Its objectives encourage reconstruction, target-specific salient content, and instance-level factor recovery.

3 METHODOLOGY

  • SAGE uses only background/target dataset labels to factorize a frozen RAE latent in Stage 1, then freezes that factorization for salient-conditioned diffusion training in Stage 2.The frozen encoder is DINOv3-based RAEv2; the spatial latent is factorized directly without a learned bottleneck.Source: §3 · p. 3, §3.1 · p. 3

3.1 SALIENT FACTOR DISCOVERY

  • Common and salient encoders produce same-shape factors whose sum is decoded by the frozen RAE decoder; a zero salient factor represents no target-specific content.The factors are required to be decodable and composable, not statistically independent.Source: §3.1 · p. 3
  • Asymmetric reconstruction omits the salient factor for background images and combines it with the common factor for target images.Latent terms tie decoder inputs to real latents, avoiding plausible but unreliable compositions.Source: §3.1 · p. 4
  • Swap adversarial training moves target-specific content into the salient factor by erasing it from target reconstructions and adding it to background common factors.The salient encoder starts at zero, so salient content is acquired only when required by the swap objective.Source: §3.1 · p. 4
  • Background salient factors receive an L2 penalty and target salient factors an L1 penalty, while cycle consistency requires instance-specific factors to remain recoverable after swaps.Cycle NCE uses globally pooled representations before and after swap–re-encoding to discourage overlap between common and salient content while preserving subtype grouping.Source: §3.1 · pp. 4–5

3.2 SALIENT-CONDITIONED GENERATION

  • Stage 2 max-pools the reference salient representation into a conditioning token and trains an RAEv2-initialized diffusion transformer on target images with flow matching.At inference, different noise samples vary common content while the salient condition fixes salient characteristics.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments test SAGE on Digits-ImageNet, FFHQ, and retinal OCT for reconstruction, subtype discovery, factor manipulation, and salient-conditioned generation. Results report high-fidelity reconstruction, subtype structure, and generation that preserves discovered subtypes; OCT edits are qualitative, not clinical evidence.

4.1 EXPERIMENTAL SETTINGS

  • Evaluation uses Digits-ImageNet, FFHQ, and OCT-Kermany, with subtype labels reserved for evaluation; comparisons include decoder-based and representation-based baselines.Metrics cover reconstruction, subtype discovery, clustering, and generation.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 SALIENT FACTOR DISCOVERY

  • SAGE achieves rFID 1.78 on Digits-ImageNet and 1.65 on FFHQ, while generative baselines trained from scratch exceed 120.Its full reconstruction combines common and salient factors, while salient-only decoding renders the digit or eyeglasses.Source: Table 1 · p. 6, §4.2 · p. 6, Figure 4 · p. 6

4.3 SUBTYPE DISCOVERY

  • Digit probe accuracy is 0.950 versus at most 0.281 for baselines; FFHQ probe accuracy and ARI/NMI are 0.983 and 0.939/0.865.The salient space groups ten digits and two eyewear types, separates three sunglasses styles, and identifies a cluster of mislabeled no-glasses images; an audit corrected 125 of 3,437 target validation labels.Source: §4.3 · pp. 7–8

4.4 FACTOR MANIPULATION

  • Swapping transfers a target attribute to donor content, erasure removes it while preserving the rest, and salient interpolation traverses subtypes while preserving common content.Examples include transferring a digit or particular glasses style and changing clear reading glasses into sunglasses on a fixed face.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 DISCOVERING DISEASE SUBTYPES IN RETINAL OCT

  • With only normal/disease training labels, SAGE separates three OCT diseases, achieving linear-probe accuracy 0.960 and ARI/NMI 0.415/0.415.Adding a disease salient factor introduces fluid-filled cavities, while removing it yields a retina closer to normal; these edits are qualitative, not clinical evidence.Source: §4.5 · p. 8, Figure 8 · p. 9

4.6 SALIENT-CONDITIONED GENERATION

  • On Digits-ImageNet, salient conditioning raises subtype accuracy from 27.7% to 90.5% and Vendi from 2.65 to 12.50; on FFHQ, Vendi is 6.96 versus 2.32.On FFHQ, both conditions preserve eyewear type (96.6% versus 98.5%); salient conditioning also lowers Cos-to-ref from 0.779 to 0.470.Source: Table 2 · p. 9, §4.6 · p. 9

4.7 ABLATION STUDIES

  • Without the swap adversarial loss, the digit probe falls from 0.950 to 0.123 and ARI/NMI approach zero; removing image-space cycle loss worsens rFID from 1.78 to 2.84.Removing target sparsity, Cycle NCE, or either consistency loss raises salient ImageNet probe accuracy from 0.045 to between 0.086 and 0.302.Source: §4.7 · pp. 9–10, Table 3 · p. 9

5 CONCLUSION

SAGE learns salient factors in a frozen representation autoencoder’s spatial latent using background/target labels. The paper reports reconstruction and editing, subtype discovery without subtype labels, and salient-conditioned generation across image domains.

5 CONCLUSION

  • SAGE learns salient factors in the high-dimensional spatial latent of a frozen representation autoencoder using only background/target labels.The paper reports that these factors reconstruct and edit individual images and support subtype discovery and generation.Source: §5 · p. 10
  • The authors state that code, trained models, and audited FFHQ labels will be released upon publication.Experiments use public MNIST, ImageNet, FFHQ, and OCT-Kermany data, and the RAEv2 encoder and decoder remain frozen.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors disclose using generative AI to assist with experimental code and manuscript polishing, and say they reviewed the outputs.They state responsibility for the final content, including text, claims, code, and results.Source: AI USE STATEMENT · p. 10

SUPPLEMENTARY MATERIALS FOR SAGE: SALIENT FACTOR DISCOVERY AND GENERATION WITH VISUAL FOUNDATION REPRESENTATIONS

The supplementary results show how SAGE’s factors decode, compare across clustering and visualization methods, and expose finer sunglasses structure without style labels.

A.1 ADDITIONAL FACTOR-DECODING EXAMPLES

  • SAGE’s common-only decoding removes the digit or eyewear while retaining scene or identity, and salient-only decoding preserves image-specific target details.The additional examples show this for Digits-ImageNet and FFHQ.Source: Figure 9 · p. 15, §A.1 · p. 15

A.4 FINE-GRAINED SUBTYPE STRUCTURE WITHIN SUNGLASSES

  • Without fine-style labels, SAGE separates sunglasses into three appearance groups: Wayfarer-like C1, mixed-style C2, and Aviator-like C3.The displayed groups contain 168, 390, and 175 examples, respectively; their geometry and lens shapes differ despite sharing one coarse label.Source: §A.4 · pp. 15–16, Figure 11 · p. 17

A.2 CLUSTERING ACROSS REPRESENTATION SPACES

  • SAGE has the highest FFHQ ARI/NMI in the original space at k = 2 (0.939/0.865) and k = 3 (0.937/0.865), outperforming DINOv3 + SepCLR.It also leads across the PCA readouts and settings reported in Table 4.Source: §A.2 · p. 15, Table 4 · p. 16

A.3 SALIENT REPRESENTATION VISUALIZATIONS ACROSS METHODS

  • On Digits-ImageNet, SAGE forms clear digit groups where SepCLR and DINOv3 + SepCLR mix identities; on FFHQ, all three reveal eyewear-related groups.SAGE has the highest background/target silhouette score in both t-SNE and original spaces on both datasets.Source: §A.3 · p. 15, Figure 10 · p. 16, Table 5 · p. 17

A.5 FFHQ EYEWEAR LABEL AUDITING AND REMAINING SAGE ERRORS

The supplementary section audits FFHQ eyewear labels and remaining errors, illustrates factor swapping and erasure, and describes OCT evaluation and reference-conditioned generation results.

A.5 FFHQ EYEWEAR LABEL AUDITING AND REMAINING SAGE ERRORS

  • Manual inspection corrected 125 of 3,437 FFHQ validation eyewear labels (3.64%); SAGE surfaced no-glasses errors, while reading-versus-sunglasses corrections came from the audit.The 39 verified no-glasses images include 34 in a separate group and five among glasses groups, associated with face paint or costume eye masks.Source: §A.5 · p. 17, Figure 12 · p. 18, Figure 13 · p. 18

A.6 ADDITIONAL SWAPPING AND ERASURE EXAMPLES

  • Swapping a target salient factor into background common content transfers the target attribute, whereas decoding target common content alone removes that attribute.The additional examples cover Digits-ImageNet and FFHQ.Source: §A.6 · pp. 18–19, Figure 14 · p. 19

A.7 STAGE 1 METRICS ON OCT-KERMANY

  • OCT-Kermany metrics use the official test set of 1,000 images from patients absent from training and validation, and report linear-probe accuracy, clustering metrics, and reconstruction quality.The table cautions that rFID is sensitive to finite-sample bias at this evaluation size and is not directly comparable with larger-set scores.Source: Table 6 · p. 19, §A.7 · p. 19

A.8 SALIENT-CONDITIONED GENERATION

  • Salient conditioning generates new faces and scenes that preserve the reference digit or eyewear type, while Raw DINOv3 GMP nearly reproduces the reference and may change or distort the digit.Multiple samples from one reference are also evaluated using Vendi score.Source: §A.8 · pp. 19–20, Figure 15 · p. 20

A.9 FULL LOSS ABLATION

The full-loss ablation examines how cycle NCE and the background penalty affect salient leakage, digit accuracy, and common-only reconstruction.

A.9 FULL LOSS ABLATION

  • Table 7 expands the loss ablations with the unfactorized latent as a reference and probes of both learned factors.Source: §A.9 · p. 20
  • Cycle NCE lowers the salient ImageNet probe from 0.100 to 0.045 while digit accuracy stays at 0.950.Source: §A.9 · p. 20
  • The squared ℓ2 background penalty outperforms ℓ1 on salient ImageNet probe (0.045 vs. 0.067) and common-only reconstruction (rFID 1.89 vs. 2.00).The authors choose squared ℓ2 because it better keeps shared content out of the salient factor.Source: §A.9 · p. 20

B.1 DATASET DETAILS

This section documents dataset construction and preprocessing, baseline configurations, evaluation protocols, and additional Stage 1 details.

B.1 DATASET DETAILS

  • Training uses only background/target labels, with subtype labels reserved for evaluation; Digits-ImageNet overlays MNIST digits on disjoint ImageNet photos, while FFHQ uses automatically annotated eyewear groups.The OCT setup treats normal scans as background and CNV, DME, and DRUSEN jointly as target; its source scans contain potentially exploitable saturated-white borders that are cleaned before use.Source: §B.1 · p. 21

B.2 BASELINE DETAILS

  • The baselines include generative CA models and SepCLR; DINOv3 + SepCLR uses SAGE’s frozen RAEv2 encoder and common/salient architecture to isolate the training framework’s contribution.For generation, Raw DINOv3 GMP and salient conditioning differ only in whether the salient encoder is applied.Source: §B.2 · pp. 21–22

B.3 DINOV3 + SEPCLR BASELINE

  • DINOv3 + SepCLR uses GAP to form 1,024-dimensional branch vectors, maps them to 32-dimensional evaluation representations, and trains the common/salient Transformers, fully connected layers, and projectors.Its stated objectives and loss hyperparameters follow SepCLR.Source: §B.3 · p. 22, Table 9 · p. 22

B.4 LINEAR-PROBE EVALUATION

  • Digits-ImageNet probes use standardized frozen features and logistic regression with stratified five-fold cross-validation, while FFHQ uses balanced logistic regression on manually verified eyewear labels.Digits scores are mean accuracy; FFHQ scores are mean balanced accuracy across folds, using salient sparsity weight λ = 0.01 for the main SAGE result.Source: §B.4 · p. 22

B.5 CLUSTERING AND T-SNE PROTOCOL

  • Clustering evaluates pooled salient vectors with k-means at class-matched k, reporting ARI and NMI; t-SNE is used only for qualitative visualization.The t-SNE protocol uses cosine distances, PCA initialization, perplexity 30, and 1,000 iterations; clustering scores average over three k-means seeds.Source: §B.5 · pp. 22–23

C ADDITIONAL STAGE 1 DETAILS

  • The additional Stage 1 section describes network architectures, training settings, and full objective definitions summarized in Section 3.1.Source: C ADDITIONAL STAGE 1 DETAILS · p. 23

C.1 NETWORK ARCHITECTURES

The appendix details SAGE’s common and salient encoders, adversarial and consistency objectives, and the training and sampling setup for salient-conditioned generation.

C.1 NETWORK ARCHITECTURES

  • Frozen RAE features feed common and salient encoders with residual Transformer blocks; a separate frozen DINOv1 backbone supplies features to background and target discriminator heads.The salient encoder’s output convolution is zero-initialized.Source: §C.1 · p. 23, Figure 16 · p. 23, Figure 17 · p. 24

C.2 STAGE 1 TRAINING SETUP

  • FFHQ trains for 200 epochs and Digits-ImageNet for three; both use AdamW at 3 × 10−5 while the visual backbones and RAE decoder remain frozen.OCT-Kermany trains for 60 epochs with a class-balanced batch of 32 normal and 32 diseased scans.Source: §C.2 · p. 24

C.3 SWAP ADVERSARIAL OBJECTIVE

  • The encoders use a non-saturating adversarial objective, while the discriminator uses hinge losses on scalar real/fake scores; they are updated alternately each step.The hinge losses are ℓ+(s) = max(0, 1 − s) and ℓ−(s) = max(0, 1 + s).Source: §C.3 · p. 24

C.4 CONSISTENCY OBJECTIVES

  • Image-space cycles re-encode swapped fakes to recover donor and target factors, while latent swaps encourage additive factor recovery across varied donors using stop-gradient targets.For fake background images, the cycle also penalizes salient activations.Source: §C.4 · p. 25

C.5 CYCLE NCE LOSS

  • Cycle NCE aligns each re-encoded factor with its same-factor original, uses cross-factor negatives, excludes same-factor off-diagonal pairs, and averages common and salient losses.Original representations are stop-gradient targets; re-encoded representations carry gradients.Source: §C.5 · pp. 25–26, Figure 18 · p. 26

D ADDITIONAL STAGE 2 DETAILS

  • Stage 2 combines a salient condition with noisy latent and time embeddings; different noise samples vary remaining content, while a null condition yields unconditional target samples.Sampling integrates from τ = 1 to τ = 0 before decoding the resulting spatial latent.Source: D ADDITIONAL STAGE 2 DETAILS · pp. 26–27

D.1 STAGE 2 TRAINING AND SAMPLING SETUP

  • Stage 2 trains on target images; both datasets use 50-step Euler sampling with internal guidance, while only Digits-ImageNet also uses classifier-free guidance.The classifier-free guidance scale is 1.5 for Digits-ImageNet; FFHQ uses none.Source: §D.1 · p. 27

D.2 WITHIN-CONDITION DIVERSITY EVALUATION

The evaluation measures variation among samples generated from the same reference, pairs SAGE and baseline outputs under identical conditions, and compares generated diversity with real within-subtype diversity.

D.2 WITHIN-CONDITION DIVERSITY EVALUATION

  • Diversity is measured within each reference because scoring across references can make a copying model appear diverse through variation inherited from the references.The study reports these results in Table 2.Source: §D.2 · p. 27
  • Each reference has 16 noise-paired samples; SAGE and Raw DINOv3 GMP share reference lists and initial noises, and real within-subtype samples provide a diversity reference.FFHQ uses 30 references per verified eyewear subtype; Digits-ImageNet uses 10 per digit class.Source: §D.2 · pp. 27–28
  • The Vendi Score ranges from 1 for 16 identical samples to 16 for pairwise orthogonal features; reference-image cosine similarity measures copying tendency.The score is computed from eigenvalues of the cosine-kernel matrix of the generated samples’ normalized DINOv2-L features.Source: §D.2 · p. 28

E LIMITATIONS

SAGE’s limitations include incomplete disentanglement of correlated attributes and less faithful OCT reconstruction under natural-image representation priors.

E LIMITATIONS

  • Common retains linearly decodable digit information (probe accuracy 0.336 versus 0.328 for the unfactorized latent), so target information is not assigned exclusively to salient.Background/target labels do not uniquely determine how correlated attributes split between factors.Source: E LIMITATIONS · p. 28
  • OCT-Kermany reconstruction has SSIM 0.448, versus 0.571 on Digits-ImageNet and 0.723 on FFHQ, under the frozen natural-image-pretrained encoder and decoder.The authors note that a biomedical-pretrained representation autoencoder could replace RAEv2 without changing the method.Source: E LIMITATIONS · p. 28

Related papers

Loading 2609.39635v1…