PaperlayerView exact revision on arXiv

Source-linked AI summary

FoMo: Forking Moment in Generative Trajectory as a Perceptual Distance

Jaihyun Lew, Mingi Jung, Minjun Park, Wooseok Song, Sungroh Yoon

arXiv:2609.25716v1cs.CV

TL;DR

Reference-based IQA relies on human labels that are costly or noisy, while pairwise labels capture only relative comparisons. FoMo uses diffusion-trajectory fork times to generate pointwise perceptual-distance labels without human annotation. The authors report that it outperforms human-annotated approaches, including KADID-10k MOS labels, and that RankNet-style supervision substantially outperforms 2AFC-style binary classification.

  • Problem

    MOS labels are expensive to collect reliably, while 2AFC labels are more reliable but provide only relative pairwise comparisons.

    Source: §1 · p. 2

  • Method

    FoMo generates pointwise perceptual-distance labels without human annotation by using diffusion-trajectory fork times as a perceptual-distance proxy.

    Source: §1 · p. 2

  • Results

    FoMo outperforms human-annotated approaches, including KADID-10k MOS labels, and RankNet-style supervision substantially outperforms 2AFC-style binary classification.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    FoMo's pointwise scores support consistent global ranking across arbitrary image pairs and offer a scalable, annotation-free alternative to MOS and pairwise human labeling.

    Source: §1 · p. 2

    Models trained with diffusion-generated data may fail on unseen out-of-distribution image domains, such as those with artificial distortions uncommon in nature.

    Source: I Limitations · p. 21

Abstract

from arXiv · show

Reference-based image quality assessment (IQA) metrics aim to reflect how humans perceive the perceptual distance between a pair of images. To learn how the human visual system (HVS) operates, recent reference-based IQA metrics heavily rely on human-annotated data. Mean opinion score (MOS)-based pointwise scoring, which assigns a scalar quality value per image, is preferable for annotation but is prohibitively expensive to collect at scale and is known to be noisy due to inconsistent human judgments. As an alternative, two-alternative forced choice (2AFC) pairwise labels have gained popularity due to their reliability and efficiency, but they capture only relative comparisons between pairs. In this paper, we propose a fully automated data generation pipeline that generates pointwise perceptual distance labels between image pairs without any human annotation. Our approach exploits the generative dynamics of diffusion models as a perceptual distance proxy, where the coarse structure of an image is generated in the early timesteps and the fine details are generated in the later timesteps. Images that fork early in the generation process share only coarse structure and are perceptually far apart; images that fork late differ only in fine detail. We demonstrate that the diffusion trajectory aligns well with the human visual system, and use this forking moment, FoMo, as a reference-grounded distance label to supervise the training of a reference-based IQA metric. The pointwise labels, which support universal comparison between arbitrary image pairs, enable an information-rich training objective. Extensive experiments across diverse backbone architectures confirm the effectiveness of our generation pipeline, outperforming human-annotated datasets in multiple benchmarks.

1 Introduction

Human-annotated IQA faces costly, noisy MOS labels and pairwise labels that do not encode a global ordering. FoMo uses diffusion forking moments to generate reference-grounded pointwise distances without human annotation, and the paper reports strong performance across benchmarks and architectures.

1 Introduction

  • Human MOS collection is expensive and noisy, while reliable 2AFC labels capture pairwise preferences without directly encoding a global image ordering.KADID-10k required 30 ratings per image from over 2,200 subjects, and scores for the same distorted image can differ across datasets.Source: §1 · p. 2
  • FoMo treats the timestep where two diffusion trajectories branch as a reference-grounded distance label: early forks yield perceptually farther pairs, while late forks differ mainly in fine detail.The pipeline generates pointwise perceptual distances without human annotation.Source: §1 · p. 2
  • Controlled image-pair analyses found that forking-moment orderings align with human judgments, motivating an automated pipeline for comparing arbitrary pairs without annotation or reconciliation.The pipeline provides pointwise labels that support global comparison.Source: §1 · p. 2
  • FoMo outperforms human-annotated approaches, including KADID-10k MOS labels, across multiple benchmarks and diverse architectural backbones.The paper presents FoMo as a scalable, annotation-free alternative to MOS and pairwise human labeling.Source: §1 · p. 2
  • FoMo supervision supports globally consistent ranking, and a RankNet-style global objective improves reference-based IQA across diverse benchmarks and architectures.Pointwise scores directly train consistent global ranking rather than binary pairwise preference.Source: §1 · pp. 2–3

2 Preliminary

Diffusion and related generative processes progressively remove image information before recovering an image from an intermediate state. FoMo relies on the trajectory’s timestep-dependent perceptual structure: branching later preserves more shared detail, while branching earlier yields more distinct outputs.

2 Preliminary

  • DDPMs use a forward Markov process to add Gaussian noise over discrete timesteps and a learned reverse process to denoise; score-based models extend this to a continuous-time SDE framework.Flow matching and rectified flows instead use a probability-flow ODE along linear interpolations between data and noise.Source: §2 · p. 3
  • Across diffusion, flow matching, and rectified flows, forward corruption progressively destroys image information and a learned reverse process recovers an image from an intermediate state.FoMo is described in a continuous-time SDE framework, while most experiments use FLUX with a rectified-flow formulation.Source: §2 · p. 3
  • The reverse trajectory recovers fine detail at low noise, discriminative structure and texture at intermediate noise, and coarse global attributes at high noise [6].The passage describes information loss as proceeding roughly from high to low spatial frequencies.Source: §2 · p. 3
  • Branching late preserves most perceptual detail and yields closer pairs, whereas branching early independently regenerates more content and produces more different pairs.The shared reverse trajectory preserves perceptual content up to the selected divergence timestep.Source: §2 · p. 3
  • Decatur et al. show that shared early denoising steps can be reused before trajectories branch, with the branching timestep controlling visual similarity [7].FoMo turns this branching structure into a scalable source of perceptual distance labels.Source: §2 · p. 3

3 Empirical Grounding

The section tests whether diffusion-trajectory forking times reflect human perceptual similarity within and across reference images. Human judgments align with the resulting ordering, including comparisons across different references, with weaker agreement for near-ties.

3 Empirical Grounding

  • The single-reference study ranks five variants per reference; the cross-reference study asks which of two pairs from different references is more similar.Later noise injection corresponds to smaller perturbations and higher expected similarity in the first study.Source: Figure 1 · p. 4, §3.1 · p. 4, §3.2 · p. 4

3.1 Single-Reference Validation

  • Within one reference, human rankings aligned with divergence-timestep order at Spearman correlation 0.970, near the 0.960 inter-rater correlation.The study collected 1,982 responses from 30 participants across 190 ImageNet validation references.Source: §3.1 · p. 4

3.2 Cross-Reference Validation

  • Human choices matched the forking-time ordering in 90.2% of individual responses and 92.8% of item majorities across distinct references.Agreement increased with timestep gap, reaching 100% of majority decisions in each bin above 20 timesteps.Source: §3.2 · pp. 4–5
  • Across 20,000 generated pairs, forking labels correlated with LPIPS-Alex at 0.932, LPIPS-VGG at 0.915, DISTS at 0.904, and DreamSim at 0.904.When forking timesteps differed by more than 20 steps, the metrics agreed with the label over 99.4% of the time.Source: §3.2 · p. 5

4 Method

The method automatically generates reference-grounded distance labels by branching diffusion denoising at sampled timesteps, then uses those pointwise labels to train a globally ordered perceptual metric.

4 Method

  • The pipeline provides fully automated labeling for training a perceptual distance metric, following the empirical validation of diffusion forking moments.Source: §4 · p. 5

4.1 Dataset Construction

  • Forking at a uniformly sampled step sets the noise level; denoising the remaining steps produces a variant paired with its timestep as a distance label.Higher noise at the fork corresponds to greater perceptual deviation; Figure 3 labels smaller fork steps as farther from the reference.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Objective Function

  • Unlike human-annotated 2AFC triplet losses, pointwise distances support binary cross-entropy over all B × B batch comparisons to supervise global distance ordering.The formulation adapts RankNet and uses a sigmoid and stop-gradient operation.Source: §4.2 · p. 6

5 Experiments

Experiments compare FoMo with existing training data and objectives across reference-based IQA benchmarks and backbone architectures. Ablations examine objective and label choices, batch size, diffusion generator, and forking-timestep range.

5.1 Experimental Settings

  • Experiments use 480k training pairs split evenly between real ImageNet and synthetic FLUX reference images, with pretrained backbones and trained prediction heads.DreamSim instead tunes its feature extractor with LoRA; other feature extractors remain fixed.Source: §5.1 · pp. 6–7

5.2 Quantitative Evaluations

  • Across four benchmarks and seven backbones, FoMo achieves the best overall performance, with large PIPAL gains and comparable-to-better results elsewhere.On PIPAL, FoMo outperforms existing datasets and objectives on all seven backbones; on TID2013, CSIQ, and LIVE, it is comparable on CNNs and outperforms most baselines on Transformers.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Ablation study

  • Ranked binary cross-entropy consistently outperforms 2AFC-style loss and L1 regression, and FoMo labels outperform KADID-10k under the rank-based objective.The comparison varies both label type and training objective.Source: §5.3 · p. 8, Table 2 · p. 8
  • Larger batches improve DINOv3 performance, while LPIPS-Alex peaks at batch size 64.The Transformer trend is monotonic; the CNN degrades slightly beyond 64.Source: §5.3 · p. 8, Table 3 · p. 8
  • 0.622 is the best human-annotated CNN baseline that every tested generator surpasses using 10k pairs; all but SD-XL at least match the Transformer baseline (0.508).FLUX.1 is strongest for both backbones, so the pipeline works beyond FLUX but generator choice affects performance.Source: §5.3 · p. 9, Table 4 · p. 9
  • In the 50-step schedule, the first 35 timesteps are most important; the final 15 add little perceptual significance, though DISTS and DINOv3 perform best using the full range.The main experiments retain all 50 steps for overall robustness.Source: §5.3 · p. 9, Table 5 · p. 9

6 Conclusion

The paper frames perceptual distance as a forking moment in a diffusion denoising trajectory and uses it to generate training pairs without human annotation. Human-study alignment and benchmark improvements support the pipeline and ranking-based objective.

6 Conclusion

  • The pipeline forward-diffuses a reference to a sampled timestep and denoises it back to synthesize training pairs with calibrated perceptual distances, without human annotation.A human study found that forking moments align well with human perception.Source: §6 · p. 9
  • RankNet-style supervision on generated data substantially outperforms 2AFC-style binary classification, providing richer gradient signal and implicit transitivity enforcement.Source: §6 · p. 9
  • Experiments across diverse backbones and evaluation benchmarks show consistent improvements over strong baselines.Source: §6 · p. 9

A Experimental Details

The experiments use backbone-specific prediction heads, with a symmetry adjustment for Transformer distance predictions.

A Experimental Details

  • LPIPS-Alex/VGG use linear prediction layers over feature-pyramid levels, while DISTS learns α and β to integrate intermediate features.DINOv3, CLIP, and MAE use fixed feature extractors with three appended ViT layers and a [CLS] token for distance-logit prediction.Source: A Experimental Details · p. 13
  • Transformer predictions are symmetrized by averaging the head’s outputs for both image orders.The concatenated-sequence head can return different values for (u, v) and (v, u).Source: A Experimental Details · p. 13
  • Random input ordering encourages Transformer models to behave symmetrically; LPIPS, DISTS, and DreamSim compute symmetric distances directly.The two Transformer orders are similar but not perfectly identical; no symmetry adjustment is applied to the other three metrics.Source: A Experimental Details · p. 13

B Full Results

The appendix reports full multi-benchmark IQA evaluations and situates FoMo among annotation and diffusion-based alternatives.

B Full Results

  • Table 6 compares training datasets and objectives across four reference-based IQA benchmarks and CNN and Transformer backbones using five-seed SROCC means and standard deviations.Source: Table 6 · p. 13

C.1 Reference-based Image Quality Assessment

  • Tables 7 and 8 report KROCC and PLCC, respectively, for the four-benchmark comparisons across CNN and Transformer backbones.Source: Table 7 · p. 14, Table 8 · p. 14
  • Ranked binary cross-entropy consistently outperforms triplet-based 2AFC and L1 label regression; FoMo labels are more helpful than KADID-10k [23] labels on PIPAL SROCC.Source: Table 9 · p. 14
  • MOS ratings can be inconsistent across raters and sessions, while 2AFC labels capture relative comparisons rather than an explicit global image ranking.PieAPP [31] identifies variation in rating scales; Talebi et al. [38] and dipIQ describe limits of pairwise optimization for global ordering.Source: §C.1 · p. 15
  • Pairwise optimization considers only a small fraction of comparisons per gradient step, and human annotation cost and coverage remain bottlenecks.Source: §C.1 · p. 15

C.2 Diffusion Models as Perceptual Signal

  • Prior diffusion approaches target geometric correspondence, generative style or instance similarity, medical anomaly detection, or face-specific NR-IQA; FoMo instead uses denoising to generate labels.FoMo uses no diffusion representations at inference and does not compute distance from generated images. Prior methods include DIFT [39], Diffusion Hyperfeatures [26], DiffSim, AnoDDPM, and DifFIQA [1].Source: §C.2 · p. 15

D Human Alignment Experiment

The supplementary material documents human-study materials and explains how FoMo labels are adjusted for spatial augmentation.

D Human Alignment Experiment

E Data Augmentation

  • Human pairwise datasets permit flips and rotations but not aggressive random resize crops, which can change perceptual ordering by exposing only a local region.Source: E Data Augmentation · p. 15
  • FoMo labels can be recomputed analytically for spatial transformations using λs, the log-SNR at the fork, as a resolution-invariant proxy for perceptual divergence.Source: E Data Augmentation · p. 19
  • Resizing retains fork timestep s but shifts t in log-SNR space; cropping retains t and recomputes s under the rescaled schedule.Source: E Data Augmentation · p. 19

F Comparison against Off-the-Shelf Metrics

FoMo supervision is comparable to released checkpoints and slightly ahead overall, leading in 25 of 48 cells and all 12 LPIPS-Alex cells.

F Comparison against Off-the-Shelf Metrics

  • 25 of 48 cells favor FoMo supervision, including all 12 LPIPS-Alex cells, across four architectures.FoMo is slightly ahead overall under one shared configuration, without human-annotated supervision or per-architecture or per-benchmark tuning; released checkpoints reflect metric-specific design and training choices.Source: Table 10 · p. 20, F Comparison against Off-the-Shelf… · p. 20

G Per-Sample Label Variance

Forking is stochastic, but the resulting variation in measured distance is small relative to the distance differences represented by the labels.

G Per-Sample Label Variance

  • Across generation regimes, the standard deviation stays below 0.05 and reaches at most 10.1% of its accompanying mean distance.The mean distance itself changes six- to eightfold across the forking schedule.Source: G Per-Sample Label Variance · p. 20
  • Different fork noise gives variants from the same reference and step different distances, yet all variants receive the same label.The analysis uses K = 8 variants per step for 120 ImageNet references across five forking steps.Source: G Per-Sample Label Variance · p. 20
  • Seed variation reorders same-reference variants in at most 2.8% of LPIPS-Alex comparisons and 7.0% of DISTS comparisons.The authors regard this noise as negligible for training because it is independent across 480k training pairs and averages out.Source: G Per-Sample Label Variance · p. 20

H Per-Distortion-Type Analysis

FoMo supervision leads on 40 of 42 distortion types with a Transformer backbone and 20 of 42 with a CNN, with results varying by distortion family. The paper also identifies out-of-domain limits, ethical safeguards, and potential societal benefits and risks.

H Per-Distortion-Type Analysis

  • With a Transformer, FoMo leads on almost every family across benchmarks; with a CNN, it leads throughout PIPAL but trails on most families of three legacy synthetic-distortion benchmarks.Transformer gains are largest for PIPAL super-resolution and CSIQ noise and contrast families; CNN deficits are by hundredths of a point.Source: H Per-Distortion-Type Analysis · pp. 20–21
  • Failures across both backbones cluster in additive pixel noise, small-region corruption, and global photometric shifts, which the diffusion-generated training data do not contain.The authors attribute the gap to whole-image resynthesis, which does not produce these distortion patterns.Source: H Per-Distortion-Type Analysis · p. 21
  • Table 12 summarizes per-distortion-type results across four benchmarks, defining “led” as the number of types where FoMo is ahead of the strongest human-annotated recipe for that backbone and benchmark.PIPAL uses its training split, the only split that exposes distortion types. The protocol is symmetrized native-resolution with seed-0 checkpoints.Source: Table 12 · p. 21

I Limitations

  • Models trained on diffusion-generated data may fail on out-of-distribution domains, including artificial distortions uncommon in nature.The authors suggest training a diffusion model on the new domain to generate samples for training there.Source: I Limitations · p. 21

J Ethics Statement

  • The human annotation study collected no personally identifiable information and was deemed exempt from formal IRB review under a minimal-risk behavioral research exemption.Participants were informed of the task, which involved image-pair preference judgments and no risk beyond normal screen use.Source: J Ethics Statement · pp. 21, 23

K Broader Impacts

  • Automated label generation may lower barriers to building human-aligned IQA metrics, while more accurate metrics could also aid synthetic-media misuse and inherit diffusion-model biases.The paper names image restoration and synthesis, including medical imaging, compression, and accessibility, as applications that could benefit from better metrics.Source: K Broader Impacts · p. 23
Loading 2609.25716v1…