PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGDiffusion++: From Macro-Retrieval to Micro-Fidelity Alignment for Garment Generation

Yuhan Li, Xianfeng Tan, Fangao Zeng, Wenxiang Shang, Pipei Huang, Hao Zhou, Zhiyu Jin, Wenjun Zhang, Bingbing Ni

arXiv:2608.29280v1cs.CVcs.AI

TL;DR

Industrial garment synthesis must preserve both global structure and fine material detail, yet SFT collapses high-frequency patterns and generic RL can exploit reward blind spots. RAGDiffusion++ addresses this with capacity-enhancing data and architecture, attribute-aware reward guidance, and adversarially regularized trajectory-level RL. Across three benchmarks, it reports consistent state-of-the-art performance while improving structural and textural fidelity.

  • Problem

    High-Frequency Trajectory Collapse makes high-frequency garment patterns nearly un-sampleable, while generic reward models can induce artifact hacking.

  • Method

    RAGDiffusion++ combines Dual-Image-Stream FLUX and STGarment-Plus, Garment-RM, and adversarially regularized GRPO to reshape sampling toward faithful details.

  • Results

    RAGDiffusion++ consistently achieves state-of-the-art performance across STGarment-Plus, VITON-HD, and DressCode.

  • Takeaways & Limitations

    The framework aligns garment generation at both macro-topological and micro-texture levels by combining high-frequency priors, perceptive rewards, and artifact prevention.

Abstract

from arXiv · show

Standard clothing asset generation---restoring forward-facing flat-lay garment images from diverse real-world contexts---holds immense commercial value yet demands both macroscopic topological accuracy and microscopic physical fidelity. Although our previous work RAGDiffusion effectively eradicated large-scale structural hallucinations via retrieval-augmented macro-constraints, achieving industrial-grade micro-texture realism remains an unsolved bottleneck. We formally identify this limitation as High-Frequency Trajectory Collapse: supervised fine-tuning (SFT) converges to the conditional mean of the training distribution, which is dominated by smooth, low-frequency textures, causing high-frequency patterns (e.g., fabric weaves, intricate logos) to become nearly un-sampleable. Naively applying Reinforcement Learning (RL) post-training further triggers Artifact Hacking, where models exploit semantic biases in generic reward models by generating deceptive checkerboard noise. Our key insight is that RL can fundamentally reshape the sampling distribution of flow models---elevating the probability of high-fidelity trajectories under accurate reward guidance---while adversarial regularization prevents exploitation of reward blind spots. Realizing this principle requires three prerequisites: (i)inherent capacity, established through a 27,725-pair high-complexity garment dataset (STGarment-Plus) and a Dual-Image-Stream FLUX architecture upgrade; (ii)perceptive reward, provided by a novel attribute-aware reward model (Garment-RM) trained on 500K images via fine-grained contrastive learning, achieving 84.67% human preference accuracy; and (iii)hacking prevention, enforced by our Adversarial-Regularized GRPO (AR-GRPO) strategy that integrates a dynamic discriminator into the RL sampling trajectory to penalize artifacts while enriching authentic high-frequency details.

I. INTRODUCTION

RAGDiffusion++ targets the remaining micro-texture fidelity gap after macro-structural control by identifying High-Frequency Trajectory Collapse and proposing AR-GRPO to reshape sampling while preventing artifact exploitation.

  • Motivation: Industrial garment synthesis requires both macroscopic topology and microscopic physical-material fidelity, but prior RAGDiffusion primarily addressed structural hallucinations.The remaining challenge is stable generation of accurate fabric textures and patterns.
  • Problem: High-Frequency Trajectory Collapse causes SFT outputs to favor smooth fabrics, blurry logos, and irregularized stripes or checks.The paper links this behavior to regression toward the conditional mean and reports these failure modes across generated candidates.
  • Approach: AR-GRPO reshapes trajectory-level sampling so physically correct high-frequency patterns become sampleable rather than remaining low-probability events.Its mechanism combines stochastic SDE sampling with reward-driven policy gradients and adversarial safeguards.
  • Approach: 27,725 pairs in STGarment-Plus and a Dual-Image-Stream FLUX backbone provide inherent capacity for high-fidelity texture sampling.The architecture upgrade injects pure visual features while the dataset emphasizes high-frequency materials.
  • Approach: Garment-RM uses 500K images and 12 fine-grained attributes to provide perceptive reward guidance, achieving 84.67% human preference accuracy.It is designed to steer optimization toward faithful textures instead of semantic shortcuts.
  • Results: AR-GRPO integrates a dynamic discriminator to penalize low-level artifacts while experiments report 14.3% lower FID and 34.2% lower KID than RAGDiffusion.The reported comparisons also include a 51.0% KID advantage over Qwen-Image-Edit and a 10.9% FID advantage over Nano Banana.

II. RELATED WORKS

Prior work provides controllable generation, garment restoration, retrieval-based structural correction, and reinforcement-learning alignment, but generic conditioning and rewards remain insufficient for high-frequency material fidelity.

  • Controllable text-to-image diffusion models: Controllable diffusion methods add trainable modules, prompt strategies, or cross-attention constraints, yet struggle to guarantee pixel-level fidelity for complex physical materials.Their reliance on generic associative capabilities limits fine-grained material control.
  • Garment restoration: Garment-restoration methods improve generation with pretrained backbones, but earlier systems remain vulnerable to structural inaccuracies and practical usability failures.RAGDiffusion addresses these issues with explicit external priors for macroscopic geometry.
  • Reinforcement learning in image generation: Diffusion reinforcement-learning methods optimize non-differentiable rewards, but generic rewards emphasize high-level semantics and lack granularity for garment material details.This limitation leaves generic RL frameworks unable to resolve High-Frequency Trajectory Collapse in garment synthesis.
  • Garment restoration: Industrial garment synthesis is sampling-constrained by strict frontal structures and difficult in-the-wild conditions, including occlusion, layering, and extreme viewpoints.These conditions increase structural uncertainty and pattern ambiguity.
  • Garment restoration: RAGDiffusion uses StructureNet and SLLE to retrieve and fuse structural embeddings and landmarks, anchoring geometry and reducing structural hallucinations.Its conditional pipeline synthesizes multiple structural conditions with an SDXL backbone.

A. Dual-tower embeddings extraction

The method extracts multimodal structural embeddings from in-the-wild and standard garments, then uses contrastive learning and retrieval-based local reconstruction to improve structural reliability.

  • A. Dual-tower embeddings extraction: StructureNet combines visual features with discrete garment attributes to form multimodal embeddings for in-the-wild and standard garments.Twin ViT features are concatenated with attribute embeddings and processed by a Resampler.
  • A. Dual-tower embeddings extraction: InfoNCE training maximizes similarity for matched garment pairs while separating structurally dissimilar pairs.The objective uses cosine similarity, batch size N, and temperature scalar τ.
  • A. Dual-tower embeddings extraction: The inference module extracts structural embeddings from complex in-the-wild images while filtering environmental noise and texture variation.This produces a cleaner structural representation for downstream retrieval and generation.
  • A. Dual-tower embeddings extraction: An external feature memory database supplies standard garment embeddings and paired silhouette landmarks for retrieval-based structural correction.The database addresses limited spatial perception and potential out-of-distribution errors in difficult samples.
  • A. Dual-tower embeddings extraction: SLLE retrieves K-nearest standard references and silhouette landmarks, then computes locally linear reconstruction weights.The method assumes the local embedding manifold is approximately linear and solves a least-squares reconstruction problem.
  • A. Dual-tower embeddings extraction: The final rectified embedding linearly fuses the original in-the-wild representation with its reconstructed counterpart to reduce out-of-distribution errors.This fusion preserves information while pulling noisy inputs toward the in-domain target space.

C. Conditional generative pipeline

The conditional pipeline injects retrieved structural and spatial priors into generation while using reference features to preserve garment patterns; its remaining bottleneck is faithful micro-detail generation.

  • C. Conditional generative pipeline: EP-Adapter injects the rectified structural embedding through parallel cross-attention, while Landmark Guider uses the silhouette landmark for spatial alignment.Together, these modules constrain the garment’s macro-topology during generation.
  • C. Conditional generative pipeline: ReferenceNet extracts intermediate key and value features from the in-the-wild image to preserve coarse garment patterns through attention sharing.The reference features are combined with MainNet features along the sequence dimension.
  • C. Conditional generative pipeline: The preliminary SDXL-based pipeline removed macroscopic structural hallucinations but remained limited in micro-detail faithfulness, including intricate logos and high-frequency fabrics.The degradation was partly associated with severe information loss in the SDXL VAE.
  • C. Conditional generative pipeline: Heuristic PGEA reduced decoding loss but did not fundamentally prevent High-Frequency Trajectory Collapse during denoising.This limitation motivates replacing heuristic patches with adversarial RL and a Dual Image-Stream DiT.
  • C. Conditional generative pipeline: AR-GRPO uses SDE trajectories, an adversarial discriminator, combined rewards, and policy updates to pursue fidelity while avoiding artifacts.The four phases respectively sample trajectories, train the discriminator, compute advantages, and update the generative policy.

IV. METHOD

RAGDiffusion++ treats micro-fidelity as a distribution-reshaping problem requiring latent capacity, perceptive rewards, and artifact prevention, implemented through challenging data, upgraded architecture, and AR-GRPO.

  • IV. METHOD: RL can reshape trajectory probabilities beyond SFT’s smooth modal outputs by using stochastic SDE sampling and reward-driven policy gradients.The framework targets rare but physically correct high-frequency patterns rather than smoothing pixels after generation.
  • IV. METHOD: High-frequency generation requires nonzero latent capacity, supplied by STGarment-Plus and a Dual-Image-Stream FLUX architecture.RL can amplify existing probabilities but cannot amplify a probability of absolute zero.
  • IV. METHOD: Garment-RM provides attribute-aware reward signals, while an adversarial discriminator penalizes artifacts during RL exploration.These components address perceptive reward and hacking prevention as complementary prerequisites.
  • 1) STGarment-Plus Dataset Construction:: The dataset construction combines cleaned real data with curated synthetic pairs and human verification across structural, realism, background, occlusion, and logo-fidelity dimensions.This process is designed to produce high-resolution, reliable paired data for garment restoration.
  • 1) STGarment-Plus Dataset Construction:: STGarment-Plus contains 27,725 paired examples spanning difficult scenes, complex garment categories, and high-frequency materials.It includes 25,756 training pairs and 1,969 test pairs, with over 50% of garments belonging to complex structures absent from open-source benchmarks.
  • IV. METHOD: The Dual-Image-Stream DiT replicates part of FLUX’s denoising stream and pretrained weights to create a dedicated conditioning image stream.This architecture upgrade is intended to raise capacity for conditioning on garment images.

2) DiT Framework and Training Object:

The framework upgrades FLUX into a Dual-Image-Stream DiT and trains it with flow matching, while Garment-RM learns fine-grained attribute-sensitive visual representations for reward guidance.

  • DiT Framework: FLUX is customized into a Dual-Image-Stream DiT by replicating part of the denoising stream for a dedicated conditioning-image stream.The replicated stream is initialized with pretrained weights and processes conditioning images separately from the original text and image streams.
  • Training Objective: The model forms z_t=(1−t)z_0+tz_1 from target garment latents and Gaussian noise, then predicts the velocity field with flow matching.The conditioning signal c is derived from the reference garment and retrieval priors.
  • Garment-RM: Garment-RM addresses the perceptive-reward gap by learning disentangled representations for fine-grained material, texture, and garment attributes.Generic rewards emphasize high-level semantics, whereas Garment-RM is trained through attribute classification and instance-level contrastive learning.
  • Garment-RM: AM-Softmax imposes compact within-class and separated between-class attribute features through an additive angular margin.The formulation uses normalized embeddings and class weights, with s scaling cosine values and m penalizing the ground-truth class angle.
  • Reward Formulation: The global embedding branch adds a margin-based triplet loss, and generated-image reward is defined by cosine similarity with the flat-lay reference.The pretrained Garment-RM therefore supplies a continuous garment-aware evaluation signal.

C. Adversarial-Regularized GRPO (AR-GRPO)

AR-GRPO reshapes flow-model sampling with stochastic trajectories and combines perceptual rewards with a dynamic discriminator to improve authenticity while preventing reward hacking.

  • Sampling and Optimization: AR-GRPO uses SDE-based stochastic sampling to explore trajectories and optimize the diffusion policy with group-relative policy gradients.The rollout uses G=8 trajectories over T=10 steps, beginning from pure noise and decoding terminal latents into images.
  • Multi-Reward Optimization: The reward system combines Garment-RM detail fidelity, LPIPS perceptual similarity, and discriminator-based authenticity supervision.The three raw rewards are independently normalized within each sampled group before forming advantages.
  • Policy Update: The generator is updated with a clipped policy-gradient objective whose clipping threshold bounds policy changes for stable optimization.The combined advantage is constructed from independently normalized reward advantages.
  • Adversarial Discriminator: The discriminator uses a frozen DINOv2 backbone and lightweight binary head trained with hinge loss to distinguish authentic from problematic samples.Only the DINO [CLS] token is retained because it is reported to filter bad cases better than patch tokens.
  • Adversarial Discriminator: The discriminator is initialized offline for 3000 steps and updated for three steps before each generator update during AR-GRPO.Positive samples combine natural images and 25-step ODE outputs, while negative samples come from current SDE rollouts.

A. Experimental Setup

The experiments train and evaluate RAGDiffusion++ on STGarment-Plus, benchmark it against generation and reward-model baselines, and use reconstruction, perceptual, realism, and preference metrics.

  • Datasets: STGarment-Plus contains 25,756 training pairs and 1,969 test pairs of in-the-wild upper-garment images with standard flat-lay targets.The dataset distribution is reported in Fig. 6.
  • Implementation: The implementation uses 768×768 training, AdamW, a 10-day SFT phase on 16 NVIDIA H20 GPUs, and a subsequent AR-GRPO phase on 8 NVIDIA H20 GPUs.Testing uses the Flow Match Euler Discrete Scheduler for 25 steps, while the discriminator and generator use a 3:1 update ratio.
  • Baselines: The evaluation compares SDXL-based RAGDiffusion with established garment-generation methods and benchmarks Garment-RM against generic vision and aesthetic reward models.Baselines include TryOffDiff, TryOffAnyone, IP-Adapter, CLIP, DINO, HPS v2.1, ImageReward, and PickScore.
  • Evaluation Protocols: Generation quality is measured with LPIPS, SSIM, DISTS, FID, and KID, while reward-model alignment uses Preference Accuracy, Recall@K, and Filter@K.The metrics cover reconstruction, perceptual and structural similarity, distributional realism, human alignment, retrieval, and filtering.

B. Main Generative Results

RAGDiffusion++ preserves global garment topology while improving fine-grained textures, outperforming baselines across most reported metrics and showing robustness across datasets and ablations.

  • Qualitative Results: RAGDiffusion++ produces fine-grained patterns and lifelike fabric detail while maintaining global topology through retrieved structural priors.Competing methods exhibit structural collapse, distorted colors, overly smooth textures, or hallucinated garment structures.
  • Quantitative Results: RAGDiffusion++ significantly outperforms baseline approaches across most metrics on STGarment-Plus.The authors caution that SSIM can reward oversmoothing and checkerboard artifacts, so LPIPS, DISTS, FID, KID, and human evaluation are treated as primary micro-texture indicators.
  • Cross-Dataset Evaluation: RAGDiffusion++ recovers its advantage on the more challenging Viton-HD and DC-Dress cross-dataset evaluations.On DC-Upper, SDXL-based RAGDiffusion attains marginally lower FID/KID because that dataset is dominated by simple low-frequency cotton T-shirts.
  • Ablation Study: The architecture upgrade improves clarity and lighting dynamics, while STGarment-Plus enriches fabric detail before AR-GRPO addresses remaining smooth, plastic-like textures.The ablation isolates complementary contributions from architecture, data, and reinforcement learning.
  • Reward Ablations: Removing Garment-RM produces overly smooth, blurred fabrics and poor local alignment, whereas removing the discriminator induces dense checkerboard artifacts that worsen during training.The results identify Garment-RM as important for material and pattern fidelity and the discriminator as important for industrially usable optimization.
  • Reward Dynamics: The discriminator accelerates early authenticity improvement and later stabilizes Garment-RM and LPIPS scores against reward hacking.Without the discriminator, combining Garment-RM with LPIPS delays checkerboard artifacts relative to Garment-RM alone but does not provide the same constraint.

D. Effectiveness of Garment-RM

Garment-RM provides fine-grained attribute-aware reward signals for garment generation, outperforming generic reward models in human preference alignment and supporting meaningful trajectory optimization.

  • Attribute Classification Accuracy: Garment-RM maintains uniformly high classification accuracy across 12 fine-grained garment attributes, enabling attribute-aware reward signals.Its attribute awareness explicitly penalizes structural hallucinations and incorrect textures during reinforcement learning.
  • Human Preference Accuracy: 84.67% human preference accuracy: Garment-RM outperforms aesthetic and generic vision reward models on garment-image preference prediction.The evaluation uses 1,969 test samples with human-labeled preference pairs.
  • Human Evaluation: Garment-RM achieves the highest win rates when selecting preferred top-3 images, especially for garment faithfulness.The human evaluation uses 200 cases with 8 or 16 candidate generations and three annotators.
  • Feature Space Discriminability: Garment-RM embeddings show compact intra-class clustering and distinct inter-class separation, producing dense gradients for AR-GRPO optimization.The resulting cosine-similarity reward is designed to distinguish incorrect textures during subsequent optimization.
  • Interpolation Analysis: Interpolating Garment-RM with LPIPS yields slightly better Top-1 accuracy than using Garment-RM alone.The broader analysis reports that interpolation between reward models improves Top-1 accuracy.

E. Effectiveness of AR-GRPO

AR-GRPO addresses SFT saturation by using stochastic RL exploration and adversarially regularized reward optimization to improve high-frequency garment realism and spectral alignment.

  • Breaking the SFT Saturation Bottleneck: AR-GRPO produces vibrant colors, sharp typography, and faithful fabric textures beyond the SFT saturation bottleneck.The reported results motivate stochastic RL exploration as essential for mastering micro-level physical realism.
  • Frequency-Domain Analysis: AR-GRPO reduces frequency-domain residuals and directional artifacts relative to the Base model, improving alignment with real-image spectra.The comparison uses Fourier spectra from the Base model, AR-GRPO model, and real test set.
  • Reward Perception: Garment-RM comprehensively outperforms GPT-4o and Qwen3-VL in fine-grained attribute perception, supplying reliable signals for RL optimization.The comparison concerns fine-grained attribute perception rather than overall image-generation quality.
  • Frequency-Domain Analysis: AR-GRPO consistently achieves lower spectral error across middle and high frequencies, aligning generated fabrics more closely with real-image statistics.This spectral alignment is reported as explaining the enhanced tactile realism of generated fabrics.
  • Framework Summary: RAGDiffusion++ combines Dual-Image-Stream FLUX with STGarment-Plus, Garment-RM, and adversarial AR-GRPO to establish capacity, perceptive reward, and hacking prevention.The framework is evaluated on STGarment-Plus, VITON-HD, and DressCode, where it reportedly achieves state-of-the-art performance.
Loading 2608.29280v1…