PaperlayerView exact revision on arXiv

Source-linked AI summary

RefGC-SR$^2$: Reference-guided Super-Resolution and Refinement of AI Generated Content

Jeahun Sung, Dahyeon Kye, Soo Ye Kim, Jihyong Oh

arXiv:2606.15158v2cs.CV

TL;DR

Reference-guided generation discards fine details from high-resolution references and introduces artifacts, while existing refinement and super-resolution methods address only part of the problem. RefGC-SR^2 reuses the original reference to jointly upscale and refine generated content, achieving stronger refinement, detail recovery, and overall quality than competing methods.

  • Problem

    Existing pipelines downsample high-resolution references and introduce generative artifacts, while RefGCR and RefSR methods do not jointly address artifact refinement and reference-guided resolution recovery.

  • Method

    The paper introduces a real-world triplet dataset and a frequency-aware diffusion transformer that selectively transfers reference details while refining artifacts.

  • Results

    83%, 82%, and 83% top-1 rates for refinement, detail, and overall quality, respectively, versus at most 8% for competing methods.

  • Takeaways & Limitations

    RefGC-SR^2 preserves reference identity while restoring fine details, producing higher-quality results for generated-content refinement and super-resolution.

  • Takeaways & Limitations

    The dataset covers only object-centric scenes across 12 categories, and the model may struggle with large viewpoint or geometry gaps between HRRI and LRGI.

Abstract

from arXiv · show

Reference-guided generation (e.g., object compositing, customization) has progressed rapidly, yet current pipelines share a fundamental limitation: the object-centric high-resolution reference image (HRRI) provided by users is downsampled to a fixed low-resolution (LR) before being fed into the model, so the fine-grained details are discarded before the output is even produced. In addition, the generation step then introduces its own artifacts (e.g., identity distortion) on top of this loss. Existing reference-guided generated content refinement (RefGCR) methods can correct some of these artifacts but still operate in the LR domain; reference-guided super-resolution (RefSR) methods recover resolution but assume natural-image degradations and ignore the artifact distribution of generative pipelines. To address both gaps in a single formulation, we introduce a new task: reference-guided generated content super-resolution-refinement (RefGC-SR$^2$), where the original HRRI is reused at the post-processing stage to recover lost details, refine generative artifacts, and upscale the output simultaneously. We construct the first real-world triplet data generation pipeline for this RefGC-SR$^2$ task, training a diptych-conditioned generator to synthesize paired low-quality anchors that public pretrained models cannot provide. We further present a frequency-aware diffusion transformer model for RefGC-SR$^2$ that selectively injects fine details from the HRRI while removing generative artifacts. Extensive experiments demonstrate that our RefGC-SR$^2$ model successfully (i) refines the object identity faithfully with respect to the reference, and (ii) recovers high-resolution details, so that the final result is significantly higher quality and practically more usable compared to existing RefGCR and RefSR baselines.

1 Introduction

RefGC-SR2 is introduced as a post-processing task that reuses an object-centric HRRI to jointly super-resolve reference-guided generated content and refine its generative artifacts. The paper contributes a real-world triplet dataset, a DipRefGC synthesis pipeline, and a frequency-aware model for this setting.

  • Task formulation: RefGC-SR2 is the first task satisfying all four criteria: generated-content and HRRI inputs, with super-resolution and generative-artifact refinement as joint targets.Existing enhancement tasks satisfy only subsets of these requirements.
  • Task formulation: RefGC-SR2 jointly uses a low-resolution generated content image and the original HRRI to recover fine details, preserve object identity, and refine generative artifacts.It is defined as a post-processing task for converting an initial generated output into a high-resolution, reference-faithful image.
  • Dataset construction: The RefGC-SR2 Dataset is the first real-world LRGI-HRRI-HRGT triplet dataset for supervised training and evaluation of this task.HRRI and HRGT show the same object instance under different views, poses, or scene contexts, while LRGI contains generative artifacts.
  • Dataset construction: DipRefGC synthesizes aligned LRGIs by combining HRRI appearance with HRGT-derived pose conditions through inpainting and Canny ControlNets.This produces pose-consistent, generative-artifact-containing LRGIs paired with HRRI and HRGT.
  • Model: The proposed RefGC-SR2 model injects frequency-aware modules into DiT blocks to remove LRGI artifacts while increasing resolution.Its components include frequency-adaptive mixture of LoRA experts and a frequency-based loss.

2 RefGC-SR2 Dataset

The RefGC-SR2 dataset is designed around pose-consistent real-world triplets that preserve reference appearance while including generative artifacts. It combines curated high-resolution pairs with DipRefGC-generated low-resolution anchors, yielding supervised data for joint super-resolution and refinement.

  • Dataset Requirements: RefGC-SR2 triplets require same-instance HRRI and HRGT images with differing views, poses, or contexts, plus artifact-containing LRGI aligned to HRGT pose.These requirements ensure the model learns super-resolution and refinement while preserving object pose.
  • Dataset Requirements: Existing SR, refinement, compositing, and customization datasets or generators fail to jointly model generative artifacts and pose-consistent supervision.SR datasets use hand-crafted degradations, VLM corruption misses true RefGC artifacts, and off-the-shelf generators alter object pose.
  • LRGI Synthesis: DipRefGC uses a frozen FLUX backbone with LoRA-tuned dual ControlNets to synthesize LRGIs inheriting HRRI appearance while following HRGT pose.Its diptych conditioning disentangles appearance and pose control, producing generative-artifact-containing anchors suitable for triplet supervision.
  • Dataset Construction: The final dataset contains 40K training triplets and an 200sample evaluation split, each pairing a pose-consistent 5122 LRGI with HRRI and HRGT.Curated HRRI-HRGT images are downsampled to 5122 for synthesis while original high-resolution images are retained as HRRI and HRGT.

3 RefGC-SR2 Model

The RefGC-SR2 model jointly removes generative artifacts and increases resolution while selectively exploiting a clean, viewpoint-different HRRI. It combines frequency-adaptive LoRA experts in a frozen diffusion-transformer backbone with a band-specific latent loss.

  • Model Overview: RefGC-SR2 jointly recovers high-resolution details from the LRGI, refines generative artifacts, and selectively exploits an HRRI captured under different conditions.The method addresses the need to perform artifact refinement and resolution recovery simultaneously while using reference information selectively.
  • Frequency-adaptive Mixture-of-LoRA Experts: FreqMoLE replaces each DiT block’s single LoRA with LF and HF experts combined by a learnable depth-dependent gate α, initialized near 1.0 in early layers.The gating gradually favors LF processing earlier and HF processing later, following a coarse-to-fine prior.
  • Frequency-based Loss: The frequency-based loss decomposes latents into LF and HF components, aligns LF global structure with HRGT, and matches object-region HF channel statistics with HRRI.Because HRRI and HRGT differ in viewpoint, the HF objective avoids pixelwise alignment; Lf combines the terms as Lf = λLF LLF + λHF LHF.
  • Motivations: The model analyzes FLUX-Kontext’s layer-wise frequency hierarchy, where low-frequency energy saturates within the first ∼5% of layers and high-frequency energy emerges sharply in the last ∼10%.These observations motivate frequency-specialized adaptation across diffusion-transformer layers.
  • Overall Pipeline: The pipeline encodes the LRGI and HRRI with a frozen VAE and uses a text instruction to guide 4× refinement and super-resolution toward the HRGT.The model takes the target-view LRGI, a same-instance HRRI from another viewpoint, and a task-specific text instruction.

4 Experiments

Experiments evaluate RefGC-SR^2 on proposed benchmark and in-the-wild data, showing strong quantitative, qualitative, and user-preference results. Ablations further demonstrate that frequency-aware experts, frequency-specific supervision, and DipRefGC conditioning improve restoration and identity preservation.

  • Datasets: The RefGC-SR2 Dataset contains 40K training triplets, with LRGI images downsampled from 1024 × 1024 to 256 × 256 using bicubic interpolation.The RefGC-SR2 Benchmark contains 200 held-out triplets, alongside an additional in-the-wild benchmark built from excluded HRRI and HRGT pairs.
  • Quantitative Results: RefGC-SR2 achieves the best performance across all metrics on the in-the-wild evaluation, demonstrating practical applicability.The quantitative comparison includes ImageCritic, OmniPaint, AdaRefSR, ReFIR, DiT4SR, and TSD-SR, with fine-tuned results reported where applicable.
  • Qualitative Results: Qualitative examples show that RefGC-SR2 refines generative artifacts while preserving fine details from HRRI and producing high-quality upscaling.Results are presented on both the RefGC-SR2 Benchmark and the practical in-the-wild benchmark.
  • User Study: 83%, 82%, and 83% are RefGC-SR2’s top-1 rates for refinement faithfulness, detail restoration, and overall quality, versus at most 8% for competing methods.The user study compares SR, RefSR, RefGCR, and RefGC-SR2 outputs under the in-the-wild setting.
  • Ablation Study on RefGC-SR2: Adding Lf improves identity preservation by DINO +7.5% and perceptual quality by LPIPS −19.9%, while combining Lf and FreqMoLE achieves the best performance across all metrics.FreqMoLE restores frequency-specific details such as cup transparency, and the two components provide complementary structural modulation and band-specific supervision.
  • Ablation Study on DipRefGC: Canny guidance improves pose consistency, while fine-tuning DipRefGC on compositing and customization improves reference identity preservation and reduces the distribution gap to real RefGC outputs.These findings validate both pose conditioning and RefGC-oriented fine-tuning in triplet generation.

5 Conclusion … B Related Works

RefGC-SR2 jointly performs reference-guided super-resolution and artifact refinement for generated images, supported by a dedicated dataset and model. Related work distinguishes this task from natural-image, generated-content, super-resolution, and refinement settings by their inputs, outputs, resolution changes, and artifact assumptions.

  • 5 Conclusion: RefGC-SR2 jointly performs super-resolution and artifact refinement on generated images guided by an HRRI, with a dedicated dataset and model.DipRefGC is a FLUX-based generator that synthesizes HRGT-aligned artifact-corrupted LRGI triplets for supervised training.
  • Appendix Contents: The related-task taxonomy separates super-resolution methods that enlarge spatial resolution from refinement methods that remove artifacts or degradations at fixed resolution.Figure 9 organizes eight related tasks into these two categories and depicts their input-output structures and spatial resolutions.
  • A Technical appendices and supplementary material: The supplementary material provides additional RefGC-SR2 benchmark and comparison results, with plans to release full results, data, code, and checkpoints.Only a subset is included because of space limitations.
  • B Related Works: Image Super-Resolution reconstructs an HR image from a single LR observation without an external reference, typically under natural-image degradation assumptions.Recent diffusion-based methods exploit pretrained generative priors for degradations including bicubic downsampling, blur, noise, and compression.
  • B Related Works: RefSR adds an external HRRI to LR input for recovering fine-grained details, while GCSR increases generated-content resolution without an HRRI.RefSR targets natural-image SR, whereas GCSR relies on pretrained generative priors to hallucinate missing details.
  • B Related Works: RefGCSR would super-resolve generated content using an HRRI, but it has not been systematically established as a dedicated literature task.A straightforward implementation would apply existing RefSR methods to generated content.
  • B Related Works: IR and RefR address natural-image degradations, while GCR and RefGCR refine generated content without changing resolution; RefR and RefGCR additionally use an HRRI.RefGCR is closest to RefGC-SR2 because it corrects artifacts or inconsistencies in generated content using reference-guided alignment or refinement, but its output remains refined generated content.

C Qualitative Result on Commertial Model

Qualitative comparisons on outputs from Gemini 2.5 Flash Image, GPT-Image 1.5, and Qwen-Image-Edit show that RefGC-SR2 restores fine details and enhances resolution. The method is compared against six existing baselines across these commercial and open-source model outputs.

  • Qualitative comparisons: RefGC-SR2 faithfully restores fine details while enhancing resolution on outputs from Gemini 2.5 Flash Image, GPT-Image 1.5, and Qwen-Image-Edit.These comparisons are presented in Figs. 10–12, where the generated outputs serve as LRGIs.
  • Gemini 2.5 Flash Image: Figure 10 compares RefGC-SR2 with ImageCritic, OmniPaint, AdaRefSR, ReFIR, TSD-SR, and DiT4SR on Gemini 2.5 Flash Image outputs.The figure provides a qualitative comparison on LRGIs generated by the commercial model.
  • GPT-Image 1.5: Figure 11 compares RefGC-SR2 with ImageCritic, OmniPaint, AdaRefSR, ReFIR, TSD-SR, and DiT4SR on GPT-Image 1.5 outputs.The figure provides a qualitative comparison on LRGIs generated by the commercial model.
  • Qwen-Image-Edit: Figure 12 compares RefGC-SR2 with ImageCritic, OmniPaint, AdaRefSR, ReFIR, TSD-SR, and DiT4SR on Qwen-Image-Edit outputs.The figure provides a qualitative comparison on LRGIs generated by the open-source model.

D Ablation of DipRefGC

The DipRefGC ablation shows that Canny ControlNet improves pose-mask consistency, while RefGC-oriented fine-tuning improves identity preservation. Training uses staged triplets from increasingly challenging datasets, with UltraVideo fine-tuning reducing FID.

  • D Ablation of DipRefGC: Canny ControlNet raises mask IoU from 0.480 to 0.601 in compositing and from 0.311 to 0.601 in customization.This verifies its role in enforcing pose consistency between LRGI and HRGT.
  • D Ablation of DipRefGC: RefGC-oriented fine-tuning achieves higher DINO and CLIP-I scores than the Canny-only variant in both compositing-only and customization-only settings.The improvement indicates stronger identity preservation after RefGC-oriented fine-tuning.
  • D Ablation of DipRefGC: Phase 1 trains on 40K ORIDa and uCO3D triplets, while Phase 2 adds 12K UltraVideo triplets with richer subject motion and reduces FID in both tasks.The additional fine-tuning benefits from more challenging motion-rich data while preserving pose consistency.

E User Study Protocols and Analysis · F User Study Details · F.1 Protocol

The user study used a blind, side-by-side web questionnaire to compare four anonymized methods across 20 in-the-wild cases. Participants rated refinement, detail, and overall quality on a 4-point Likert scale, with top-1 and bottom rates computed from 16 participants × 20 questions.

  • F.1 Protocol: The study was voluntary, uncompensated, and conducted through a web-based Google Forms questionnaire taking approximately 20 minutes per participant.Participants consented after receiving the study purpose and duration, could withdraw at any time, and provided no personally identifiable information.
  • F.1 Protocol: The questionnaire contained 20 questions, each corresponding to one in-the-wild test case, with HRRI, HRGT, and LRGI reference images plus four anonymized model outputs.The outputs represented DiT4SR, AdaRefSR, ImageCritic, and the RefGC-SR2 model.
  • F.1 Protocol: Participants independently rated all four outputs on refine, detail, and overall quality using a 4-point Likert scale from Very Poor to Excellent.Refinement measured artifact or undesired-structure removal; detail quality measured restoration of fine details; overall quality emphasized visual quality and reference alignment.
  • F.1 Protocol: The refine criterion assessed removal of undesired structures or artifacts deviating from the HRGT or HRRI.This criterion focused specifically on generative-artifact correction relative to the high-resolution references.
  • F.1 Protocol: Our model dominated the best rank at 82–83% and was rarely ranked worst at 3–4%, whereas ImageCritic had the highest worst-rank rates of 51%/66%/54%.Figure 14 reports these top-1 and bottom rates across the three evaluation criteria and attributes ImageCritic’s poor worst-rank performance to oversmoothing.
  • F.1 Protocol: Detail quality assessed fine-detail restoration from HRGT and HRRI, while overall quality emphasized visual quality and alignment with both references.These criteria were presented alongside the task instructions in the questionnaire interface.
  • F.1 Protocol: Top-1 rate measured the proportion of participant-question pairs where a model received rank 4, while bottom rate analogously measured rank 1.Results were aggregated across 16 participants × 20 questions and reported in Fig. 7 and Fig. 14.

F.2 Detailed Analysis

Human evaluations show that the proposed model strongly dominates baselines in preference rankings, achieving high top-1 rates and low worst-rank rates across all criteria. ImageCritic is a notable exception to its quantitative performance, receiving the worst human-evaluation rank most often despite ranking second-best quantitatively.

  • Overall preference: Our model achieves an 82–83% top-1 rate across all three criteria, while baselines remain below 9%.The full rank distribution confirms that our model dominates rank 4, the best rank, across all criteria.
  • Overall preference: Our model is ranked worst in only 3–4% of cases, whereas baselines have substantially higher worst-rank rates.This symmetric pattern reinforces the model’s preference dominance across the evaluated criteria.
  • Discrepancy with quantitative metrics for ImageCritic: ImageCritic receives the worst rank in 51%, 66%, and 54% of refine, detail, and overall-quality cases, respectively, despite ranking second-best quantitatively.Its worst-rank rates are the highest among all compared models, revealing a discrepancy between quantitative and human evaluations.

G Limitations … H.3 Overall Objective

The paper’s limitations concern dataset coverage, backbone dependence, and reference–output geometry gaps. Its training objective combines flow-matching, frequency-based, and attention-alignment losses to supervise generation, frequency transfer, and reference-focused attention.

  • G Limitations: The synthesized dataset covers object-centric scenes in 12 categories, limiting artifact diversity and broader domains such as humans or complex scenes.It is generated by DipRefGC rather than directly sampled from real RefGC pipelines.
  • G Limitations: The model is tied to the FLUX-Kontext backbone and may struggle when HRRI and LRGI have large viewpoint or geometry gaps.Future work includes collecting LRGIs directly from diverse real RefGC pipelines.
  • H Details on Auxiliary Loss Terms: The overall objective combines the proposed frequency-based loss Lf with FLUX-Kontext’s flow-matching loss LF M and ImageCritic’s attention-alignment loss Laal.The latter two are auxiliary terms inherited or adopted from existing methods.
  • H.1 Flow-Matching Loss (LF M): Flow matching supervises the diffusion transformer to predict the velocity transporting Gaussian noise toward the HRGT data distribution along a linear trajectory.The clean latent z1 comes from HRGT, while z0 is sampled from N(0, I).
  • H.1 Flow-Matching Loss (LF M): LF M is the same objective used for LoRA fine-tuning of FLUX-Kontext and provides primary supervision for outputs aligned with HRGT.The model predicts velocity at zt conditioned on LRGI, HRRI, and text-instruction inputs.
  • H.2 Attention Alignment Loss (Laal): Laal supervises cross-attention between output tokens and HRRI tokens, encouraging attention to the reference object region instead of irrelevant background.This regularization promotes faithful transfer of reference information to the refined output.
  • H.3 Overall Objective: The full objective is L = LF M + Lf + Laal, with complementary roles in generative supervision, attention localization, and frequency-selective reference transfer.Lf transfers low-frequency information from HRGT and high-frequency information from HRRI, consistent with FreqMoLE’s frequency-specialized LoRA experts.

I Broader Impacts

The paper presents RefGC-SR2 as a post-processing task that jointly performs super-resolution and artifact refinement, while discussing benefits for faithful personalization alongside misuse risks and mitigations. Key concerns include deceptive media, identity manipulation, and amplification of problematic upstream pipelines.

  • Core impact: RefGC-SR2 jointly improves resolution and refines artifacts in reference-guided generated content.The task is designed as a post-processing step.
  • Positive impacts: The method can help users obtain higher-quality outputs that faithfully preserve details from their own high-resolution reference images.The paper identifies personalized image editing and e-commerce visualization as practical applications.
  • Negative impacts: Higher-quality refinement could make synthetic imagery harder to distinguish from photographs, enabling misuse for misinformation, fake profiles, or fabricated scenes.The risk is amplified because the dataset includes human-centric categories.
  • Negative impacts: Fine-grained transfer of identity-specific details could be repurposed for non-consensual identity content.The paper frames this as a dual-use risk of personalization.
  • Negative impacts: As a post-processing stage, RefGC-SR2 could unintentionally amplify problematic outputs from upstream reference-guided generation pipelines.The model may be chained as a quality-enhancement stage when upstream systems are used for harmful purposes.
  • Mitigations: The authors mitigate risks by releasing DipRefGC and the RefGC-SR2 model under a research-only license with documentation covering intended use, limitations, and human-centric categories.They also recommend downstream content-provenance mechanisms such as visible or invisible watermarks.
Loading 2606.15158v2…