Source-linked AI summary
RRFC: Recursive Refinement via Feedback Conditioning for Iterative Image-to-Image Generation
Kareem Hassani, Chaymaa Abbas, Hadi Al Mubasher, Mariette Awad
TL;DR
Conditional image-to-image generators typically finalize outputs after one pass, limiting sample-specific correction. RRFC feeds each prediction back as conditioning for iterative refinement, helping mainly on fidelity- and identity-driven tasks while often degrading semantic-layout performance.
Problem
One-shot conditional generators lack a structured mechanism to inspect their outputs and correct sample-specific errors through targeted refinement.
Method
RRFC feeds a generator’s previous prediction back as extra input channels alongside the original condition, preserving its core architecture while enabling learned self-refinement.
Results
Across 18 architecture-task settings, RRFC significantly improves seven, degrades seven, and leaves four without reliable change, with gains concentrated on fidelity and identity tasks.
Takeaways & Limitations
RRFC’s benefit depends on overlap between its refinement objective and the evaluation metric, making it most suitable for fidelity- and identity-driven tasks.
Takeaways & Limitations
RRFC comparisons match data and training schedules but not computation, so reported effects include the additional cost of unrolling and repeated inference.
Abstract
from arXiv · showhide
Conditional image-to-image generators are single-shot: they map input features to an output in one forward pass and treat it as final, with no opportunity to improve on it. Although trained to produce the best possible result in one step, such a model leaves room for improvement if it can adaptively revise its own output over iterations. We propose Recursive Refinement via Feedback Conditioning (RRFC), a novel feedback-conditioning framework for iterative output refinement that teaches a model to adaptively revise its output by conditioning on a new signal, namely its most recent previous prediction, which is fed back as an auxiliary set of channels alongside the original input. This preserves the generator's core architecture while modifying its conditioning interface and, depending on the model family, its training or inference procedure, so RRFC can be attached to existing generators without redesign. We evaluate RRFC across six baselines spanning adversarial, equilibrium, and diffusion-based models and three paired image-to-image translation tasks. Across 18 architecture-task settings, RRFC yields seven Holm-corrected improvements, seven degradations, and four non-significant changes. The gains concentrate on reconstruction-fidelity and identity settings, while five of the seven degradations fall on the single semantic-layout task, where every model declines. These results indicate that feedback-based refinement helps when its objective overlaps with the evaluated property, and that its gains concentrate on the tasks where that overlap holds.
1 Introduction
RRFC addresses the one-shot limitation of conditional image-to-image generators by feeding each model’s previous prediction back alongside the original input, teaching refinement as a behavior without redesigning the generator’s core architecture. The paper evaluates this intervention across six architectures, three paired tasks, and three seeds to identify where self-refinement helps.
- Motivation: Conditional generative models typically produce one final output and lack a structured mechanism to inspect local errors and refine an individual sample.Existing improvements occur offline through parameter updates across the training distribution rather than targeted correction of a generated image.
- RRFC: RRFC feeds a model’s previous prediction back as extra input channels alongside the original condition, teaching it to revise its output.The framework treats refinement as learned behavior rather than architecture-specific wiring.
- RRFC: RRFC is designed as an architecture-adaptable framework that can attach to one-pass generators and models with internal iterative processes.This distinguishes the proposal from prior refinement methods built into a specific model for a specific task.
- Study: 6 architectures × 3 tasks × 2 variants × 3 seeds define the controlled comparison of models with and without RRFC.The study uses paired significance testing with multiple comparison correction and evaluates both general image-quality metrics and task-specific metrics.
- Study: The evaluation asks where self-refinement helps rather than claiming simultaneous improvement on every metric.The paper notes that target properties differ across tasks, motivating task-sensitive assessment.
2 Related Work
Prior refinement methods typically rely on architectures designed for specific tasks or operate within a single sampler. RRFC instead feeds completed outputs between full generations, addresses feedback mismatch, and treats internal refinement as a predictor of when outer refinement may duplicate computation.
- Task-specific refinement: Multi-pass refinement is established, but usually belongs to an architecture built for one task, including cascaded, stacked adversarial, and iterative adversarial designs.These approaches include coarse-to-fine synthesis from semantic layouts, defect correction after low-resolution generation, and repeated calls to a single generator.
- Feedback conditioning: RRFC returns a completed output as image channels between full generations and pairs this feedback with an objective constraining successive outputs.Unlike self-conditioning in diffusion samplers or latent-token warm starts, RRFC makes feedback available to models without an internal forward-pass trajectory.
- Train-test mismatch: Training on feedback absent at test time creates the same history mismatch recognized in sequence prediction and diffusion sampling.Prior diffusion work addresses accumulated self-error by rescaling predictions or shifting the sampling schedule.
- Internal iteration: Internal refinement predicts whether RRFC’s outer loop will duplicate computation: equilibrium and diffusion-sampling models already iterate internally during inference.This connects RRFC to inference-computation allocation methods such as learned halting.
- Quality constraints: Non-decreasing quality has appeared as a surrogate, a post-hoc conditional-monotonicity construction, or an emergent effect of larger sampling budgets rather than an enforced generative constraint.Recent plug-in training proposals share part of RRFC’s setting without occupying it.
- Study baselines: The study compares one-pass Pix2Pix, Pix2PixHD, and SAGAN with internally iterative DEQ, ControlNet, and Palette across Cityscapes, Places365, and CelebA-HQ protocols.The listed models span the two generator families defined in the related-work discussion.
3 Methodology: RRFC
RRFC converts single-shot conditional generators into iterative refiners by feeding each prediction back as auxiliary input and training successive outputs to improve. Its four components adapt to the base architecture while preserving the generator’s core design.
- Core components: RRFC feeds the previous output back as extra input channels, using a blank signal on the first pass and a detached per-sample buffer during training.The buffer stores each sample’s most recent output across epochs as the training-time estimate of the previous prediction.
- Core components: RRFC unrolls training for U = 3 passes, supervises every pass against the target with later passes weighted more, and propagates gradients where the base permits.Previous outputs entering reconstruction losses and earlier quality terms in the monotonic penalty are detached.
- Core components: The monotonic objective penalizes any refinement pass that is worse than its predecessor, using a differentiable quality measure matched to the architecture’s existing objective.The quality measure can combine pixel and perceptual terms or use an architecture-specific objective such as adversarial realism or denoising.
- Architecture adaptation: Across model families, self conditioning and exposure-matched feedback remain identical, while unrolled supervision and the monotonic objective adapt to single-pass or internally iterative generation.Single-pass models unroll training with gradients through passes; internally iterative models avoid infeasible stacked backpropagation through their internal loops.
4 Experimental Setup
The experiments compare six baseline models with and without RRFC across three paired image-to-image tasks, using task-specific and general image-quality metrics. Training, configuration selection, and evaluation use shared budgets and validation-only decisions, with paired statistical testing across 18 settings.
- Models: Six baselines—Pix2Pix, Pix2PixHD, SAGAN, DEQ, ControlNet, and Palette—are evaluated with and without RRFC, yielding 18 model-task settings.The baselines span single-forward-pass and internally iterative families.
- Tasks and data: Three paired tasks target semantic agreement, reconstruction fidelity, and identity preservation: mask-to-photo on Cityscapes, inpainting on Places365, and sketch-to-face on CelebA-HQ.Their primary metrics are mIoU, in-hole PSNR, and ArcFace cosine similarity, respectively.
- Metrics: Each task reports one primary metric alongside LPIPS, PSNR, SSIM, FID, and KID, with KID treated as directional evidence because its estimator variance can be substantial with only 200 test images.KID is the headline distributional metric.
- Training protocol: Every model version uses 1500 training and 500 validation images per task, except for 347 eligible T1 validation images, trained for 15 epochs with Adam and three seeds.Batch size is the only per-model difference and remains constant across tasks; paired differences, rather than absolute scores, are interpreted.
- Configuration selection: A single RRFC recipe is selected on validation data only, while checkpoints and refinement depth are also selected without test-set access.Per-image metrics are compared with paired two-sided Wilcoxon signed-rank tests and Holm correction across the 18 settings; bootstrap confidence intervals are reported for median change.
5 Results
Across 18 architecture-task settings, RRFC significantly improves seven cells, degrades seven, and leaves four unchanged. Benefits concentrate on inpainting and identity recovery, while semantic layout consistently declines despite improved fidelity metrics.
- Overall verdict: 7 improvements, 7 degradations, and 4 non-significant changes occur across all 18 architecture-task cells, with effects organized largely by task rather than architecture.Five of the seven significant degradations occur on the semantic task, where all six effects are negative and five are Holm-corrected significant.
- Reconstruction realism: Every inpainting metric moves favorably, including a 33.0 percent KID improvement and a 6.6 percent LPIPS improvement on Places365.In-hole PSNR significantly improves for four of six models, while FID improves for five of six models.
- Identity recovery: ArcFace identity significantly increases in three of six sketch-to-face models, by 13.0 percent on Pix2Pix, 11.2 percent on SAGAN, and 29.8 percent on ControlNet.Absolute identity scores remain near 0.10 to 0.13, below a strict same-person threshold.
- Refinement stability: Quality rises monotonically through refinement passes and remains flat to ten passes on favorable tasks, although training used an unroll of only three.Most of the gain arrives by three to five passes.
- Semantic layout: On Cityscapes mask-to-photo, the primary metric falls in all six models, by 7.6 percent on Pix2Pix and up to 34.9 percent on Palette.Aggregate structural similarity nevertheless improves by 20.2 percent and PSNR by 2.5 percent, indicating misalignment between the refinement objective and semantic evaluation.
- Cost: Three to five GAN refinement passes remain below one second per image, whereas diffusion refinement is an order of magnitude more expensive because each pass is a full sampler run.The selected depth collapses to zero where diffusion refinement does not help.
6 Analysis
RRFC helps most when its fidelity-oriented refinement objective overlaps with the evaluated property, while semantic-layout performance can decline. Its effects are driven more by task and model responsiveness than by internal iteration, primarily reshaping structure and distributional similarity rather than pixel error.
- Objective-metric alignment: Objective-metric alignment explains RRFC’s dominant pattern: effects are more favorable when the refinement objective overlaps with the evaluated property.This association is not universal at the individual architecture-task level.
- Objective-metric alignment: Every metric improves on inpainting, while semantic-task mIoU falls despite improvements in structural fidelity metrics.Figure 5 reports that inpainting improves most strongly on KID, whereas the semantic task improves in SSIM and PSNR while its primary metric declines.
- Model responsiveness: 0.85, 0.69, 0.65, 0.48, and 0.32 are the responsiveness scores for Palette, Pix2Pix, SAGAN, Pix2PixHD, and ControlNet, respectively.Palette is the most responsive baseline, while ControlNet is among the least; internally iterative models nevertheless show mixed, task-dependent effects.
- Model responsiveness: Single-pass GAN models are consistently responsive and account for most significant improvements, whereas internal iteration does not reliably predict immunity to RRFC.The absence of internal refinement is a reliable indicator of responsiveness, but its presence is not a reliable indicator of immunity.
- What refinement changes: KID and SSIM are the largest-moving metrics, SSIM changes significantly in all eighteen cells, and full-image pixel error moves least.RRFC is therefore characterized as reshaping structure and moving outputs toward the real image distribution rather than reducing average pixel error.
7 Discussion and Conclusion
RRFC transfers across generators, but its benefit is not universal: the refinement objective’s overlap with the evaluation metric governs the dominant effect. Interpretation is limited by the reduced training budget and unmatched computation.
- Discussion and Conclusion: RRFC transfers across generators as an architecture-adaptable conditioning principle, but its aggregate benefit is predictable rather than universal.The dominant effect depends on overlap between the refinement objective and the evaluation metric.
- Discussion and Conclusion: RRFC is well suited to single-pass generators on fidelity- and identity-driven tasks, improving the measured property at the cost of a small multiple of forward passes.
- Limitations and future work: All models use one reduced training budget, so the study interprets paired before-after differences rather than absolute performance, with identity gains arising from a low operating point.
- Limitations and future work: The matched arms differ in computation, so the reported effect reflects the complete RRFC intervention rather than separating feedback conditioning from its additional computation.
A Appendix
The appendix supports the adopted RRFC recipe, U3·Fr55·M50, as a robust search winner while detailing which components and task settings drive its effects. Refinement is most beneficial for inpainting and identity, but consistently harms the semantic task, with gains not attributable to test-set access.
- Recipe selection: U3·Fr55·M50 ranks first by mean primary-metric rank at 3.67, ahead of the runner-up at 5.00.It is never worse than sixth across the six search cells and is third closest to the cell-wise optimum at 0.82 percent.
- Component ablations: Unrolled supervision is the most important RRFC component, especially on the two aligned tasks.Self-conditioning matters most on inpainting, while exposure matching and the monotonic term have smaller, mixed effects.
- Per-image effects: Inpainting shows the strongest per-image fidelity gains, with Pix2Pix and SAGAN improving in-hole PSNR on about four in five images.The semantic task shows the contrasting pattern that SSIM improves while its primary metric declines.
- Selection robustness: +0.024 is the average inflation from choosing refinement depth with test-set access, and the largest gap remains negative for T2 ControlNet.The reported gains therefore do not depend on test-set access.
- Task and depth effects: Every model declines on the semantic task, five of six improve on inpainting, and most improve on identity.ControlNet is the inpainting exception, while Pix2PixHD is the identity exception.