PaperlayerView exact revision on arXiv

Source-linked AI summary

LucidFlux: Caption-Free Photo-Realistic Image Restoration via a Large-Scale Diffusion Transformer

Song Fei, Tian Ye, Lujia Wang, Lei Zhu

arXiv:2509.22414v4cs.CV

TL;DR

Restoration in the wild must handle unknown degradation mixtures while preserving perceptual fidelity and semantic consistency. LucidFlux adapts Flux.1 with lightweight, caption-free conditioning and curated supervision. It reports state-of-the-art performance across a broad benchmark suite, while high-quality outputs typically require more than about 15 denoising steps.

  • Problem

    In-the-wild restoration must recover images under unknown degradation mixtures while preserving perceptual fidelity and semantic consistency without degradation labels or side information.

    Source: §1 · p. 2

  • Method

    LucidFlux adapts Flux.1 with a lightweight dual-branch conditioner, timestep- and layer-adaptive modulation, proxy-derived SigLIP alignment, and curated training data.

    Source: §1 · p. 3

  • Results

    LucidFlux reports state-of-the-art results across a broad suite of benchmarks and metrics, surpassing competitive open- and closed-source baselines.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The paper concludes that when, where, and what to condition on governs robust caption-free restoration with large DiTs, rather than adding parameters or relying on prompts.

    Source: §5 · p. 10

    High-quality outputs typically require more than about 15 denoising steps, creating a latency–quality trade-off that can restrict real-time or interactive use.

    Source: §A.10 · p. 21

Abstract

from arXiv · show

Image restoration (IR) aims to recover images degraded by unknown mixtures while preserving semanticsconditions under which discriminative restorers and UNet-based diffusion priors often oversmooth, hallucinate, or drift. We present LucidFlux, a caption-free IR framework that adapts a large diffusion transformer (Flux.1) without image captions. Our LucidFlux introduces a lightweight dual-branch conditioner that injects signals from the degraded input and a lightly restored proxy to respectively anchor geometry and suppress artifacts. Then, a timestep- and layer-adaptive modulation schedule is designed to route these cues across the backbones hierarchy, in order to yield coarse-to-fine and context-aware updates that protect the global structure while recovering texture. After that, to avoid the latency and instability of text prompts or Vision-Language Model (VLM) captions, we enforce caption-free semantic alignment via SigLIP features extracted from the proxy. A scalable curation pipeline further filters large-scale data for structure-rich supervision. Across synthetic and in-the-wild benchmarks, our LucidFlux consistently outperforms strong open-source and commercial baselines, and ablation studies verify the necessity of each component. LucidFlux shows that, for large DiTs, when, where, and what to condition onrather than adding parameters or relying on text promptsis the governing lever for robust and caption-free image restoration in the wild.

1 INTRODUCTION

LucidFlux adapts a large diffusion transformer for restoration under unknown real-world degradations, where preserving perceptual fidelity and semantic consistency remains challenging. Its lightweight, caption-free conditioning and curated data pipeline are designed to align guidance with the backbone’s hierarchy, and the framework reports state-of-the-art results across a broad benchmark suite.

1 INTRODUCTION

  • Real-world restoration remains difficult because unknown mixtures of noise, blur, lens aberrations, and compression can reduce fidelity and induce semantic drift without degradation labels or side information.Discriminative restorers can oversmooth textures or leave artifacts on in-the-wild mixtures, while UNet diffusion priors can struggle to recover detail and preserve global structure.Source: §1 · p. 2
  • Large diffusion transformers offer capacity for mixed-degradation restoration, but direct transfer can fail and prior ControlNet-style conditioning may underuse temporal and hierarchical roles.Public web corpora also lack suitable paired, structure-rich coverage of long-tail real degradations, motivating explicit data curation.Source: §1 · pp. 2–3
  • A lightweight dual-branch conditioner uses the degraded input to anchor geometry and a lightly restored proxy to suppress artifacts, with timestep- and layer-adaptive modulation routing guidance for coarse-to-fine updates.The updates are intended to preserve texture while protecting global structure.Source: §1 · p. 3
  • Caption-free SigLIP alignment extracts semantic cues from the proxy, avoiding text prompts and their latency and semantic-drift risks.The framework also pairs this alignment with an automated three-stage curation pipeline for diverse training data.Source: §1 · p. 3
  • LucidFlux sets new state-of-the-art results across a broad suite of benchmarks and metrics, surpassing competitive open- and closed-source baselines.Ablations confirm the necessity of each module.Source: §1 · p. 3

2 RELATED WORK

Prior generative restoration methods use varied diffusion backbones, but large text-to-image transformers remain underexploited for restoration. LucidFlux adapts the 12B Flux.1 DiT with lightweight dual-branch, caption-free conditioning.

Generative Priors for IR

  • Existing restoration systems span UNet-based diffusion models and relatively small, task-specific DiTs, often with heavy ControlNet-style adapters; LucidFlux instead adapts the 12B Flux.1 DiT.The paper positions this as a way to more fully exploit modern large-scale text-to-image transformers for restoration.Source: §2 · p. 3

Semantic Alignment

  • VLM-generated captions add inference cost and can encode degradation-related terms that bias restoration toward artifacts; LucidFlux uses proxy-derived SigLIP features for caption-free semantic guidance.The features are projected through a lightweight Connector into Flux.1’s text-conditioning space.Source: §2 · p. 4

3 METHODOLOGY

LucidFlux pairs degraded and lightly restored inputs as complementary controls, then routes restoration and semantic cues through the Flux DiT across timesteps and layers. Its methodology also includes a scalable pipeline for curating structure-rich training data.

3.1 LIGHTWEIGHT DUAL-BRANCH CONDITIONER

  • The dual-branch conditioner converts the degraded image and a lightly restored proxy into separate signals, balancing detail-preserving cues against artifact-suppressed structure.The degraded input anchors geometry, while the proxy suppresses hard artifacts.Source: §3.1 · pp. 4–5

3.2 TIMESTEP- AND LAYER-ADAPTIVE CONDITION MODULATION

  • A lightweight modulation head uses timestep and layer position to adapt each branch’s per-channel scale and bias, enabling coarse-to-fine guidance without increasing conditioner capacity.Independent modulation preserves the branches’ complementary detail-preserving and artifact-robust cues.Source: §3.2 · p. 5

3.3 SIGLIP FOR CAPTION-FREE SEMANTIC ALIGNMENT

  • Frozen SigLIP extracts semantics from the restored proxy, and a lightweight connector maps them into Flux’s text-embedding space without captions.The semantic pathway removes external captions at both training and inference.Source: §3.3 · p. 6

3.4 SCALING UP REAL-WORLD HIGH-QUALITY DATA FOR IMAGE RESTORATION

  • Because existing restoration datasets are modest or lack reproducible quality control, LucidFlux uses an automatic three-stage filter to retain structure-rich, high-quality images.The stages screen blur, suppress flat regions, and rank perceptual quality.Source: §3.4 · p. 6
  • Blur screening retains images with 150 ≤ Sblur(I) ≤ 8000, while flat-region filtering removes images with more than 50% of 240 × 240 patches below edge-richness threshold 800.The thresholds are empirically set; the flat-region criteria aim to suppress large flat backgrounds while preserving natural sky and water regions.Source: §3.4 · pp. 6–7
  • Compared with Flickr2K and DIV2K, the curated dataset has higher CLIP-IQA scores, lower flatness, and more diverse resolutions.Its blur scores are comparable to those datasets.Source: Figure 3 · p. 7, §3.4 · p. 7
  • CLIP-IQA ranking retains the top 20% of remaining images, producing 257K images; adding 84K LSDIR samples yields 342K images and 1.36M degraded training pairs.The pairs are generated with the Real-ESRGAN degradation pipeline across four epochs.Source: §3.4 · p. 7

4 EXPERIMENT

LucidFlux performs strongly on perceptual and semantic-oriented metrics across real-world and synthetic benchmarks, with reported trade-offs on synthetic distortion metrics. Its caption-free conditioning, runtime design, and curated-data ablations are evaluated alongside the benchmark comparisons.

4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • LucidFlux outperforms prior approaches on perceptual and semantic-oriented metrics and scores highest across the compared commercial methods.Prior approaches report slightly higher PSNR and SSIM on synthetic datasets.Source: Table 1 · p. 8, §4.2 · pp. 9–10, Table 2 · p. 10

4.1 IMPLEMENTATION DETAILS

  • Evaluations cover synthetic and real-world benchmarks at 1024 × 1024 resolution, using seven no-reference and three reference-based image-quality metrics.Training freezes the Flux backbone and trains only task-specific modules; it uses 8 NVIDIA A800 GPUs and completes in approximately 7 GPU-days.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • On RealLQ250, LucidFlux restores clearer edges and richer textures in challenging regions such as hair, text, and high-frequency patterns.SeeSR, DreamClear, and SUPIR tend to leave artifacts or produce oversmoothed outputs with limited texture or detail recovery.Source: §4.2 · p. 8
  • Despite its 12B backbone, LucidFlux achieves competitive total runtime by eliminating caption preprocessing.Its trainable adapters total 1.6B parameters.Source: §4.2 · p. 9, Table 3 · p. 10

4.3 ABLATION STUDY

  • Adding SigLIP alignment, timestep- and layer-adaptive modulation, and curated data successively improves three RealLQ250 IQA scores, with curated data providing the largest jump.The DBC-only CLIP-IQA / CLIP-IQA+ / MUSIQ scores are 0.585/0.609/61.582.Source: Table 4 · p. 10, §4.3 · p. 10

5 CONCLUSION

LucidFlux adapts a large diffusion transformer for caption-free restoration through carefully routed conditioning and proxy-based semantic alignment. It combines curated data and this design to achieve strong perceptual quality and semantic fidelity across real and synthetic benchmarks.

5 CONCLUSION

  • The conclusion frames when, where, and what to condition on as the key design lever for caption-free restoration with large diffusion transformers.It contrasts this emphasis with adding parameters or relying on prompts.Source: §5 · p. 10
  • A dual-branch conditioner and adaptive modulation recover high-frequency detail while preserving global structure and suppressing artifacts with a frozen Flux.1 backbone.The branches use the degraded input and a lightly restored proxy; SigLIP-based semantics provide training–inference consistency without captions.Source: §5 · p. 10
  • The automatic curation pipeline scales to 342K high-quality images and 1.36M paired samples for structure-rich supervision.The conclusion presents the data recipe and pipeline as resources for large-DiT posttraining.Source: §5 · p. 10
  • Across real and synthetic benchmarks, LucidFlux delivers state-of-the-art perceptual quality and semantic fidelity with competitive runtime and minimal trainable overhead.The conclusion points to learned data selection, multi-frame/video extensions, and higher-resolution backbones as future directions.Source: §5 · p. 10

A.1 LIKELIHOOD OF DEGRADATION-RELATED TERMS IN CAPTIONS GENERATED BY DIFFERENT MULTIMODAL LARGE LANGUAGE MODELS

The appendix measures how often VLM captions mention image degradations and describes the evaluation procedure. It reports different occurrence rates for two VLMs and notes that degradation-related descriptions can misguide restoration.

A.1 LIKELIHOOD OF DEGRADATION-RELATED TERMS IN CAPTIONS GENERATED BY DIFFERENT MULTIMODAL LARGE LANGUAGE MODELS

  • On RealLQ250, captions from LLaVA-v1.6-Vicuna-13B and Qwen2.5-VL-7B-Instruct are evaluated for degradation-related mentions using Gemini-2.5-Flash-Image as an external evaluator.Source: §A.1 · p. 15
  • 17% of LLaVA-v1.6-vicuna-13b captions and 24% of Qwen2.5-VL-7B-Instruct captions contain degradation-related terms.The paper defines these captions as ones explicitly mentioning blur, noise, low resolution, or compression artifacts.Source: §A.1 · p. 15, Table 5 · p. 16
  • Figure 5 shows that captions with explicit degradation descriptions can misguide restoration and yield lower perceptual quality than captions focused on content and style.The appendix also reports output variation across runs using captions from the same VLM.Source: Figure 5 · p. 16, §A.1 · p. 16

A.2 IMPACT OF DEGRADATION-RELATED CAPTIONS ON MODEL RESTORATION

The appendix compares neutral and degradation-explicit VLM captions and reports effects on restoration. Caption variability and degradation-related descriptions are associated with inconsistent or lower-quality outputs, while caption generation adds inference overhead.

A.2 IMPACT OF DEGRADATION-RELATED CAPTIONS ON MODEL RESTORATION

  • LLaVA captions use either a neutral prompt about key subjects and style or a prompt that explicitly retains degradation descriptions.The comparisons examine caption variability and degradation-related content as factors in restoration outputs.Source: §A.2 · p. 16
  • Across qualitative results, same-VLM caption variability yields different restoration outputs, while degradation-explicit captions produce lower perceptual quality.The authors also identify caption-generation overhead and potential degradation bias as practical limitations of VLM guidance.Source: §A.2 · p. 16
  • With a fixed Flux.1 backbone, the caption-usage ablation contrasts GT captions, inference-time VLM captions, and a caption-free pathway.The table defines latency as additional inference overhead.Source: Table 6 · p. 17

A.3 ABLATION ON CAPTION USAGE FOR A CLEAN ATTRIBUTION OF THE CAPTION-FREE DESIGN

The caption ablation finds that VLM captions add inference latency without improving image quality over LucidFlux’s caption-free variant.

A.3 ABLATION ON CAPTION USAGE FOR A CLEAN ATTRIBUTION OF THE CAPTION-FREE DESIGN

  • The caption-free model achieves the best MUSIQ (70.2005), NIMA (5.1072), and NIQE (4.2893), while caption-based settings add ∼10 seconds per image.Even captions generated from the ground-truth reference do not yield superior image quality.Source: §A.3 · p. 17

A.4 COMPARISON OF BACKBONE UPDATE STRATEGIES

Frozen-backbone adaptation yields the strongest reported perceptual performance among the tested DiT update strategies while maintaining stable optimization.

A.4 COMPARISON OF BACKBONE UPDATE STRATEGIES

  • CLIP-IQA 0.7122 / MUSIQ 73.01: frozen-backbone adaptation outperforms attention-only updates and full fine-tuning while maintaining stable optimization.Attention-only updates reach CLIP-IQA 0.654 / MUSIQ 66.27; full fine-tuning reaches CLIP-IQA 0.594 / MUSIQ 63.30.Source: §A.4 · p. 17

A.5 ADDITIONAL RELATED WORKS

LucidFlux’s data pipeline targets limitations in existing restoration datasets, while the method builds on transformer-based generative priors and specializes Flux.1 for caption-free restoration.

A.5 ADDITIONAL RELATED WORKS

  • LucidFlux uses automated blur, flat-region, and perceptual-quality filtering to produce reproducible, scalable, structurally rich data for billion-parameter diffusion backbones.The passage contrasts this with LSDIR’s manual filtering, SUPIR’s undisclosed quality control, and DreamClear’s 1280 V100 GPU-day generation cost.Source: §A.5 · p. 17
  • Transformer-based text-to-image models include diffusion and rectified-flow variants, but the supplied passage gives only a partial account of this background.The excerpt identifies scaling and long-range dependencies as properties of these Transformer backbones.Source: §A.5 · p. 17
  • Diffusion and rectified-flow Transformers capture long-range latent dependencies; LucidFlux specializes the Flux.1 MM-DiT for caption-free restoration.The passage also describes SD3’s multimodal transformer and PixArt-α’s training recipe, but the provided text gives no further detail on the truncated background passage.Source: §A.5 · p. 18

A.6 EXTENDED DATASET ANALYSIS

The extended analysis examines semantic coverage and image quality across curation settings, adds qualitative comparisons and inference details, and notes practical limitations and the authors’ use of language models.

A.6 EXTENDED DATASET ANALYSIS

  • LucidFlux’s filtered data spans a broader semantic range than Flickr2K and DIV2K, indicating richer image–text diversity.The analysis visualizes embeddings from 10K filtered images and all available images from Flickr2K and DIV2K.Source: §A.6 · p. 18, Figure 6 · p. 18

A.6.1 ANALYSIS OF FILTERING SETTINGS AND SEMANTIC COVERAGE

  • 234,287 samples remain under the baseline filtering configuration; at fixed sample size, threshold variants yield similar semantic coverage, while increasing sample size expands it.The pipeline applies blur, flat-region, and IQA filtering sequentially; stricter thresholds reduce retained samples and relaxed thresholds increase them.Source: §A.6.1 · pp. 18–19

A.6.2 IMPACT OF FILTERING SETTINGS ON IMAGE QUALITY METRICS

  • The baseline configuration scores highest on nearly all seven no-reference quality metrics, while stricter or relaxed thresholds can reduce specific quality indicators.Selecting the top 10% by CLIP-IQA lowers QAlign from 4.39 to 3.98 and MUSIQ from 73.01 to 69.88; Flat Variant (Set 4) has MANIQA 0.4973 and NIQE 3.35.Source: §A.6.2 · p. 19, Table 9 · p. 19

A.7 EXTENDED VISUAL COMPARISONS

  • Extended comparisons across RealLQ250, DRealSR, RealSR, DIV2K-Val, and LSDIR-Val report sharper edges, more faithful textures, and better semantic-structure preservation than open-source SOTA methods.Figures 9–14 provide representative examples across these benchmarks.Source: §A.7 · p. 20, Figure 9 · p. 23, Figure 11 · p. 25

A.8 VISUAL COMPARISON WITH CLOSE-SOURCE COMMERCIAL METHODS

  • On RealLQ250, LucidFlux delivers the sharpest structures and most faithful fine-grained details among the evaluated commercial methods, while maintaining structural fidelity and reliability.The passage contrasts this with residual artifacts from HYPIR-FLUX and Seedream 4.0, over-smoothed textures from Topaz, and difficulty recovering high-frequency details from Gemini-NanoBanana.Source: §A.8 · pp. 20–21

A.9 INFERENCE DETAILS

  • Inference uses 28 FlowMatch Euler sampling steps in FP16 and applies wavelet color alignment.The stated default instruction is to restore the image into a high-quality, clean, high-resolution result.Source: §A.9 · p. 21

A.10 LIMITATIONS

  • Practical limits include substantial model compute, resolution- and batch-dependent inference memory, and a latency–quality trade-off because high-quality outputs typically require more than ~15 denoising steps.The authors identify compression, low-precision inference, memory-efficient attention, and progressive distillation as possible mitigations.Source: §A.10 · p. 21

A.12 LLM USAGE

  • The authors report using large language models only for editorial assistance; technical content, experiments, analyses, and conclusions were authored and validated by them.They state that no ideas, methods, code, figures, citations, or results were generated by an LLM.Source: §A.12 · p. 22
Loading 2509.22414v4…