PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeStyle: Free Control of Style-Content Dual-Reference Generation from Community LoRA Mining

Jinghong Lan, Wei Cheng, Yunuo Chen, Ziqi Ye, Peng Xing, Yixiao Fang, Rui Wang, Yufeng Yang, Xuanyang Zhang, Xianfang Zeng, Difan Zou, Gang Yu, Chi Zhang

arXiv:2606.20506v1cs.CVcs.AI

TL;DR

Dual-reference image generation must preserve a content reference while adopting a separate style, yet clean, large-scale triplets with broad style coverage are lacking. FreeStyle mines community LoRAs and uses stage-specific training constraints; it ranks first on style metrics across both benchmarks while maintaining competitive performance on other axes.

  • Problem

    Dual-reference generation must preserve content structure and semantics while adopting a separate style and following instructions, but large-scale triplets with clean separation and broad style coverage are lacking.

    Source: §1 · p. 2

  • Method

    FreeStyle mines community LoRAs as style and content anchors for triplet construction and combines two-stage, leakage-targeted training with a benchmark for both generation settings.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    FreeStyle ranks first on VLM-Style and Ver-S on both benchmarks, while remaining competitive on other axes (SRef VLM-Style 7.142).

    Source: §7.2 · p. 11

  • Takeaways & Limitations

    The experiments support jointly optimizing data, training constraints, and evaluation for more robust and balanced dual-reference generation than improving any single component alone.

    Source: §8 · p. 14

    Stylistic semantics differ across base models, limiting cross-model transferability.

    Source: §8 · p. 12

Abstract

from arXiv · show

Style-content dual-reference generation aims to synthesize an image that preserves the structure and semantics of a content reference while adopting the style of a separate style reference.Despite recent progress, this setting remains challenging because models must balance content fidelity, style alignment, and instruction following avoiding semantic leakage from the style reference.A key bottleneck is the lack of large-scale triplet data with clean content-style separation and broad long-tail style coverage.In this work, we propose FreeStyle, a scalable dual-reference generation framework based on community LoRA mining.We treat community LoRAs as compositional anchors for style and content, and design a rigorous generation and filtering pipeline to construct large-scale Style-Reference and Content-Reference triplets across multiple base models.To address content leakage, we adopt a two-stage curriculum with stage-specific disentanglement mechanisms: an attention-level enrichment constraint that suppresses style-reference leakage in the style-transfer stage, and a frequency-aware RoPE modulation strategy that targets positional-correspondence-based leakage in the harder dual-reference stage.We also introduce a benchmark covering both style-reference and dual-reference generation, with evaluations on style similarity, content preservation, aesthetics, instruction following, and leakage rejection. The benchmark incorporates a style-invariant Content Alignment Score (CAS) and introduces a calibrated VLM-based Rejection Score for evaluating generation reliability and leakage suppression.Extensive experiments show that our model achieves a strong balance among style alignment, content preservation, and leakage suppression.

1. Introduction

FreeStyle addresses the challenge of generating images that preserve a content reference while adopting a separate style reference and following text instructions. It combines community-LoRA-based triplet construction, stage-specific leakage controls, and a benchmark for style-reference and dual-reference generation.

Introduction

  • Dual-reference generation must balance content fidelity, style alignment, and instruction following while preventing semantic content in the style reference from causing hallucinations or structural distortions.The setting combines a content reference, a separate style reference, and a text instruction.Source: §1 · p. 2
  • FreeStyle mines community LoRAs as compositional style and content anchors to construct large-scale SRef and CRef triplets with broad long-tail style coverage and clean separation.Its mining, generation, and filtering pipeline is designed to address the lack of scalable triplet data with broad style coverage.Source: §1 · p. 2
  • Style-reference leakage arises from disproportionate late-step attention to style tokens, while dual-reference leakage can arise from high-frequency RoPE positional correspondence that enables patch copying.The two-stage curriculum uses an attention-level enrichment constraint in Stage 1 and frequency-aware RoPE modulation in Stage 2 to suppress these distinct leakage pathways while retaining style richness and global stylistic structure.Source: §1 · p. 2
  • The benchmark evaluates both style-reference and dual-reference generation, using style-invariant CAS [81] for structural agreement and a VLM-based Verification Score for reliability and content preservation under cross-reference leakage.The protocol separates style-transfer reliability from content preservation to expose trade-offs that a single aggregate score could obscure.Source: §1 · p. 2

2. Related Work

Related work spans reference-conditioned image generation and stylization, while dual-reference generation remains constrained by limited triplet data and evaluation. FreeStyle addresses data limitations by mining community LoRAs as compositional anchors.

2.1. Reference-Based Generation and Stylization

  • Content-reference methods condition generation with spatial signals, decoupled cross-attention, or personalization through fine-tuning, low-rank adaptation, and token optimization.Examples include Control-Net, T2I-Adapter [50], IP-Adapter [82], and personalization methods [21] [30] [63] [68].Source: §2.1 · p. 2
  • Style-reference methods range from neural stylization to diffusion and GAN approaches, while leakage remedies address semantic content carried by style references.Approaches include attention- and transformer-based methods [45] [54], diffusion stylization through attention, inversion, or LoRA merging [27] [32], [10], [19] [66], and GANs [34] [35]. StyleAlign [27], InstantStyle [76], DEADiff [58], and CleanStyle [18] address leakage; most methods do not target large-scale supervision for explicit dual-reference generation.Source: §2.1 · pp. 2–3

2.2. Content and Style Dual-Reference Generation

  • Dual-reference generation must preserve content, adopt style, and follow instructions; existing editors can be unstable, and the field lacks diverse, clean triplets and systematic trade-off benchmarks.USO [79] advances task-specific control with compositional triplets from subject-driven generation and de-stylization, but the cited work identifies remaining data and evaluation gaps.Source: §2.2 · p. 3

2.3. Data Construction for Controllable Generation

  • Existing triplet pipelines trade off scale, diversity, and separation quality, whereas FreeStyle mines community LoRAs to build large-scale, diverse, cleanly separated triplets across multiple base models.Prior approaches include preference mining such as Premier, synthetic stylization such as MegaStyle [22] and OmniStyle [77], curated hybrid pipelines such as TeleStyle [85], and de-stylization such as USO [79].Source: §2.3 · p. 3

3. Method Overview

FreeStyle is a three-component framework for style- and content-dual-reference image generation, combining complementary data construction, a two-stage curriculum with stage-specific leakage controls, and a benchmark for both generation settings.

Overview

  • FreeStyle combines three tightly coupled components in a framework for style- and content-dual-reference image generation.Source: §3 · p. 3

Data

  • Community-LoRA mining treats LoRA weights as compositional style and content anchors to build large-scale dual-reference triplets with broad long-tail style coverage.A complementary style-transfer pipeline creates triplets through controlled stylization of diverse content images for clean supervision of basic style-reference generation.Source: §3 · p. 3

Training

  • Training progresses from style-reference generation on style-transfer data to dual-reference generation using LoRA-mined triplets mixed with style-transfer data.Stage 1 uses an attention-level enrichment constraint, while Stage 2 uses frequency-aware RoPE modulation to address distinct content-leakage mechanisms.Source: §3 · p. 3

Benchmark

  • The benchmark covers style-reference and dual-reference generation, using CSGO's style-invariant Content Alignment Score (CAS) [81] and a VLM-based Verification Score for reliability under cross-reference leakage.The Verification Score measures style-transfer and content-preservation reliability.Source: §3 · p. 3

4. Data Pipeline

The data pipeline creates complementary style-transfer and dual-reference triplets for two-stage training. It combines filtered style-transfer examples with community LoRA mining, validation, and large-scale generation.

4.1. Style-Transfer Data Construction

  • Stage 1 uses Nano Banana Pro to stylize web images with approximately 645 validated triggers, retaining outputs that pass DINOv2 [53] content and ONEIG style checks.Style references come from a different content source, and GPT [1] supplies style-transfer instructions.Source: §4.1 · pp. 3–4

4.2. Community LoRA Mining and Triplet Construction

  • The pipeline mines community LoRAs from Civitai, TensorArt, and Liblib, selects weights built upon Illustrious [55], FLUX-dev, and Qwen-Image [59], then constructs diverse prompts.Source: §4.2 · p. 4
  • Human reviewers retain stable LoRAs when at least 7 of 9 preview images are consistent, then use aesthetic-score-weighted sampling to reduce content-LoRA candidates.Stable content and style LoRAs are categorized from generated previews; higher-scoring content LoRAs are sampled more often.Source: §4.2 · p. 4
  • At least 20 images per LoRA are checked against web previews with Qwen3-VL [3], while Qwen3-VL [3] also generates prompts linking content references to targets.The reference-generation prompt pool contains approximately 40k combinations, and verification uses binary judgments with majority voting.Source: §4.2 · pp. 4–6
  • Bilateral LoRA-pair validation succeeds at approximately 0.4 versus 0.6 for style and 0.8 for content alone; pre-screened batch generation reaches approximately 0.8.Pairs undergo bilateral content and style verification against single-LoRA references before target triplet generation.Source: §4.2 · p. 5
  • 273k FLUX, 33k Qwen, and 172k Illustrious triplets were generated and filtered, yielding a massive, compositionally diverse dual-reference dataset.The final triplets pair content and style references with a generative prompt and target image.Source: §4.2 · pp. 5–6

5. Style-Content Disentanglement Training

FreeStyle uses a two-stage curriculum to address distinct leakage mechanisms: disproportionate style-reference attention in style-only training and positional correspondence in dual-reference training. It pairs attention constraints with frequency-aware RoPE modulation to control leakage in these settings.

5. Style-Content Disentanglement Training

  • Stage 1 trains on style-transfer data, while Stage 2 mixes in dual-reference triplets and style-transfer data to handle content, style, and text conditions together.The stages target different leakage channels, so the Stage 1 attention constraint loses discriminative power in Stage 2, where positional correspondence becomes the concern.Source: §5 · p. 6, §5.2 · p. 8

5.1. Attention Constraint for Style-Reference Generation

  • Style-reference leakage cases show broader, more persistent attention bands, with the strongest asymmetry late in denoising at the first transformer block.This time- and depth-localized pattern motivates constraining style-reference attention where it is most predictive of leakage.Source: §5.1 · p. 7, Figure 5 · p. 8
  • At the first transformer block, paired regularizers bound style-reference enrichment to 0–0.6 and normalized attention entropy to 0.06–0.14, with λe = λh = 0.1.The enrichment bound prevents over-attention, while the entropy band guards against attention collapsing onto a few style tokens or spreading uniformly; together, the terms gently shape attention while preserving style diversity.Source: §5.1 · p. 7

5.2. Frequency-Aware RoPE Modulation for Dual-Reference Generation

  • For dual-reference generation, frequency-aware RoPE modulation suppresses high-frequency components linked to local copying and amplifies low-frequency components that preserve global style structure.The fixed, timestep-independent modulation applies only to the style-reference branch, leaving the content-reference branch unchanged.Source: §5.2 · p. 8

6. Benchmark

FreeStyle’s benchmark standardizes evaluation of style-reference and dual-reference generation, separating style fidelity, content preservation, instruction following, leakage control, and aesthetics. It combines diverse reference pairings with feature-based and VLM judgments, a style-invariant Content Alignment Score, and verification scores.

6. Benchmark

  • The benchmark addresses fragmented evaluations by providing a fixed reference set, standardized prompts, and a mixed protocol that evaluates style fidelity, content preservation, and leakage control separately.The benchmark, prompts, and evaluation code are slated for public release.Source: §6 · p. 8

6.1. Benchmark Construction

  • The benchmark uses 200 diverse content images and 200 style images, with pairings sampled without replacement and GPT-generated [1] instruction variants across SRef and CRef+SRef settings.SRef transfers style while preserving source content; CRef+SRef uses separate content and style references and also requires following the text prompt.Source: §6.1 · pp. 8–9

6.2. Evaluation Metrics

  • Evaluation combines encoder similarities, VLM judgments, and aesthetic scores across style, content, and instruction-following axes, using complementary perceptual and semantic assessments.Style metrics include ONEIG [7], CSD [69], and VLM-S; content metrics include DINOv2 [53], CAS, and VLM-C; CRef+SRef instruction metrics include CLIP-T [60] and VLM-F. Aesthetic metrics are LAION-Aesthetic and V2.5-Aesthetic. VLM scores use Qwen3-VL [3].Source: §6.2 · p. 9
  • CAS measures content preservation with style-normalized DINOv2 features, reducing sensitivity to appearance differences; lower CAS indicates better structural agreement.Adopted from CSGO [81], it compares instance-normalized patch-token features after removing channel-wise style-carrying statistics, following the observation in [31] that such features retain more spatial structure.Source: §6.2 · p. 9
  • The Verification Score complements average similarity metrics with repeated binary VLM judgments, reporting separate style-transfer and content-preservation success proportions.Each image-reference pair is judged three times, with at least two positive judgments counting as a successful match; Ver-C specifically tests content preservation under potential semantic leakage from the style image.Source: §6.2 · pp. 9–10

7. Experiments

FreeStyle shows strong style-transfer performance across style-reference and dual-reference benchmarks, while ablations associate its leakage reduction and style gains with the attention constraint, RoPE modulation, and mined training data.

7.1. Implementation Details

  • Training uses Qwen-Image-Edit-2511 [59] with flow matching: Stage 1 runs 120k steps, then Stage 2 mixes mined triplets and style-transfer data at 3:1 for 24k steps.Both stages use 8× H100 GPUs; attention enrichment begins in Stage 1, and frequency-aware RoPE modulation begins in Stage 2.Source: §7.1 · p. 10

7.2. Benchmark Results

  • Feature-based scores trade off: CSGO leads CSD (0.665) but has VLM-Content (1.516) and Ver-C (0.715), whereas Qwen-Image-Edit leads DINOv2 (0.865) and CAS (0.918).The paper recommends reading feature-based metrics alongside corresponding VLM judgments and Verification Scores; Qwen-Image-Edit's VLM-Style score is 3.487.Source: §7.2 · p. 11
  • FreeStyle ranks first among open-source methods on SRef VLM-Style (7.142) and Ver-S (0.482), and on CRef+SRef VLM-Style (5.467) and Ver-S (0.409).It also ranks second on SRef Ver-C (0.928), LAION, and V2.5, and remains competitive across other axes in CRef+SRef. USO excels on content but scores poorly on style and instruction following; FLUX.2 [klein] leads on instruction metrics but has VLM-Style 2.388.Source: §7.2 · p. 11
  • In qualitative comparisons, FreeStyle captures target brushwork, texture, and palette while preserving content structure, and shows less semantic leakage than competing baselines in dual-reference generation.Other baselines introduce structural artifacts or copy semantic content from the style reference.Source: §7.2 · p. 11

7.3. Ablation Studies

  • Removing the Stage 1 attention-enrichment constraint raises VLM leakage score from 0.522 to 2.674; enabling it suppresses style-reference semantic contamination while preserving style transfer.Without the constraint, style-reference entities such as objects or architectural elements may appear in the target scene.Source: §7.3 · p. 11
  • Under CRef+SRef, RoPE modulation reduces VLM leakage score from 1.047 to 0.453 while suppressing positional-correspondence leakage and maintaining strong style transfer.Without modulation, content-specific features such as object silhouettes or texture patterns from the style image bleed into the output.Source: §7.3 · p. 11
  • Against the OmniStyle-trained variant, FreeStyle raises ONEIG from 0.277 to 0.468, VLM-Style from 3.038 to 7.142, and Ver-S from 0.186 to 0.482.Qualitative comparisons show more consistent reproduction of overall style while preserving source content, particularly for complex and long-tail styles.Source: §7.3 · p. 11

8. Discussion and Conclusion

FreeStyle combines community-LoRA data construction, stage-specific disentanglement, and systematic evaluation for style- and content-dual-reference generation. The discussion identifies cross-reference semantic contamination as the central challenge and notes remaining limits in curation, transfer, and evaluation.

8. Discussion and Conclusion

  • The central challenge is suppressing cross-reference semantic contamination: leakage stems from attention over-allocation in style-reference generation and positional correspondence in dual-reference generation, motivating stage-specific constraints.The paper argues that neither stronger style nor content fidelity in isolation addresses this challenge; data construction and training constraints are co-optimized for each failure mode.Source: §8 · pp. 11–12
  • Long-tailed, rapidly changing LoRA quality, cross-model domain shift, and coarse conflict metrics limit curation, transfer, and evaluation; future work targets quality assessment, style alignment, and granular leakage metrics.The discussion says existing metrics, including the proposed Verification Score, do not finely characterize style-content conflict intensity.Source: §8 · p. 12
  • Figure 7 shows FreeStyle preserving content layout while transferring style with less semantic leakage than baselines; jointly optimizing data, training constraints, and evaluation supports more robust, balanced dual-reference generation.The framework comprises a community-LoRA-based data pipeline, a two-stage training strategy with stage-specific disentanglement mechanisms, and a systematic benchmark.Source: Figure 7 · p. 13, §8 · p. 14

9. Data Usage and Ethics Disclaimer

The work uses publicly sourced materials solely for non-commercial academic research and disclaims ownership of third-party content. Its released benchmark, prompts, and code are provided without warranty, and users bear responsibility for lawful use.

Data Usage and Ethics Disclaimer

  • Publicly accessible images, community LoRAs, and metadata were used only to study and evaluate reference-based image generation, not for commercial products or services.The authors state that the work is conducted strictly for non-commercial academic research.Source: §9 · pp. 14, 16
  • Third-party content remains the property of its respective rights holders; the authors report reasonable efforts to respect applicable source terms and licenses.They describe using materials only as necessary for scientific analysis, academic evaluation, and reproducibility.Source: §9 · p. 16
  • The released benchmark, prompts, and code are provided without warranties, and the authors and affiliated institutions disclaim liability for damages arising from their use.Users are solely responsible for ensuring their use complies with applicable laws, regulations, platform terms, and third-party rights.Source: §9 · p. 16
  • The authors say they will promptly remove specific content upon a reasonable request from a rights holder, and users agree to the disclaimer by using associated materials.Source: §9 · p. 16

A. Appendix · A.1. Model Architecture without Extra Image Encoders

FreeStyle integrates style references without an external image encoder, applying constraints directly to the model’s first attention block. A clustering experiment supports using VAE latent features for style guidance because they distinguish style clusters more clearly than the tested image encoders.

A.1. Model Architecture without Extra Image Encoders

  • The model integrates style references without relying on an additional external image encoder.Its architecture is summarized in the middle panel of Figure 5.Source: §A.1 · p. 21
  • The design avoids external encoders because their limited ability to cluster and classify artistic styles could impair model performance.Source: §A.1 · p. 21
  • During training, FreeStyle disentangles attention layers and applies its constraints directly to the first block.Source: §A.1 · p. 21
  • For the clustering experiment, four style LoRAs were paired with the same 150 base prompts to generate test images.The generated images were embedded with CLIP, DINOv2, other common image encoders, and the VAE, then visualized using t-SNE.Source: §A.1 · p. 21
  • VAE features distinguished the style clusters relatively clearly compared with features from the other tested image encoders.The experiment treats style LoRAs as clustering centers for stylistic concepts.Source: §A.1 · p. 21
  • The clustering results support using VAE-extracted latent features for explicit style guidance.Source: §A.1 · p. 21

A.2. Why Constrain the First Block … CONTENT STYLE TARGET CONTENT STYLE TARGET

The supplement explains why constraining the first transformer block can suppress semantic leakage and provides additional evidence on dataset composition, evaluation prompts, and generation quality across diverse styles. It also showcases the breadth of the triplet data and the method’s performance on long-tail style and dual-reference examples.

A.2. Why Constrain the First Block

  • Early blocks, especially block 0, establish global layout and semantics, so leakage there can corrupt the image in ways later blocks can hardly correct.This motivates applying the attention constraint at the first block, where it can suppress cross-reference semantic leakage most effectively.Source: §A.2 · p. 21, Figure 11 · p. 21

A.4. CRef and SRef Dataset Composition

  • The SRef and CRef dataset composition spans Qwen, FLUX, and Illustrious, with triplet totals derived from combinations of LoRAs and their average generated-image counts.For Qwen, the reported composition uses 608 valid dual-reference LoRA combinations averaging 8.85 images, 53 style LoRAs averaging 8.00 images, and 19 content LoRAs averaging 13.12 images; VAE latent features also show clearer style-cluster separation than CLIP and DINOv2, motivating direct VAE-latent guidance.Source: Table 5 · p. 21, §A.4 · p. 21, Figure 12 · p. 22

A.9. Additional Triplet Data Showcases; CONTENT STYLE TARGET CONTENT STYLE TARGET

  • Additional triplet showcases illustrate the data pipeline’s high-quality outputs, stylistic diversity, and accurate content preservation.The supplementary figures extend the triplet examples across four additional showcase parts.Source: §A.9 · p. 22, Figure 23 · p. 32, Figure 24 · p. 33

A.3. Trigger-Word Statistics

  • The collected style and content LoRA trigger words exhibit vocabulary diversity that supports prompt-pool construction.The supplement presents phrase-level word clouds for both style and content triggers.Source: §A.3 · p. 21, Figure 13 · p. 22

A.5. Additional Visual Comparisons for SRef

  • Additional SRef comparisons show consistent style transfer and low content leakage across challenging long-tail styles and complex object structures.Examples span oil painting, watercolor, 3D rendering, and abstract art.Source: §A.5 · p. 22, Figure 14 · p. 23

A.6. Extended CRef+SRef Comparisons (Group I)

  • The extended dual-reference examples demonstrate generalization across diverse artistic domains while maintaining content fidelity.Shown styles include 3D modeling, origami, pointillism, geometric abstraction, and graffiti.Source: §A.6 · p. 22, Figure 15 · p. 24

A.8. System prompt in VLM-based metrics

  • The VLM evaluation prompts separately assess style-independent content consistency, visual-style similarity, content and style verification, instruction following, and content leakage.Content judgments are instructed to ignore style differences, while leakage judgments distinguish shared style from reuse of specific content.Source: §A.9 · pp. 25, 26, 27, 29, 30

A.7. Extended CRef+SRef Comparisons (Group II)

  • Across long-tail styles, the dual-reference results show robust generalization, with the method outperforming competing baselines on challenging artistic styles.Examples include paper-cutting, japonism, fauvism, printmaking, flat vector, smooth clay, multilayer paper, leaf art, constructive illustration, and handmade clay.Source: §A.7 · p. 22, Figure 22 · p. 31

Related papers

Loading 2606.20506v1…