PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeStyle: Free Control of Style-Content Dual-Reference Generation from Community LoRA Mining

Jinghong Lan, Wei Cheng, Yunuo Chen, Ziqi Ye, Peng Xing, Yixiao Fang, Rui Wang, Yufeng Yang, Xuanyang Zhang, Xianfang Zeng, Difan Zou, Gang Yu, Chi Zhang

arXiv:2606.20506v1cs.CVcs.AI

TL;DR

Dual-reference generation must preserve content structure while adopting a separate style without semantic leakage, but large-scale, cleanly separated triplets remain scarce. FreeStyle mines community LoRAs for data and applies stage-specific disentanglement within a two-stage framework, achieving a strong balance across style, content, instruction following, and leakage suppression.

  • Problem

    Dual-reference generation must jointly preserve content structure, match a separate style, and follow instructions, despite limited large-scale triplets with clean separation and broad style coverage.

  • Method

    FreeStyle mines community LoRAs as compositional style-content anchors, combining scalable triplet construction, two-stage disentanglement, and a benchmark with leakage-aware evaluation.

  • Results

    FreeStyle ranks first on VLM-Style and Ver-S on both SRef and CRef+SRef benchmarks while remaining competitive across content, aesthetics, and instruction-following axes.

  • Takeaways & Limitations

    The results support jointly optimizing data construction, training constraints, and evaluation for more robust and balanced dual-reference generation.

  • Takeaways & Limitations

    Community LoRA quality evolves rapidly, cross-model style domain shift limits transferability, and existing metrics remain insufficiently fine-grained for style-content conflicts.

Abstract

from arXiv · show

Style-content dual-reference generation aims to synthesize an image that preserves the structure and semantics of a content reference while adopting the style of a separate style reference.Despite recent progress, this setting remains challenging because models must balance content fidelity, style alignment, and instruction following avoiding semantic leakage from the style reference.A key bottleneck is the lack of large-scale triplet data with clean content-style separation and broad long-tail style coverage.In this work, we propose FreeStyle, a scalable dual-reference generation framework based on community LoRA mining.We treat community LoRAs as compositional anchors for style and content, and design a rigorous generation and filtering pipeline to construct large-scale Style-Reference and Content-Reference triplets across multiple base models.To address content leakage, we adopt a two-stage curriculum with stage-specific disentanglement mechanisms: an attention-level enrichment constraint that suppresses style-reference leakage in the style-transfer stage, and a frequency-aware RoPE modulation strategy that targets positional-correspondence-based leakage in the harder dual-reference stage.We also introduce a benchmark covering both style-reference and dual-reference generation, with evaluations on style similarity, content preservation, aesthetics, instruction following, and leakage rejection. The benchmark incorporates a style-invariant Content Alignment Score (CAS) and introduces a calibrated VLM-based Rejection Score for evaluating generation reliability and leakage suppression.Extensive experiments show that our model achieves a strong balance among style alignment, content preservation, and leakage suppression.

1 Fudan University 2 StepFun 3 Westlake University 4 University of Hong Kong

FreeStyle mines community-created style and content LoRAs, composes them through standardized workflows, and produces diverse image triplets supporting controllable style transfer and style-subject generation.

  • Community-created style and content LoRAs are collected from multiple platforms and automatically composed through standardized workflows.
  • The resulting FreeStyle dataset contains diverse style–content image triplets spanning multiple base models, artistic styles, and subject categories.
  • FreeStyle enables style transfer and style-subject controllable image generation across a broad range of visual domains.

1. Introduction

FreeStyle addresses dual-reference generation by mining community LoRAs to build scalable, broadly styled triplet data and by introducing stage-specific mechanisms for suppressing content leakage. It also provides a benchmark with style-invariant content evaluation and VLM-based reliability assessment.

  • Motivation: Dual-reference generation must preserve content structure while adopting a separate style reference without semantic leakage, making it more challenging than single-reference guidance.Style references may contain objects and layouts whose semantics leak into outputs, causing hallucinations or structural distortions.
  • Data construction: FreeStyle mines community LoRAs as compositional style and content anchors to construct large-scale SRef and CRef triplets across multiple base models.The framework targets broad long-tail style coverage while addressing limited, costly, or imperfectly separated existing triplet data.
  • Disentanglement: The two-stage curriculum uses attention-level enrichment in Stage 1 and frequency-aware RoPE modulation in Stage 2 to target complementary leakage pathways.Stage 2 suppresses high-frequency positional components that encourage copying while amplifying low-frequency components that preserve global stylistic structure.
  • Evaluation: The benchmark evaluates style-reference and dual-reference generation using standard feature metrics, style-invariant Content Alignment Score (CAS), and a VLM-based Verification Score.CAS measures structural agreement after factoring out style, while the Verification Score separately evaluates style-transfer reliability and content preservation under potential cross-reference leakage.

2. Related Work

Prior work supports content and style control through reference-based conditioning, stylization, and multi-image editing, but dual-reference generation additionally requires content-style disentanglement and instruction following. FreeStyle addresses data limitations by mining community LoRAs to construct large-scale, diverse triplets with clean separation across multiple base models.

  • Reference-Based Generation: Reference-based generation controls content through spatial signals, decoupled cross-attention, and personalization via fine-tuning, low-rank adaptation, or token optimization.Examples include Control-Net, T2I-Adapter, IP-Adapter, and personalization methods.
  • Style-Reference Generation: Style-reference generation extracts visual attributes through neural style transfer, attention and transformer methods, feature matching, contrastive objectives, wavelets, and diffusion-based stylization.Diffusion approaches include shared or swapped attention, inversion-based methods, and LoRA merging; GANs provide domain-specific style control.
  • Dual-Reference Generation: Dual-reference generation must preserve content, adopt style, and follow text instructions, requiring multi-image fusion and content-style disentanglement beyond single-reference or text-guided editing.General-purpose editors support multi-image inputs but are not optimized for this setting and can be unstable.
  • Triplet Data: Existing triplet-data pipelines remain limited in scale, diversity, or separation quality, while preference-based and synthetic stylization datasets do not fully address content-style disentanglement.Premier mines user interactions but is not designed for disentanglement; MegaStyle and OmniStyle scale more easily but inherit limitations in style range and related properties.
  • Triplet Data: Community LoRAs serve as scalable compositional anchors for style and content, enabling large-scale triplets with broad style diversity and clean separation across multiple base models.This mining strategy is presented in contrast to existing preference-based and synthetic stylization pipelines.

3. Method Overview

FreeStyle is a three-component framework for style- and content-dual-reference image generation, combining complementary data construction, a two-stage disentanglement curriculum, and an evaluation benchmark.

  • Framework components: FreeStyle comprises three tightly coupled components: data construction, two-stage training, and benchmarking for style- and content-dual-reference generation.The framework combines complementary datasets, stage-specific leakage mitigation, and evaluation of style, content, and reliability.
  • Data: The data pipeline combines controlled stylization for clean style-reference supervision with community-LoRA mining for large-scale dual-reference triplets and broad long-tail style coverage.LoRA weights serve as compositional anchors for both style and content.
  • Training: Training uses a two-stage curriculum from style-reference generation to dual-reference generation, pairing attention-level enrichment with Stage 1 and frequency-aware RoPE modulation with Stage 2.Stage 2 mixes LoRA-mined triplets with style-transfer data and addresses a harder content-leakage mechanism.
  • Benchmark: The benchmark evaluates style-reference and dual-reference generation using the style-invariant Content Alignment Score and a VLM-based Verification Score for reliability under cross-reference leakage.The Verification Score measures style-transfer and content-preservation reliability.

4. Data Pipeline

The data pipeline builds complementary triplets for two-stage training: style-transfer data with separated content and style, and large-scale dual-reference data mined from community LoRAs. It combines multi-source collection, model-specific generation workflows, bilateral verification, and filtering of stable LoRAs and compatible pairs.

  • Stage 1: Style-transfer data: Stage 1 constructs style-transfer triplets using Nano Banana Pro and bilateral consistency filtering to separate content structure from visual style.Content fidelity is checked against the source with DINOv2 feature similarity, while style consistency is measured against the style anchor.
  • Stage 1: Style-transfer data: The style-transfer pipeline crawls diverse web images, validates approximately 645 stable style triggers, and applies a fixed “transfer into [trigger] style” template.The content collection spans landscapes, human subjects, and everyday objects.
  • Stage 2: Community-LoRA mining: Stage 2 mines community LoRAs from Civitai, TensorArt, and Liblib, selecting weights based on Illustrious, FLUX-dev, and Qwen-Image backbones.Separate ComfyUI workflows and compatible vocabularies are designed for each base model.
  • Stage 2: Community-LoRA mining: The pipeline filters single-LoRA models with 3×3 preview grids, generates at least 20 reference images per LoRA, and verifies outputs with binary Qwen3-VL judgments and majority voting.The filtering targets stable, representative reference images for both style and content LoRAs.
  • Stage 2: Community-LoRA mining: Compatible style–content LoRA pairs are selected through Cartesian-product testing and bilateral verification against single-LoRA references before batch generation of target triplets.The resulting dataset contains content and style references, a generative prompt, and a target image, with exceptionally massive and diverse compositional combinations.

5. Style-Content Disentanglement Training

FreeStyle uses a two-stage curriculum to disentangle style and content references. It controls Stage 1 leakage with attention regularization and addresses Stage 2 positional-copying leakage through frequency-aware RoPE modulation.

  • Two-stage curriculum: The two-stage curriculum first trains style-transfer generation, then mixes dual-reference triplets with style-transfer data for simultaneous content, style, and instruction conditioning.Stage 1 targets style-reference capability; Stage 2 uses the content reference as a structural scaffold while transferring style from the style reference.
  • Stage 1: attention disentanglement: Leakage cases show broader, more persistent style-reference attention bands and disproportionate late-denoising attention, motivating a time- and depth-sensitive enrichment constraint.The analysis treats leakage as a time-varying failure pattern rather than a static model property.
  • Stage 1: attention disentanglement: The enrichment score normalizes each semantic group’s attention by its size fraction, distinguishing proportional attention from enrichment or suppression across denoising steps and transformer depth.E_g = 1 denotes size-proportional attention, E_g > 1 enrichment, and E_g < 1 suppression.
  • Stage 2: positional disentanglement: In Stage 2, enrichment loses discriminative power because attention no longer shows the Stage 1 late spike, so frequency-aware RoPE modulation targets positional patch-copying leakage.The modulation suppresses high-frequency locality and amplifies low-frequency global guidance only on the style-reference branch, leaving the content-reference branch unchanged.

6. Benchmark

The benchmark standardizes style-reference and dual-reference generation evaluation with diverse references, fixed prompts, and separate axes for style fidelity, content preservation, instruction following, leakage control, and aesthetics. It combines encoder metrics, VLM judgments, CAS, and verification scores to assess both alignment and reliability.

  • Benchmark design: The open benchmark uses fixed references, standardized prompts, and separate evaluation axes to enable more reliable cross-method comparison.It covers style-reference generation and content-style dual-reference generation.
  • Benchmark design: The benchmark contains 200 content and 200 style reference images spanning diverse objects, scenes, compositions, and artistic domains.Pairings are sampled without replacement to prevent any single reference from dominating evaluation.
  • Evaluation protocol: Each fidelity and instruction-following axis combines encoder-based metrics with VLM-based judgments, while aesthetic scores assess raw visual appeal.Style uses ONEIG, CSD, and VLM-S; content uses DINOv2, CAS, and VLM-C; instruction following uses CLIP-T and VLM-F.
  • Content Alignment Score: CAS measures style-invariant structural agreement between generated and content-reference features, with lower CAS indicating better content preservation.It uses instance-normalized DINOv2 patch-token features to reduce sensitivity to style-carrying channel statistics.
  • Verification Score: The VLM-based Verification Score adds a binary reliability signal for successful style transfer and content preservation.Ver-S measures correctly transferred style, while Ver-C measures preserved content under potential semantic leakage from the style image.

7. Experiments

FreeStyle is evaluated on style-reference and dual-reference benchmarks, where it achieves the strongest reported balance of style transfer, content preservation, aesthetics, and leakage suppression. Ablations further validate its attention constraint, frequency-aware RoPE modulation, and community-LoRA-mined training data.

  • Main results: FreeStyle ranks first on SRef VLM-Style (7.142) and Ver-S (0.482), while placing second on Ver-C (0.928) and LAION aesthetics.These results indicate strong style transfer with competitive content preservation and aesthetics among open-source methods.
  • Main results: On CRef+SRef, FreeStyle again ranks first on VLM-Style (5.467) and Ver-S (0.409), with competitive performance across other axes.The same pattern holds on the harder dual-reference benchmark.
  • Qualitative comparison: Qualitatively, FreeStyle captures target brushwork, texture, and palette while preserving content structure and substantially reducing semantic leakage versus competing baselines.Baselines instead introduce structural artifacts or copy semantic content from the style reference.
  • Ablations: The attention-map constraint lowers VLM leakage from 2.674 to 0.522 by suppressing style-reference entities while preserving accurate style transfer.Removing the Stage 1 enrichment loss frequently causes objects or architectural elements from the style image to appear in the target scene.
  • Ablations: Frequency-aware RoPE modulation decreases VLM leakage from 1.047 to 0.453 by suppressing positional-correspondence leakage under CRef+SRef.Without modulation, object silhouettes or texture patterns from the style image bleed into the output.
  • Ablations: Community-LoRA-mined training data improves ONEIG from 0.277 to 0.468, VLM-Style from 3.038 to 7.142, and Ver-S from 0.186 to 0.482 over OmniStyle data.All other settings are kept identical, and qualitative comparisons show a more pronounced perceptual difference.

8. Discussion and Conclusion

FreeStyle identifies cross-reference semantic contamination as the central challenge in dual-reference generation and addresses it through stage-specific disentanglement. Its jointly optimized data pipeline, training constraints, and benchmark produce more robust and balanced generation, while leaving challenges in curation, cross-model transfer, and evaluation.

  • Core challenge: The central challenge is suppressing cross-reference semantic contamination under multi-condition control, rather than improving style strength or content fidelity in isolation.Attention over-allocation drives leakage in style-reference generation, while positional correspondence drives leakage in dual-reference generation.
  • Limitations: Community LoRA quality is long-tailed and rapidly evolving, creating an ongoing challenge for automated curation.The paper also identifies domain shift across base models and insufficient metrics for fine-grained characterization of style-content conflict intensity.
  • Framework: FreeStyle combines community-LoRA data construction, two-stage training with stage-specific disentanglement mechanisms, and a systematic benchmark.The framework co-optimizes data construction and training constraints for the two generation settings.
  • Findings: Jointly optimizing data, training constraints, and evaluation yields more robust and balanced dual-reference generation than improving any single component alone.This conclusion is supported by the paper’s extensive experiments.

9. Data Usage and Ethics Disclaimer

The work is limited to non-commercial academic research and uses publicly accessible third-party materials solely for reference-based image-generation study and evaluation. The disclaimer preserves rights holders’ ownership while placing legal and usage responsibilities on users.

  • Research Purpose and Data Sources: The study is conducted strictly for non-commercial academic research, using publicly accessible reference images, community LoRA weights, and metadata solely for reference-based image-generation research.No part of the work is intended for or deployed in a commercial product or service.
  • Third-Party Rights: The authors do not claim ownership of third-party content, whose copyrights, trademarks, and intellectual-property rights remain with their respective owners.The materials are used only as necessary for scientific analysis, fair-use academic evaluation, and reproducibility, with reasonable efforts to respect applicable terms and licenses.
  • Disclaimer and Liability: The released benchmark, prompts, and code are provided “as is,” without warranties, and the authors and affiliated institutions accept no liability for resulting damages.This includes damages arising from use, misuse, or inability to use the data, models, or methods.
  • User Responsibilities and Takedowns: Users are solely responsible for ensuring compliance with applicable laws, regulations, platform terms of service, and third-party rights in their jurisdictions.Rights holders may request removal of specific content, and the authors state they will promptly remove corresponding material upon a reasonable request.

A. Appendix … CONTENT STYLE TARGET CONTENT STYLE TARGET

The appendix details FreeStyle’s encoder-free architecture, first-block attention constraint, dataset construction, and evaluation prompts. Additional qualitative results show broad style generalization, content fidelity, low leakage, and diverse triplet data across challenging long-tail styles.

  • A.1. Model Architecture without Extra Image Encoders: FreeStyle avoids external image encoders, directly applying attention constraints because existing encoders may poorly cluster artistic styles and impair performance.The design disentangles attention layers and applies constraints directly to the first block.
  • A.2. Why Constrain the First Block: Early denoising blocks, especially block 0, encode global layout and semantics, motivating first-block constraints before later layers refine local texture and detail.Figure 11 states that leakage at block 0 cannot be corrected by subsequent layers.
  • A.3. Trigger-Word Statistics: Style and content LoRA trigger-word statistics evidence vocabulary diversity, while VAE latent features separate four style clusters more clearly than CLIP and DINOv2.The clustering experiment used 150 shared prompts for images generated from four style LoRAs.
  • A.4. CRef and SRef Dataset Composition: The curated SRef and CRef datasets combine retained dual-reference LoRA combinations with generated images across Qwen, FLUX, and Illustrious base models.Table 5 reports composition statistics, with total triplets calculated from the Cartesian product and released samples obtained by sampling style references.
  • A.5. Additional Visual Comparisons for SRef: Additional SRef examples maintain consistent style transfer and low content leakage across oil painting, watercolor, 3D rendering, abstract art, and other long-tail styles.The examples also cover more complex object structures.
  • A.6. Extended CRef+SRef Comparisons (Group I): Extended dual-reference examples demonstrate style transfer with content fidelity across 3D modeling, origami, pointillism, geometric abstraction, graffiti, and quick sketch.The comparisons are presented as Group I results across diverse artistic domains.
  • A.7. Extended CRef+SRef Comparisons (Group II): Further dual-reference comparisons show robust generalization to challenging long-tail styles including paper-cutting, japonism, fauvism, printmaking, clay, leaf art, and constructive illustration.The passage states that the method consistently outperforms competing baselines on these styles.
  • A.8. System prompt in VLM-based metrics; A.9. Additional Triplet Data Showcases; CONTENT STYLE TARGET CONTENT STYLE TARGET: The appendix specifies VLM-based prompts for content similarity, style similarity, content and style verification, instruction following, and content leakage, while showcasing abundant, diverse, high-quality LoRA-generated triplets.The leakage prompt distinguishes stylistic similarity from reused visual content and assigns low scores when only broad categories or style are shared.
Loading 2606.20506v1…