PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture of Style Experts for Diverse Image Stylization

Shihao Zhu, Ziheng Ouyang, Yijia Kang, Qilong Wang, Mi Zhou, Bo Li, Ming-Ming Cheng, Qibin Hou

arXiv:2603.16649v3cs.CV

TL;DR

Existing stylization methods often emphasize color while missing semantic and material details, and their style-injection mechanisms do not adequately distinguish diverse styles. StyleExpert learns style representations and uses them to route styles through specialized MoE LoRA experts. The authors report strong content preservation and style fidelity, along with generalization to unseen styles.

  • Problem

    Existing stylization methods often reduce semantic style transfer to color mapping, while style datasets underrepresent semantic and material styles and style injection does not adequately capture diverse semantic characteristics.

  • Method

    StyleExpert trains a style encoder with InfoNCE and uses its style latents to condition an MoE router that selects specialized LoRA experts within DiT layers.

  • Results

    StyleExpert achieves high style fidelity and content preservation, outperforms recent state-of-the-art approaches, and generalizes well to unseen styles.

  • Takeaways & Limitations

    The framework supports diverse styles across multiple semantic levels using a balanced dataset and style-specific expert selection.

Abstract

from arXiv · show

Diffusion-based stylization has advanced significantly, yet existing methods are limited to color-driven transformations, neglecting complex semantics and material details. We introduce StyleExpert, a semantic-aware framework based on the Mixture of Experts (MoE). Our framework employs a unified style encoder, trained on our large-scale dataset of content-style-stylized triplets, to embed diverse styles into a consistent latent space. This embedding is then used to condition a similarity-aware gating mechanism, which dynamically routes styles to specialized experts within the MoE architecture. Leveraging this MoE architecture, our method adeptly handles diverse styles spanning multiple semantic levels, from shallow textures to deep semantics. Extensive experiments show that StyleExpert outperforms existing approaches in preserving semantics and material details, while generalizing to unseen styles. Our code and collected images are available at the project page: https://hh-lg.github.io/StyleExpert-Page/.

1. Introduction

Existing diffusion-based stylization methods often reduce semantic style transfer to color mapping, while datasets and style-injection mechanisms underrepresent complex semantic and material attributes. StyleExpert addresses these gaps with a semantic-diverse dataset and an MoE-based style-conditioning approach.

  • Style transfer includes color transfer and semantic transfer, but prominent methods often fail to capture texture, lines, and material beyond dominant colors.
  • Existing datasets are imbalanced toward color-based styles, while training-free generation often produces irrelevant textures, noise, artifacts, and poor-quality images.
  • VAE latent codes provide limited semantic information, and cross-attention or prompt-based methods treat diverse styles uniformly during injection.
  • The proposed dataset pipeline uses style-centric LoRAs, OmniConsistency LoRA, CLIP-based reference selection, and Qwen filtering to create content-style-stylized triplets.
  • StyleExpert integrates an InfoNCE-trained style encoder with an MoE router, while constructing 500k content-style-stylized triplets for semantic stylization research.

2. Related Work

Diffusion-based style transfer has expanded beyond optimization methods, but existing approaches remain constrained by unstable training-free inference, prompt requirements, and difficulty handling diverse semantic styles. StyleExpert uses style-conditioned MoE routing to specialize style processing.

  • Diffusion-based style transfer methods increasingly build on models such as Stable Diffusion and FLUX after earlier optimization-based approaches.
  • StyleExpert's two-stage design trains a style encoder with InfoNCE before using its prior knowledge to inform the MoE router and accelerate convergence.
  • Training-free methods may suffer unstable performance, inference-time computational overhead, or inability to use a single style image.
  • Mixture of Experts: MoE models expand capacity through expert networks and a router that activates a subset of experts, but their use in image generation remains limited.
  • Mixture of Experts: Unlike prior MoE-finetuned models that feed hidden states to the router, StyleExpert uses pre-trained style-encoder features from a style image to control expert selection.

3. Method

StyleExpert first learns style representations with contrastive training, then uses style latents to route inputs through specialized LoRA experts embedded in DiT layers. The method combines shared and style-specific adaptations to handle diverse styles.

  • Training stages: The method trains a style representation encoder with InfoNCE, then uses its prior knowledge to guide the MoE router during a second training stage.
  • DiT foundation: Multimodal attention combines text and image embeddings, and the Flux-Kontext base model additionally supports image-control inputs.
  • Style representation encoder: The encoder maps images with shared style labels toward similar representations using temperature-scaled cosine similarity and SigLIP features processed by an MLP.
  • Style representation encoder: InfoNCE compares representations across independently sampled batches and uses a positive mask to weight pairs sharing the same style label.
  • MoE fine-tuning: Because one LoRA cannot handle styles at varying granularities, StyleExpert embeds multiple LoRA experts in self-attention and FFN layers and routes styles using style latents.
  • MoE fine-tuning: Each layer combines the original transformation with a shared expert and a weighted sum of selected specialized experts, increasing capacity for diverse styles.

4. Stylized Dataset Curation

The authors build a semantically diverse stylization dataset by curating style LoRAs and content images, generating stylized outputs, filtering failures, and selecting coherent style references for triplets.

  • 8,000+ styles in OmniStyle-150K are predominantly color-centric, motivating a dataset that better represents semantic and material styles.
  • Approximately 650 community style LoRAs were manually curated and deduplicated into 209 high-quality LoRAs.
  • Approximately 2,700 photographs spanning people, landscapes, architecture, animals, and complex multi-person scenes provide diverse content.
  • Qwen rewrites captions to describe objective content without color, atmosphere, or lighting information that could interfere with style application.
  • The pipeline generates approximately 500,000 stylized images, filters failures and inconsistencies with Qwen-VL, and retains around 40,000 high-fidelity images.The generated collection is named StyleExpert-500K, while the filtered dataset is named StyleExpert-40K.
  • CLIP-based similarity selects a suitable generated style reference for each stylized image, producing coherent content-style-stylized triplets.

5. Experiments

Experiments evaluate StyleExpert against recent stylization methods and internal baselines on unseen styles, showing strong style and content performance, efficient computation, and benefits from the style encoder.

  • Experiment Settings: The benchmark trains on 188 styles and tests on 21 unseen styles, using 50 content-style pairs per test style.
  • Experiment Settings: Evaluation measures content fidelity, style similarity, aesthetic quality, and semantic attributes using DINO, CSD, DreamSim, LAION aesthetic, and Qwen Semantic Score.
  • Qualitative Comparisons: On unseen styles, StyleExpert more faithfully transfers lines, atmosphere, and materials, whereas several baselines primarily perform color transfer.
  • Quantitative Comparisons: StyleExpert achieves state-of-the-art results on CLIP Score, CSD Score, Aesthetic Score, Qwen Semantic Score, and DreamSim.Its Qwen Semantic Score is 75.12, significantly surpassing competing methods.
  • Efficiency: Compared with standard LoRA at equal total LoRA rank, StyleExpert incurs less base-model computational overhead but uses more trainable parameters.
  • Ablation Study: Removing the style encoder causes unstable, slower MoE convergence and degrades performance, particularly on CSD and DreamSim.

6. Discussions and Conclusions

StyleExpert combines MoE fine-tuning with a pre-trained style encoder to render diverse styles and generalize to unseen styles. Experiments report strong style fidelity, content preservation, and performance relative to recent state-of-the-art approaches.

  • The style encoder maps visually similar styles near each other while separating illustration-style images, supporting generalization to unseen styles.Contra and classic pixel art are mapped closer together in the latent space than illustration-style images.
  • Similar styles produce three times higher expert-selection overlap than dissimilar styles, confirming style-specific routing.Reference images share fixed content while the router selects experts from style-encoder latent codes.
  • StyleExpert uses MoE fine-tuning with a pre-trained style encoder to render diverse styles across multiple semantic levels.The framework is supported by a dataset with balanced color-centric and semantic-centric styles.
  • Experiments show high style fidelity and content preservation, outperforming recent state-of-the-art approaches.

Supplementary Materials

The supplementary materials define evaluation procedures for semantic use of style references and explain how related metrics are computed. They also document implementation details for prompt-based evaluation and CLIP scoring.

  • The Qwen Semantic Score tests whether stylized outputs incorporate semantic information from the style reference image.A VLM receives the style reference and stylized output and determines whether style semantics were incorporated.
  • Qwen3-VL-8B-Instruct assigns a score of 1 when its response begins with “YES” and 0 otherwise.
  • Dataset curation sums scores from five sampled triplets per style, whereas quantitative comparisons average scores across the test set.The resulting style-level score ranges from 0 to 5, while the model-level metric ranges from 0 to 1.
  • The reported CLIP score averages similarities to both content and style references, discouraging models from merely copying the content image.

B.1. IoU Experiment Details

The IoU experiment compares expert selections for similar and dissimilar styles while holding content fixed. It averages overlap across sampled triplets and groups MoE layers into early, middle, and late stages.

  • The experiment forms similar and dissimilar style pairs by comparing CLIP similarity among test images sharing the same content.An anchor image is compared with all other test images that share its content.
  • For each MoE-enabled layer, IoU measures overlap between expert indices selected for the anchor and each comparison style.
  • The experiment averages IoU over 100 sampled triplets to obtain layer-wise comparisons.
  • Early, Mid, and Late Stage denote the first, middle, and final thirds of the MoE-injected layers.
  • Some compared stylization methods require content or task prompts, so evaluation supplies prompts tailored to each method category.

C. Dataset Curation Details

Dataset curation removes style descriptors from content captions because they can conflict with styles’ intrinsic palettes or compositions. Additional filtering checks whether generated triplets preserve layout and key attributes.

  • Initial VLM-generated captions can retain colors, atmosphere, or other style descriptors despite requests for content-only descriptions.
  • The rewriting step removes conflicting descriptors so prompts remain compatible with styles defined by intrinsic color palettes or atmospheres.Examples include blue dot, Digital Yellow, Van Gogh, Elegant Line, Pop Art, 3D Texture, Cinematic, and Contra.
  • Clean prompts avoid specifying backgrounds or colors that could conflict with an artistic style and degrade stylization quality.
  • Triplets are filtered for significant layout changes by checking composition and attributes such as the number or gender of persons.

D. Dataset Comparison

StyleExpert-500K provides more semantically diverse and balanced styles than OmniStyle-150K, while StyleExpert achieves stronger human preference and broader qualitative performance across competing methods.

  • StyleExpert-500K spans Color, Line, Texture, and Semantic levels, whereas OmniStyle-150K largely degenerates to simple color transfer.The comparison notes missed illustration and symbolic styles in OmniStyle-150K, along with poor aesthetic quality when reference colors do not suit the content.
  • StyleExpert-500K maintains color-semantic balance and achieves higher CSD, CLIP similarity, and Aesthetic scores with lower DreamSim distance than OmniStyle-150K.It performs comparably on DINO while showing better stylistic diversity and overall visual quality on the other reported metrics.
  • 74.5% Top-1 preference was achieved by StyleExpert in a 30-participant user study with 1,200 total votes.Participants evaluated style consistency, content preservation, and overall aesthetics against randomized baseline outputs.
  • Expanded comparisons report that StyleExpert consistently achieves the best overall stylization performance against a broader range of methods.The comparisons include Qwen-Image-Edit, Nano-Banana, ChatGPT, OmniStyle, CSGO, USO, OmniGen2, and DreamO.

G. Additional Visual Results

Additional visualizations evaluate StyleExpert across diverse content and style inputs, including unseen styles, while the accompanying prompts define content and style consistency checks used in dataset preparation.

  • Additional results cover diverse content subjects and style inputs across Figures 13–16.The figures present further generations from StyleExpert rather than a quantitative comparison.
  • The visualizations show transfer of color, atmosphere, intricate textures, and lines across multiple semantic levels.Examples include traditional color transfer, overall atmosphere, and detailed texture and line transfer.
  • Figure 10 compares StyleExpert-500K dataset samples with OmniStyle-150K dataset samples.The comparison is presented as a sample-level visualization of the two datasets.
  • The style-analysis prompt defines shared artistic style through texture, line quality, and material or rendering characteristics rather than color, lighting, or atmosphere alone.It asks for a binary same-style judgment between a reference style image and an output image.
  • The caption-rewriting prompt removes style, color, texture, material, lighting, and atmosphere descriptors while retaining factual object, scene, and action details.The prompt is designed to produce content-only descriptions for downstream checking.
  • The content-matching prompt checks whether a stylized image strictly matches a cleaned caption while ignoring color, material, and style details.The surrounding prompt presents the stylized image and caption for a yes-or-no semantic consistency check.
Loading 2603.16649v3…