PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion

Zhongjie Duan, Hong Zhang, Yingda Chen

arXiv:2604.24351v1cs.LGcs.AIcs.CVcs.SE

TL;DR

Controllable diffusion methods are often isolated by backbone, training recipe, and runtime interface, limiting their reuse and composition. Diffusion Templates addresses this fragmentation with a unified plugin framework, whose diverse case studies show that heterogeneous controls can share one runtime abstraction.

  • Problem

    Controllable diffusion methods are typically isolated around particular architectures, condition types, training recipes, parameter formats, and runtime hooks.

  • Method

    Diffusion Templates decouples base-model inference from capability injection through independently trainable Template models, a shared Template cache interface, and a pipeline for loading and composing controls.

  • Results

    The framework unifies controllable generation across a diverse model zoo spanning structural, attribute, editing, enhancement, reference, inpainting, and age-control tasks under one runtime abstraction.

  • Takeaways & Limitations

    Diffusion Templates provides a common systems interface for packaging and composing heterogeneous controllable-generation capabilities without repeatedly redesigning the underlying diffusion pipeline.

  • Takeaways & Limitations

    The current work mainly demonstrates representative qualitative examples, leaving quantitative evaluation of controllability, compositionality, transferability, efficiency, and compatibility for future studies.

Abstract

from arXiv · show

Controllable diffusion methods have substantially expanded the practical utility of diffusion models, but they are typically developed as isolated, backbone-specific systems with incompatible training pipelines, parameter formats, and runtime hooks. This fragmentation makes it difficult to reuse infrastructure across tasks, transfer capabilities across backbones, or compose multiple controls within a single generation pipeline. We present Diffusion Templates, a unified and open plugin framework that decouples base-model inference from controllable capability injection. The framework is organized around three components: Template models that map arbitrary task-specific inputs to an intermediate capability representation, a Template cache that functions as a standardized interface for capability injection, and a Template pipeline that loads, merges, and injects one or more Template caches into the base diffusion runtime. Because the interface is defined at the systems level rather than tied to a specific control architecture, heterogeneous capability carriers such as KV-Cache and LoRA can be supported under the same abstraction. Based on this design, we build a diverse model zoo spanning structural control, brightness adjustment, color adjustment, image editing, super-resolution, sharpness enhancement, aesthetic alignment, content reference, local inpainting, and age control. These case studies show that Diffusion Templates can unify a broad range of controllable generation tasks while preserving modularity, composability, and practical extensibility across rapidly evolving diffusion backbones. All resources will be open sourced, including code, models, and datasets.

1 Introduction

Diffusion Templates presents controllable generation capabilities as reusable plugins, decoupling base-model inference from capability injection through a common systems interface. The framework standardizes heterogeneous controls across training and deployment and supports a diverse model zoo spanning many controllable generation tasks.

  • Motivation: Controllable diffusion methods are effective but typically isolated by model architecture, condition type, and training recipe.
  • Motivation: This fragmentation hinders infrastructure reuse, capability transfer across backbones, and deployment integration because methods use different modifications, parameter formats, preprocessing, objectives, and runtime hooks.
  • Framework: Diffusion Templates decouples base-model inference from controllable capability injection and treats capabilities as reusable plugins with a common interface for training, loading, and composition.
  • Framework: Template models convert task-specific inputs into Template caches, whose interface supports heterogeneous mediating representations such as KV-Cache and LoRA.
  • Scope and results: The released model zoo spans structural control, attribute adjustment, editing, enhancement, reference, inpainting, and age-control tasks under one runtime abstraction.

2 Related Work

Diffusion research progressed from latent and transformer-based foundation models to diverse controllable-generation methods, including parameter-efficient adaptation, structural adapters, and image-conditioned control. Diffusion Templates addresses their fragmented integration through reusable capability interfaces inspired by modular plugin systems and broader KV-Cache abstractions.

  • Diffusion Foundation Models: Latent diffusion, exemplified by LDM, made high-quality generation computationally feasible at scale, while Stable Diffusion evolved through SD-XL and Stable Diffusion 3.These developments established increasingly capable diffusion foundation models for downstream control interfaces.
  • Controllable Generation: Controllable diffusion methods include LoRA, Textual Inversion, DreamBooth, ControlNet, T2I-Adapter, and IP-Adapter for specialization, personalization, structural constraints, and image-conditioned control.The cited methods differ in how they adapt models or inject textual, structural, and image conditions.
  • Fragmentation and Integration: Most control techniques remain isolated through distinct training scripts, parameter formats, and runtime hooks, making multi-control composition costly and engineering-intensive.Diffusion Templates treats each control method as a pluggable capability under a unified interface to reduce integration and maintenance cost while preserving composability.
  • Plugin and Runtime Abstractions: Plugin and tool-use systems in LLMs motivate stable capability contracts, while KV-Cache research supports viewing intermediate runtime state as a reusable interface beyond efficiency optimization.Diffusion Templates similarly treats the diffusion base model as a core runtime and controls as independently activated, composed, and scheduled plugins.

3 Framework Design

Diffusion Templates separates base diffusion inference from reusable control-capability injection through a standardized interface. Its Template cache, Template model, and Template pipeline components support flexible capability definition, cache composition, and unified execution across controllable diffusion tasks.

  • Framework Overview: The framework decouples base-model generation from external control-capability injection, allowing multiple capabilities to be activated, composed, and scheduled without modifying denoising internals.The base pipeline remains responsible for generation quality, while Template models provide reusable control signals through an intermediate interface.
  • Core Components: The framework comprises three components: Template models map task-specific inputs to standardized caches, Template cache represents capabilities, and Template pipeline loads and composes them.Template input formats are defined by the corresponding Template models.
  • Template Cache: Template cache is defined through diffusion-pipeline input arguments, aligning with existing engineering abstractions and enabling stable plugin deployment across downstream tasks.This avoids rewriting denoising internals and supports capability integration by extending pipeline arguments.
  • Template Cache: KV-Cache is the recommended Template cache because it supports strong representation, direct generation influence, sequence concatenation, and limited framework modification.Its concatenation enables multiple templates to operate jointly under the same pipeline.
  • Template Models and Training: Template models expose separate no-gradient process inputs and gradient-related forward interfaces, preserving architectural flexibility while enabling unified runtime execution and staged training.Stage I produces cacheable intermediate features, while Stage II optimizes the gradient-relevant forward path under Template-cache objectives.
  • Template Pipeline: Template pipeline inference runs each Template model, merges caches according to cache type, and passes the merged cache with normal generation arguments to the base diffusion pipeline.Template models execute outside the iterative denoising loop; round-robin scheduling with lazy loading can reduce peak memory usage.

4 Model Zoo · 4.1 Structural Control · 4.2 Brightness Adjustment

The model zoo evaluates Diffusion Templates through diverse Template models trained on FLUX.2-klein-base-4B6 under fixed generation settings. Structural control uses KV-Cache to support multiple structural conditions, while brightness adjustment uses a dedicated lightweight model conditioned on normalized mean RGB intensity.

  • 4 Model Zoo: The model zoo trains diverse Template models on FLUX.2-klein-base-4B6 to evaluate Diffusion Templates’ expressiveness and extensibility.Unless otherwise specified, images use random seed 0, classifier-free guidance scale 4, and 50 inference steps.
  • 4 Model Zoo: Experiments use a fixed random seed of 0, classifier-free guidance scale of 4, and 50 inference steps.These settings apply to all images in the section unless otherwise specified.
  • 4.1 Structural Control: The structural-control Template model communicates structural information through KV-Cache rather than residual branches.This follows the general ControlNet idea of augmenting a frozen pretrained diffusion backbone with a trainable branch, while changing the control-signal injection mechanism.
  • 4.1 Structural Control: The structural-control model supports four types of structural conditions.The supplied passage introduces the four-condition capability but does not enumerate all four types.
  • 4.2 Brightness Adjustment: Naively rescaling RGB intensities for brightness control often produces visually unnatural results.The framework therefore uses a dedicated brightness-adjustment Template model instead of direct RGB rescaling.
  • 4.2 Brightness Adjustment: The brightness-adjustment Template follows AttriCtrl’s lightweight architecture, using a positional encoding layer and several fully connected layers.This design is trained specifically for brightness adjustment.
  • 4.2 Brightness Adjustment: During training, brightness is represented by a scalar equal to mean RGB intensity normalized to [0, 1].This scalar serves as the control input to the brightness-adjustment Template model.

4.3 Color Adjustment · 4.4 Image Editing · 4.5 Super-Resolution

The paper extends Diffusion Templates to finer-grained color adjustment, image editing, and super-resolution. These Templates preserve controllability or editing quality while using shared architectures and, for super-resolution, recovering high-frequency details after resizing.

  • 4.3 Color Adjustment: Color Adjustment: The Template model controls color through three inputs representing the mean R, G, and B channel values.Its training pipeline is otherwise identical to brightness adjustment.
  • 4.3 Color Adjustment: Color Adjustment: The color control is soft rather than exact, producing coherent channel-value trade-offs instead of pixel-by-pixel target matching.Figure 4 illustrates warm, natural, and cool outcomes using colors #D0B98A, #808080, and #5EA3AE.
  • 4.4 Image Editing: Image Editing: The authors train a Template model with the structural-control architecture to transfer the base model’s editing capability into the Template pathway.This addresses the increased sequence length and expense of editing relative to pure text-to-image generation.
  • 4.4 Image Editing: Image Editing: The resulting Template achieves editing quality comparable to the base model while providing an empirical efficiency benefit.The supplied passage states this benefit but does not provide its numerical value.
  • 4.5 Super-Resolution: Super-Resolution: The authors train a Template model to evaluate framework task coverage, using the same architecture as the image-editing Template.This evaluation is motivated by specialized models such as Real-ESRGAN, which are highly effective for super-resolution.
  • 4.5 Super-Resolution: Super-Resolution: The model bilinearly resizes low-resolution images to target resolution, then recovers missing high-frequency details without explicitly modeling an upscaling factor.Figure 6 shows sharp outputs at large scaling factors, although the approach remains slower than dedicated super-resolution pipelines.

4.6 Sharpness Enhancement

The sharpness-enhancement control uses quantile-normalized Canny edge density as a lightweight proxy for relative sharpness, with lower control values producing softer images.

  • Sharpness Enhancement: Sharpness is controlled by computing the fraction of Canny edge pixels and quantile-normalizing it to [0, 1].The resulting edge-density statistic is provided as the model input.
  • Sharpness Enhancement: Because sharper images generally contain richer high-frequency boundaries, edge density serves as a practical proxy for relative sharpness.Figure 7 shows that lower control values yield softer images.

4.7 Aesthetic Alignment · 4.8 Content Reference

Aesthetic alignment addresses subjective image properties using preference-based supervision, while content reference converts reference-image representations into injectable LoRA capabilities. The content transferred from a reference image remains only partially controllable and may emphasize style or concrete attributes.

  • 4.7 Aesthetic Alignment: Aesthetic alignment is challenging because reliable continuous supervision is often unavailable for subjective properties such as aesthetics.Brightness, color, and sharpness can be measured directly from images, unlike aesthetics.
  • 4.7 Aesthetic Alignment: Pairwise human preference annotations from GenAI-Arena and Pick-a-Pic provide an alternative supervision format for aesthetic alignment.Annotators indicate which of two images they prefer.
  • 4.8 Content Reference: The content-reference model uses SigLIP2 as an image encoder and maps its visual representation to LoRA weights through fully connected layers.It is trained on an image-text paired dataset.
  • 4.8 Content Reference: Reference images can be converted directly into LoRA representations and injected into the generation pipeline to condition new images on extracted information.This extends the aesthetic-alignment experiment’s use of LoRA as a capability carrier.
  • 4.8 Content Reference: Content reference supports image editing with a shared reference image across prompts such as adding a hat or changing a cat’s head direction.Figure 5 illustrates these two editing conditions.
  • 4.8 Content Reference: Figures 6 and 7 show super-resolution and sharpness-control results using the shared prompt “A cat is sitting on a stone.”The figures use the same prompt for their respective demonstrations.
  • 4.8 Content Reference: The transferred reference content is not explicitly controllable and may emphasize global visual style or preserve more concrete attributes.Figure 9 presents these differing transfer behaviors.

4.9 Local Inpainting

Local inpainting regenerates only a masked image region while preserving the remaining content, using a dedicated Template model whose control is soft rather than strictly guaranteed.

  • Task definition: Local inpainting takes an input image and a mask indicating the region to regenerate.The objective is to modify only the masked area while preserving all remaining content.
  • Template model: The framework trains a dedicated local-inpainting Template model for this task.
  • Control limitation: The model provides soft control and cannot strictly guarantee that unmasked regions remain completely unchanged.Such failures are described as infrequent.

4.10 Age Control

The study evaluates age control in human portrait generation using a scalar-controlled Template model trained on IMDB-WIKI, testing whether the brightness-adjustment formulation extends to richer human attributes.

  • Age Control: The age-control model is trained on the IMDB-WIKI dataset for human portrait generation.The passage frames this as an evaluation of Template-model controllability.
  • Age Control: The model uses exactly the same architecture as the brightness-adjustment model.This enables a direct test of extending the same scalar-control formulation beyond low-level visual attributes.
  • Age Control: The control signal is a scalar age value ranging from 10 to 90.

4.11 Template Fusion

Template Fusion enables multiple Template models to operate within one generation pipeline, with the fusion strategy determined by each model’s cache format. KV-Cache and LoRA caches use format-specific concatenation, while heterogeneous cache formats can be enabled simultaneously without conversion.

  • Cache-format-specific fusion: Multiple Template models can be fused within a single generation pipeline, with the strategy determined by their emitted cache formats.The framework supports fusion at the pipeline level rather than requiring a single cache representation.
  • Cache-format-specific fusion: KV-Cache fusion concatenates caches along the sequence dimension.This applies to Template models whose cache representation is KV-Cache.
  • Cache-format-specific fusion: LoRA fusion concatenates corresponding LoRA parameters along the rank dimension.This applies to Template models whose cache representation is LoRA.
  • Heterogeneous fusion: Heterogeneous cache formats can be fused by enabling their associated modules simultaneously without converting them to a unified representation.The modules can operate together even when their Template models emit caches in different formats.

5 Conclusion and Future Work

Diffusion Templates is presented as a unified, open plugin framework that decouples base-model inference from capability injection through reusable Template models and a shared Template cache interface. As a prototype, it motivates future work on more efficient interfaces, broader foundation-model support including video, and standardized quantitative evaluation.

  • Conclusion: Diffusion Templates decouples base-model inference from capability injection, packaging heterogeneous controllable generation methods as reusable Template models connected through a shared Template cache interface.The design targets modularity during both training and deployment.
  • Future Work: The prototype framework requires continued improvement in functionality and practical usability.The paper identifies several directions for future development.
  • Future Work: Future work should explore Template cache formats offering improved efficiency, compatibility, or controllability across model architectures and downstream tasks.KV-Cache and LoRA are the current capability interfaces, but alternatives may provide better trade-offs.
  • Future Work: The framework should extend beyond image-generation backbones to video-generation models, supporting flexible control over temporal consistency, motion patterns, and compositional structure.This direction seeks broader foundation-model coverage.
  • Future Work: Future studies should quantitatively evaluate controllability, compositionality, transferability, efficiency, and compatibility under standardized benchmarks.Such benchmarks would enable more rigorous comparison among Template models than the current qualitative demonstrations.
Loading 2604.24351v1…