PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Text-Guided Convolutional Adapter for the Diffusion Model

Aryan Das, Koushik Biswas, Swalpa Kumar Roy, Badri Narayana Patro, Vinay Kumar Verma

arXiv:2602.14514v2cs.CV

TL;DR

Existing structure-preserving diffusion methods can be parameter-heavy and often treat structural controls independently from text prompts. Nexus Prime and Slim introduce prompt-guided, efficient adapters using cross-attention while preserving a frozen backbone, with experiments reporting competitive or state-of-the-art performance and reduced computational cost.

  • Problem

    Existing structure-preserving adapters can introduce substantial parameter overhead and generally fail to leverage semantic synergy between visual controls and text prompts.

  • Method

    Nexus Adapters are lightweight auxiliary modules that jointly condition diffusion generation on structural inputs and text prompts through cross-attention while leaving the denoising backbone frozen.

  • Results

    Nexus Prime achieves superior fidelity, while Nexus Slim offers competitive performance with reduced computational cost across various conditioning tasks.

  • Takeaways & Limitations

    Prompt-aware adapters can narrow the gap with heavier baselines while retaining lightweight design advantages for fine-grained conditional image generation.

  • Takeaways & Limitations

    The method relies on a pretrained Latent Diffusion Model with a frozen denoising UNet as its generative backbone.

Abstract

from arXiv · show

We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters

1. Introduction

Diffusion models generate semantically rich images but often struggle with precise structure and fine details. Nexus Adapters address this gap with prompt-aware, parameter-efficient conditioning that jointly uses structural and textual cues.

  • Motivation: Diffusion models often struggle to follow exact layouts, structures, and small visual details from text prompts alone.This limits applications combining high-level semantics with low-level controls such as edge maps, segmentation masks, or human poses.
  • Existing limitations: ControlNet and T2I-Adapter add structural conditioning, but existing approaches can require backbone modification, extensive fine-tuning, or substantial parameter overhead.ControlNet-family architectures may approach the size of the base model encoder, while T2I-Adapter provides lightweight structural injection.
  • Existing limitations: Existing adapters commonly process structural inputs independently of prompts, preserving structure without leveraging the prompt’s semantic context.Prompt-aware adaptation is presented as a way to provide stronger guidance and support more effective task adaptation.
  • Nexus contribution: Nexus Adapters jointly attend to prompt and structural cues through cross-attention within an efficient convolutional architecture.The proposed design targets conditional alignment while using fewer parameters than heavier adapter alternatives.
  • Nexus contribution: Extensive experiments across diverse conditional tasks show that Nexus Adapters parallel or surpass recent baselines while remaining highly efficient.The contribution statement identifies state-of-the-art performance and efficiency as central outcomes.

2. Related Work

Related work spans generative modeling, diffusion-based text-to-image synthesis, structure-preserving methods, and modular adapters for controllable generation. These approaches improve controllability but retain challenges in fine-grained structure, efficiency, and joint prompt-condition modeling.

  • Image synthesis: Early image synthesis research used GANs, VAEs, and flow-based models to address realistic generation through efficient sampling and stable optimization.Initial work focused largely on unconditional generation before conditional synthesis gained attention.
  • Diffusion models: Diffusion models generate images by progressively denoising Gaussian noise and have become prominent for text-to-image synthesis.Models including GLIDE, DALL-E, Imagen, and Stable Diffusion achieve strong visual fidelity but may miss fine-grained structure or spatial consistency.
  • Structure preservation: Structure-preserving methods seek to align text semantics with structural features through optimization, attention manipulation, or guidance gradients.These methods may require inference-time optimization or lack direct parameter learning, reducing efficiency.
  • Controllable diffusion: ControlNet and T2I-Adapter extend frozen diffusion backbones with trainable modules for conditions such as depth, edges, and segmentation maps.Although these modules improve controllability, adapters can match or exceed the base model’s parameter size.
  • Efficiency and prompt awareness: Existing structure-preserving approaches often introduce substantial adapter overhead and process visual controls and text prompts independently.This separation prevents them from leveraging semantic synergy between the two inputs during feature modulation.

3. Methodology

The method combines a frozen Stable Diffusion backbone with a lightweight Nexus Adapter that injects structural and prompt information during denoising. Nexus Blocks extract multiscale condition features and use cross-attention to align them with frozen CLIP text embeddings before fusion into the UNet.

  • Diffusion backbone: A pretrained latent diffusion model uses a convolutional autoencoder and a frozen UNet to denoise noisy latent representations under CLIP text conditioning.The autoencoder encodes and decodes images, while the UNet estimates clean latents during reverse diffusion.
  • Nexus Adapter: Condition images are transformed into multiscale features, while prompts are embedded by a frozen CLIP encoder for joint structural and semantic guidance.The condition image is downsampled and processed through K = 4 hierarchical transformation blocks.
  • Nexus Adapter: The Nexus Adapter operates alongside the frozen UNet, preserving its denoising prior while injecting task-specific condition signals through additive feature fusion.The adapter introduces no modifications to the frozen backbone.
  • Nexus Blocks: Each Nexus Block refines visual features and applies cross-attention with text-derived keys and values to modulate spatial representations semantically.The attention output is reshaped and combined with the input through a residual connection.
  • Nexus Blocks: The Prime Block uses sequential standard convolutions, whereas the Slim Block uses depthwise convolutions to reduce computational cost and parameter count.Prime applies 3 × 3 and 1 × 1 convolutions; Slim uses depthwise convolutions with group size 2.

4. Experiment

The study evaluates the proposed architecture across four structural conditioning modalities using COCO 2017, with quantitative assessment on 5k validation samples.

  • The experiments compare model performance against recent strong baselines across diverse tasks.
  • Four conditioning modalities—Canny, Depth, Sketch, and Segmentation—are evaluated independently on COCO 2017, which contains approximately 164k training images.Segmentation uses COCO-Stuff annotations, Depth uses MiDaS-generated maps, Sketch uses extracted edge maps, and Canny uses standard edge detection.
  • Quantitative evaluation uses 5k COCO validation samples to measure visual fidelity and semantic alignment.FID measures visual fidelity, while CLIP Score with ViT-L/14 assesses alignment between generated images and input prompts.
  • All evaluated models use Stable Diffusion v1.5 with frozen backbone weights.

5. Results and Discussion

Across quantitative, qualitative, and ablation studies, Nexus Prime provides the strongest semantic–structural generation quality, while Nexus Slim offers lower complexity with competitive results and robustness.

  • Quantitative Comparison: 23.77 GFlops and 59.29M total parameters make Nexus Slim the lowest-complexity adapter, while Nexus Prime uses 33.32 GFlops and 85.82M total parameters.T2I-Adapter requires 29.97 GFlops and 77.37M total parameters.
  • Quantitative Comparison: Nexus Prime achieves the highest CLIP scores on Canny, Depth, and Sketch, ranks second on Segmentation, and Nexus Slim surpasses T2I-Adapter on most tasks.The results identify Prime as the semantic–structural leader and Slim as the efficiency-oriented alternative.
  • Quantitative Comparison: Nexus Prime attains the lowest FID on Canny, Depth, and Sketch, while Nexus Slim remains competitive and offers an efficiency–quality compromise.Nexus Slim ranks within the top three across tasks and outperforms several baselines.
  • Qualitative Comparison: Qualitative comparisons across Canny, Depth, Sketch, and Segmentation show Nexus Prime best preserving geometric, semantic, and fine visual details, with Nexus Slim generally performing strongly.The comparison includes reference images, condition maps, and outputs from baseline and proposed models.
  • Ablation Studies: Without prompts, Nexus adapters preserve structural fidelity and semantic coherence with minimal degradation, whereas ControlNet-based methods degrade sharply in prompt-free settings.The validation compares removing prompts from the adapter alone with removing them from both Stable Diffusion and the adapter.

6. Conclusion

Nexus Prime and Slim are efficient, prompt-aware adapters that integrate structural inputs with textual prompts through cross-attention while leaving the diffusion backbone frozen. Experiments show Prime provides superior fidelity, while Slim offers competitive performance at lower computational cost.

  • Nexus Prime and Slim integrate structural inputs and textual prompts using cross-attention without altering the frozen diffusion backbone.This addresses the prompt-independence of earlier adapters while preserving conditional structure.
  • Nexus Prime achieves superior fidelity across conditioning tasks.
  • Nexus Slim offers competitive performance with reduced computational cost.
  • Quantitative and qualitative analyses confirm the effectiveness of the proposed adapters for controllable, high-quality image generation.

7. Additional Experiments

Additional experiments evaluate Nexus Adapters on fine-grained CUB-200 generation without fine-tuning and after 1k-step fine-tuning. Fine-tuning improves performance across Sketch and Depth conditions, while the adapters remain competitive with heavier state-of-the-art models.

  • CUB-200 experiments compare Nexus Adapters with baselines for fine-grained image generation under Sketch and Depth conditions.The evaluation includes configurations without fine-tuning and with fine-tuning.
  • 1k-step fine-tuning is used to measure adaptation and efficacy against established baselines.
  • Fine-tuning consistently improves performance across both Sketch and Depth conditions.
  • Nexus Slim and Prime remain competitive with ControlNet++, CtrLoRA, and UniCon while retaining lightweight design advantages.

8. Training and Inference Details

Training uses mixed precision for both Nexus architectures, while inference applies classifier-free guidance with 35 denoising steps. Cost comparisons position ControlNet-based methods as substantially more resource-intensive than T2I-Adapter.

  • Training: Training uses batch size 2 with four gradient-accumulation steps, effectively simulating batch size 8.The setup stabilizes training under GPU memory constraints.
  • Inference: Inference uses classifier-free guidance at scale 7.5 and 35 denoising steps.This balances text-prompt fidelity, structural adherence, image quality, and computational efficiency.
  • Cost comparison: ControlNet and ControlNet++ each require ∼129 hours of training and 38 ms per image, while CtrLoRA requires 150 hours and 45 ms.These estimates are measured over 200K steps with batch size 2 and average per-image latency on one A100 GPU.

9. Additional Ablations

Ablations examine adapter depth and behavior under conflicting prompts. Four adapter blocks provide the best reported FID and CLIP scores, while prompt-aware conditioning supports structural preservation and semantic alignment.

  • Conflict through Prompt: Conflicting-prompt tests assess whether generated objects respect both conditional-image structure and textual prompts.
  • Conflict through Prompt: Nexus adapters achieve a stronger balance between prompt and condition integration than existing baselines.The reported comparison emphasizes semantic alignment alongside structural preservation.
  • Number of Adapter Blocks: The 4-block configuration achieves the best FID and CLIP scores for Depth and Sketch conditioning.Increasing block count consistently improves both metrics.
Loading 2602.14514v2…