Source-linked AI summary
SDO: Subspace Deconflicting Operator for Multi-Adapter Composition
Zhongsheng Wang, Zhedong Lin, Qian Liu, Xinyu Zhang, Jiamou Liu
TL;DR
Composing independently trained adapters in a shared diffusion backbone can mix identities, leak attributes, and destabilize multi-character scenes. SDO rewrites adapter updates to suppress conflicting shared subspaces, improving identity fidelity and compositional stability, especially as more adapters are composed.
Problem
Independently trained adapters can conflict within shared backbone layers, making identity-consistent multi-character generation difficult as scene complexity increases.
Method
SDO extracts low-rank subspace signatures from adapter updates, estimates pairwise conflicts, and applies a permutation-equivariant rewrite before reconstructing deployable adapter parameters.
Results
SDO improves identity fidelity, character separation, and compositional stability, with the largest gains in challenging 4- and 5-adapter regimes.
Takeaways & Limitations
Explicit subspace-level reasoning is particularly beneficial for identity-consistent composition under high compositional complexity.
Takeaways & Limitations
The study focuses on incompatibility among adapter updates in parameter space rather than regulating adapter interactions during generation.
Abstract
from arXiv · showhide
Composing independently trained adapters within a shared diffusion backbone provides a modular approach to multi-character generation, but naive joint deployment often causes identity mixing, cross-character attribute leakage, and unstable scene composition. We study this interference from a parameter-space perspective and hypothesize that it arises partly from conflicts between overlapping dominant subspaces in shared layers. To address this issue, we propose \textbf{SDO}, a \textbf{S}ubspace \textbf{D}econflicting \textbf{O}perator for multi-adapter composition. SDO reconstructs layer-wise low-rank updates from the selected adapters, extracts compact subspace signatures, measures pairwise conflict through output-subspace overlap, and applies a permutation-equivariant transformation that suppresses harmful shared directions while retaining identity-specific characteristics. The resulting representations are mapped back to standard adapter updates and can be directly incorporated into existing diffusion inference pipelines. Experiments demonstrate that SDO consistently improves identity fidelity and compositional stability, with particularly clear gains as the number of jointly composed adapters increases.
1 Introduction
The introduction frames multi-character adapter composition as a parameter-space interference problem and proposes SDO, which deconflicts overlapping subspaces to preserve identities and stabilize composition. Experiments show the clearest gains in challenging four- and five-character settings.
- Motivation: Multi-character generation must preserve identity consistency, visual separation, and stable representation while rendering multiple identities in one coherent scene.The challenge grows with scene complexity and extends beyond single-subject realism.
- Problem: Independently trained adapters offer parameter-efficient, reusable, plug-and-play composition, but direct deployment in a shared diffusion backbone causes identity mixing, attribute leakage, and unstable composition.These failures make naive joint deployment unreliable.
- Limitation: Existing conditioning, attention, spatial, and inference-time methods regulate feature- or decoding-level interactions without directly resolving incompatibility among injected parameter updates.The paper therefore treats interference as a more fundamental shared-backbone parameter-space issue.
- Method: SDO reconstructs layer-wise adapter updates, extracts compact subspace signatures, estimates pairwise conflict, and applies a permutation-equivariant transformation that suppresses harmful overlap while preserving identity-specific behavior.The transformed representations are mapped back to standard adapter updates for integration with existing diffusion pipelines.
- Results: SDO improves identity fidelity, character separation, and compositional stability, with the largest gains in challenging 4- and 5-character settings.In these settings, naive composition and prior methods suffer severe identity mixing and attribute leakage.
2 Related Work
Prior work advances personalized diffusion from single-identity modeling toward multi-subject and inference-time multi-LoRA composition. However, existing methods mainly regulate generation or reduce directional overlap, while rarely characterizing conflicts among independently trained adapter updates in shared diffusion layers.
- Personalized diffusion: Personalized diffusion evolved from single-identity personalization toward composing multiple learned identities within one scene.DreamBooth, Textual Inversion, and Custom Diffusion established subject-specific personalization, while later methods improved scalability, controllability, and data efficiency.
- Multi-subject composition: Multi-subject methods use disentangled representations, attention and layout control, and structured or spatial denoising guidance to improve compositional controllability.These approaches reduce some common failures in multi-subject generation but leave jointly rendering multiple personalized subjects coherently as a harder problem.
- Multi-adapter composition: Inference-time multi-LoRA methods coordinate independently trained identity adapters, but primarily regulate adapter interactions during generation rather than rewriting adapter updates.This setting matches users composing selected subsets from a pool of independently trained identity adapters on demand.
- Weight-space composition: Adapter fusion and model-merging studies show that composition quality depends on weight-space geometry, with overlapping or conflicting updates causing destructive interference.AdapterFusion and Task Arithmetic motivate composable parameter directions, while TIES-Merging highlights failures from naive combination.
- Subspace interference: Although orthogonality-based methods reduce overlap among low-rank directions, existing approaches rarely characterize whether composed adapters occupy conflicting parameter-space directions in shared diffusion layers.Independently trained adapters must jointly modify shared layers in the same frozen diffusion backbone during generation; SDO addresses this gap.
3 Problem Definition
Section 3 formulates multi-adapter composition as a subspace-conflict problem in parameter space: independently trained character adapters are jointly deployed in a frozen diffusion backbone, where overlapping dominant directions can cause interference. The goal is to transform selected adapters into a more compatible set while preserving identity-specific behavior, mitigating conflict, and remaining permutation-equivariant.
- Problem Setup: A frozen text-to-image diffusion backbone jointly activates a selected subset of 2 to n independently trained character adapters for multi-character scene generation.Each adapter is represented by layer-wise updates injected into a designated layer set.
- Problem Setup: The common baseline composes selected adapters through layer-wise weighted summation, with each adapter assigned an inference loading scale.The singleton case requires no composition and reduces to standard single-adapter generation.
- Interference: Naive joint composition causes identity mixing, cross-character attribute leakage, and unstable rendering, with interference becoming more pronounced as more adapters are activated.The adapters are trained independently but interact when deployed together in the shared backbone.
- Subspace-Conflict View: The paper hypothesizes that interference arises partly when adapters’ dominant parameter-space directions substantially overlap and compete within shared layers.Adapters with distinct dominant directions tend to remain compatible, whereas overlapping directions are associated with increased interference.
- Transformation Objective: Given a selected adapter set, the objective is to learn a set-conditioned transformation that produces a more compatible set for joint deployment.The transformation is required to preserve identity-specific behavior, reduce joint-composition interference, and be independent of input ordering.
4 Subspace Deconflicting Operator
SDO treats multi-adapter interference as geometric conflict among overlapping dominant directions in shared backbone layers, rather than solely as an inference-time or prompting artifact. It rewrites selected adapters collectively with a permutation-equivariant, conflict-aware transformation before joint deployment.
- Motivation: Multi-adapter failures arise when independently trained updates injected into shared layers rely on overlapping dominant directions.SDO’s premise locates interference partly in parameter-space geometry rather than solely in inference-time or prompting artifacts.
- Conflict modeling: SDO suppresses geometric overlap among dominant adapter-specific directions while retaining update magnitude information in the signature representation.The conflict prior is pairwise and defined on dominant output-side bases; magnitude is further enforced through structure-level objectives.
- Set-level rewriting: SDO rewrites a selected adapter set before deployment instead of merging the adapters into a single module.At each layer ℓ, it applies a cardinality-preserving transformation to the selected set X^(ℓ).
- Set-level rewriting: The operator is set-conditioned and conflict-aware, so each transformed adapter depends on its own state, other adapters, and pairwise conflict relations.This produces a permutation-equivariant set-to-set mapping for context-aware adapter rewriting before joint deployment.
5 Implementation Workflow
SDO processes a selected adapter set layer by layer through low-rank signature extraction, permutation-equivariant conflict-aware transformation, and reconstruction into standard adapter updates. It preserves set structure and adapter correspondence while enabling direct use in standard multi-adapter inference pipelines.
- Layer-wise pipeline: At each injected layer, SDO transforms the selected adapter updates through three stages: signature extraction, set transformation, and structured reconstruction.The resulting updates can be directly incorporated into existing adapter-based inference.
- Signature extraction: Each adapter is compressed into a low-rank signature retaining dominant output directions and magnitudes, then vectorized for consistent set-level processing.The signature representation preserves the output-side structure needed for subsequent transformation.
- Conflict-aware transformation: SDO applies the transformation only to output-side factors while fixing input-side bases, preserving input selectivity and constraining rewriting to shared output-direction incompatibilities.The method avoids simultaneously modifying both the read-in and write-out structure of each update.
- Conflict-aware transformation: The core operator jointly updates adapter signatures using each adapter’s state, the surrounding set, and pairwise output-subspace conflicts while preserving unordered-set structure and input-output correspondence.Subspace overlap strengthens attention to potentially conflicting adapters during context-aware correction.
- Structured reconstruction: Updated signatures are reshaped and factorized to reconstruct valid low-rank updates, and SDO returns the same number of adapters for standard joint inference rather than a merged module.The reconstructed adapters retain transformed output subspaces and magnitude structure.
6 Experiments
Experiments evaluate SDO across identity pools, diffusion backbones, adapter counts, and complementary quantitative, qualitative, human, ablation, and subspace analyses. SDO is most beneficial for difficult 4- and 5-LoRA compositions, improving identity separation and compositional stability while remaining competitive in easier settings.
- Experimental Setup: Experiments compare SDO with four baselines across 2-, 3-, 4-, and 5-LoRA compositions in in-domain, cross-topic, and cross-backbone settings.The settings use SM and FL identity pools with FLUX and SDXL backbones; SDO is trained only on FLUX and tested on direct cross-backbone transfer.
- Quantitative Results: SDO delivers stronger identity preservation and more stable composition in 4- and 5-LoRA SM-FLUX and SM-SDXL settings.The cross-backbone result provides preliminary evidence that the learned deconfliction strategy transfers across backbones.
- Quantitative Results: In FL-FLUX, SDO remains competitive and shows its largest gains in the difficult 4- and 5-LoRA cases, though the margin is more modest.The passage attributes less pronounced improvements to weaker identity specificity and greater intra-identity variation in the FL pool.
- Ablation Study: Ablations show that the full SDO performs best, with removing conflict priors most clearly harming identity metrics and removing global context also degrading performance.Replacing structured reconstruction primarily affects compositional stability, while gaps are larger in the 5-LoRA setting.
- Qualitative and Human Evaluation: Qualitative and human evaluations show that SDO better preserves identity separation, character completeness, and coherent scenes as adapter count increases, with clearer advantages at 5 LoRAs.Baselines show identity blending, missing characters, and cross-character attribute leakage; human preference advantages are clear at 4 LoRAs and more pronounced at 5.
- Subspace Analysis: Subspace visualizations show clearer separation after SDO on both FLUX and SDXL, consistent with reduced average pairwise overlap and stronger gains in difficult compositions.The shared 2D PCA projection provides an intuitive geometric view suggesting reduced parameter-space conflict among selected adapters.
7 Conclusion
The conclusion frames multi-adapter interference as a parameter-space subspace-conflict problem and presents SDO as a conflict-aware rewriting operator before composition. SDO uses structured low-rank representations to reconstruct deployable adapter parameters and improve identity-consistent multi-adapter generation.
- SDO formulates multi-adapter interference as a subspace-conflict problem in parameter space.
- SDO is a learnable permutation-equivariant operator that performs conflict-aware rewriting before adapter composition.
- Structured low-rank representations let SDO reconstruct rewritten updates into deployable adapter parameters.
- SDO improves identity-consistent multi-adapter generation, with the strongest improvements reported in challenging 4- and 5-adapter regi.