Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models

1 Introduction

• LoRA’s low-cost personalization and sharing create moderation risks when adapters encode problematic content that uploaders do not faithfully disclose.Existing difference-discovery and concept-verification methods leave open what to seek when the hidden concept is unknown.Source: §1 · p. 1

discover?

• Hidden-concept discovery seeks a conditioning input that jointly maximizes output coherence, LoRA-induced distinctiveness, and LoRA-subspace alignment.These criteria combine generation behavior with evidence from the adapter’s learned parameters.Source: discover? · p. 2
• Inputs with stronger alignment to the learned LoRA subspace are more likely to induce coherent and distinctive LoRA-specific behavior after rendering.This correspondence provides a render-free signal for concentrating expensive visual evaluation on promising candidates.Source: discover? · p. 2
• SGES searches the open-ended prompt space using subspace alignment alone, then PVI renders shortlisted prompts with both the LoRA and base model for visual identification.The stages reserve diffusion rendering for a compact candidate set.Source: discover? · p. 2
• The paper identifies coherence, distinctiveness, and subspace alignment as essential characteristics of a forensic target, and uses their correspondence to motivate SGES and PVI.The resulting framework performs large-scale search without diffusion rendering and reserves visual evaluation for a compact set of candidates.Source: discover? · p. 2
• The work constructs a diverse benchmark, compares representative discovery and verification methods, and introduces a distributional metric for fidelity to hidden LoRA content.The metric evaluates how faithfully a discovered result captures the encoded content.Source: discover? · p. 2

2.1 Diffusion Models

• Diffusion models learn to reverse a gradual noising process, while text representations condition denoising through cross-attention.Text tokens and their context affect generation, and inference can strengthen conditioning by combining conditional and unconditional noise predictions.Source: §2.1 · pp. 2–3

2.2 Low-Rank Adaptation

• LoRA freezes pretrained weights and learns compact low-rank updates, reducing trainable parameters while enabling personalization from relatively small training sets.Only the low-rank factors are optimized.Source: §2.2 · p. 3
• LoRA modules can modify text representations, denoising behavior, or their interactions depending on the adapted projections.The resulting weights can be stored and shared independently of the base model.Source: §2.2 · p. 3
• The leading right singular directions define the update’s dominant input subspace, and inputs orthogonal to it produce no LoRA response.This input-space interpretation motivates the paper’s parameter-space design.Source: §2.2 · p. 3

2.3 Content Moderation for LoRA Adapters

• Difference-discovery methods identify model changes, but adaptation can also alter collateral concepts, so detected changes do not identify the creator-intended target.Output-based discovery may also require extensive diffusion rendering during search.Source: §2.3 · p. 3
• Concept-specified methods such as prompt recovery and direct verification test concepts supplied in advance, limiting their coverage when the concept is unknown.The paper instead formulates creator-intended concept discovery jointly from LoRA parameters and generation behavior.Source: §2.3 · p. 3

3.1 Problem Setting

• The dominant hidden concept is the creator-intended primary training target repeatedly reinforced during adaptation, distinguished from incidental collateral concepts.Discovery assumes access to the adapter and compatible base model but not reliable metadata, triggers, reference images, or training data.Source: §3.1 · p. 3
• The formulation represents prompt-conditioned LoRA and base-model outputs as semantic distributions over diffusion randomness.A fixed semantic image encoder maps generated images into normalized representations.Source: §3.1 · pp. 3–4

3.2 Dominant Hidden Concept Formulation

• Output coherence favors little variation among LoRA samples, while LoRA-induced distinctiveness favors outputs that differ from the compatible base model.Coherence alone is insufficient because a consistent output may be a common concept already supported by the base model.Source: §3.2 · p. 4
• Subspace alignment favors prompts matching leading right-singular input directions, based on the assumption that the primary target leaves stronger evidence there than incidental content.The alignment functional assigns higher values to stronger prompt–subspace alignment.Source: §3.2 · p. 4
• The dominant carrier prompt combines the three properties, with nonnegative weights balancing coherence, distinctiveness, and subspace alignment.Its induced output distribution represents the discovered concept.Source: §3.2 · p. 4

3.3 Target-Concept Discrepancy

• Target-concept discrepancy measures expected cosine distance between LoRA outputs induced by the carrier prompt and the target concept’s semantic mean direction.A smaller discrepancy indicates a more faithful reveal of the target concept.Source: §3.3 · p. 4
• Evaluation estimates the target reference direction from finite reference images and averages cosine distance over generated outputs, but this measure is unavailable during hidden-concept discovery.It is used only for evaluation.Source: §3.3 · p. 4

3.4 Threat Model

• The moderator has white-box access to the adapter and compatible base model, can inspect parameters and query both models, but lacks training data, known triggers, predefined queries, and reference images.The goal is to recover a carrier prompt using only the available adapter and base-model information.Source: §3.4 · pp. 4–5

4.1 Framework Overview

• SCOPE separates render-free alignment search from output-side identification, returning the highest-ranked candidate as the dominant carrier prompt.SGES produces the high-alignment shortlist; PVI renders it, rejects less-coherent LoRA outputs, and ranks survivors using consensus and empirical distinctiveness.Source: §4.1 · p. 5

4.2 Subspace-Guided Evolutionary Search

• SGES evaluates LoRA-subspace alignment without diffusion rendering, narrows the prompt space through Atomic pools and contextual SAS scoring, and uses evolutionary search to form a shortlist.Because alignment is context dependent, SGES re-evaluates Atomics within complete prompts.Source: §4.2 · pp. 5–6, Figure 1 · p. 6

4.3 Paired Visual Identification

• PVI renders shortlisted prompts with LoRA and base models under matched seeds, filters for coherence, then ranks survivors by cross-prompt consensus and LoRA-induced distinctiveness.Consensus suppresses visual outliers, while distinctiveness favors outputs separated from the base model.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

5.2.1 Hidden-Concept Discovery

• On the common faithful SD1.5 cohort, SCOPE has the lowest discrepancy in every category (adapter-weighted mean 0.114), 70.2% below no-concept VisDiff (0.383) and 37.7% below image-assisted PEZ (0.183), without concept input.Source: Table 1 · p. 8, §5.2.1 · p. 8

5.2.2 Concept-Specificity Verification

• 0.980 accuracy, precision, recall, and F1 make SCOPE the top performer on concept-specificity verification, with balanced precision and recall.PAIA reaches perfect recall but 0.911 precision, while PEZ has 0.959 precision and 0.922 recall; SCOPE uses one fixed discovered prompt for both target and non-target decisions.Source: Table 2 · p. 9, §5.2.2 · p. 9

5.2.3 Composition of the Top-Ranked Prompt

• Top-ranked prompts combine trigger fragments and target-related descriptions; multi-unit names can be recovered through composed-word Atomics rather than exact trigger reconstruction.The examples include composed forms for Chun-Li and Brad Pitt, separate Word Atomics for Elise Rowan, and a semantic route for Watercolor.Source: §5.2.3 · p. 9, Figure 3 · p. 10

5.3.1 Impact of the Core Components

• Removing SGES raises mean discrepancy to 0.393, removing PVI to 0.130, and removing cross-prompt consensus to 0.208.Removing composed-word discovery raises the mean from 0.114 to 0.122; removing intra-prompt coherence filtering changes it from 0.123 to 0.114, while distinctiveness filtering changes it from 0.119 to 0.114.Source: Table 3 · p. 10, §5.3.1 · p. 10

5.2.4 Visual Discovery Results

• SCOPE’s returned prompts preserve the defining identities in the Chun-Li and Julian Frost examples, while PEZ’s anime result recovers the domain rather than the same character.PEZ receives target-concept images; SCOPE and the other displayed methods do not receive target-concept input.Source: §5.2.4 · pp. 9–10, Figure 4 · p. 11

5.2.5 Discovery Across Backbones and Concept Types

• For SD1.5 and SDXL, all concept-type mean discrepancies are below 0.3, including style discovery at 0.264 and 0.249, respectively.The visual examples show returned prompts capturing shared oil-painting and stained-glass characteristics across different subjects and scenes.Source: §5.2.5 · p. 10, Figure 5 · p. 11

5.3.2 Impact of the LoRA Subspace Selection

• Leading LoRA input directions yield 0.114 mean discrepancy, versus 0.334 for dimension-matched random directions and 0.398 for trailing directions.The tested retained dimensions produce means of 0.128, 0.114, and 0.127 for k = 1, 2, and 4, respectively.Source: Table 4 · p. 11, §5.3.2 · p. 11

5.3.3–5.3.4 Search Strategy and Site Grouping

• Evolutionary search reaches 0.114 mean discrepancy, outperforming Random at 0.135 and Greedy at 0.173; merging retained site candidates raises the mean to 0.136.Removing UP raises the overall mean to 0.183, while removing DOWN raises synthetic-identity discrepancy from 0.178 to 0.271; removing TE degrades every category.Source: Table 5 · p. 12, §5.3.3 · p. 12, Table 6 · p. 12

• SGES concentrates search on parameter-supported prompts, while PVI filters by coherence, cross-prompt consensus, and LoRA-induced distinctiveness.Source: §6 · p. 13
• SCOPE returns a top-ranked carrier prompt and generated images for reviewers, even when the prompt does not reproduce the original trigger.Source: §6 · p. 13
• The output supports moderation and downstream investigation, not enforcement, and failure to find a carrier is not evidence of benignity or no undisclosed concept.Source: §6 · p. 13

7 Limitations

• The evaluation covers separately loaded, U-Net–based LoRAs for SD1.5 and SDXL under white-box access to adapters and compatible base models.The implementation searches short English-language, ASCII-only Atomics at text-conditioned LoRA sites.Source: §7 · p. 13
• Results do not establish performance for DiT generators, merged or quantized adapters, black-box access, or multilingual and nonlexical carriers.Source: §7 · p. 13
• The formulation targets one dominant hidden concept, so adapters with several comparably strong concepts or concepts outside inspected subspaces may require broader discovery.Source: §7 · p. 13
• Evaluation uses CLIP representations, one mean reference direction, one controlled non-target per adapter, and one fixed subspace-dispersion schedule.Broader safety concepts, attacks, independent semantic encoders, human evaluation, and runtime measurement remain important before deployment.Source: §7 · p. 13

8 Conclusion

• 0.114 mean target-concept discrepancy on the faithful SD1.5 cohort is 70.2% lower than the strongest same-setting no-concept-input baseline and 37.7% lower than reference-assisted PEZ.Source: §8 · p. 13
• 0.980 precision, recall, and F1 are attained in controlled target-specificity evaluation; conditional evasion under adaptive subspace dispersion is 4.0% among adapters preserving target-generation utility.Source: §8 · p. 13
• SCOPE extracts parameter-supported candidates before visual confirmation, without assuming the concept a reviewer should test, and returns prompt–image pairs for human review.Source: §8 · p. 13

A.1 Text-Conditioning Site Selection

• SGES retains text-encoder and text cross-attention key/value sites whose inputs are available from text conditioning, excluding sites dependent on latent or later denoising computations.It forms updates from valid LoRA factor pairs and uses their dominant input directions; only groups with retained sites participate.Source: §A.1 · pp. 15–16

A.2 Composed-Word Discovery

• SGES composes two- and three-unit words, validates tokenizer round-trips, and filters proposals by alignment, lexical-unit overlap, and repeated-unit use before retaining up to 75 per group.Source: §A.2 · p. 16

A.3 Atomic Candidate Pool Construction

• 3,000 frequent corpus phrases complement native and composed tokens; each present group retains 30 TOKEN and 20 PHRASE Atomics ranked by LoRA alignment.Source: §A.3 · pp. 16–17

A.4 Evolutionary Search and Shortlist Construction

• Each group contributes three prompt slots; SGES evolves 48 prompts for 100 generations across three runs, preserving family diversity and refining candidates before shortlist construction.Shorter candidates are also formed by removing Atomics while preserving representation from each originally present group.Source: §A.4 · p. 17

B.1 Benchmark and Models

• 164 adapters across SD1.5 and SDXL cover real and synthetic identities, anime characters, and artistic styles.The set combines 48 adapters from public model-sharing platforms with 116 trained locally.Source: §B.1 · p. 17
• 116 locally trained adapters include identity concepts trained for both SD1.5 and SDXL, with matched variants differing only in whether the trigger is human-readable or opaque.The variants use identical training images and optimization settings.Source: §B.1 · p. 17
• 106 adapters use faithful triggers and 58 use unfaithful triggers; baseline comparisons and main ablations use a fixed faithfully disclosed SD1.5 cohort.Cross-family and trigger-disclosure results use their respective benchmark subsets.Source: §B.1 · pp. 17–18

B.2 Local LoRA Training

• Locally trained identity adapters use LoRA rank 128 on both the denoising network and text encoder.SDXL adapts both CLIP-L and OpenCLIP-G text encoders.Source: §B.2 · p. 18
• Training uses AdamW with β = (0.9,0.999), εAdam = 10−6, weight decay 0.01, maximum gradient norm 1.0, and the standard diffusion noise-prediction objective.Source: §B.2 · p. 18
• SD1.5 adapters are trained for 3,000 optimization steps, whereas SDXL adapters are trained for 1,000 steps.Source: §B.2 · p. 18

B.3 Training Data for Locally Trained LoRAs

• Real-identity adapters use 19–46 web-collected photographs per public figure, while synthetic identities use 25–50 generated images based on fixed identity specifications.Synthetic batches keep identity-defining attributes fixed while varying image-specific factors.Source: §B.3 · p. 18
• Both identity sources share image-specific captioning, and matched faithful/unfaithful pairs hold images and caption content fixed while replacing only the human-readable trigger with an opaque alphanumeric trigger.Captions describe visible factors that vary across examples.Source: §B.3 · p. 18

B.4 Baseline Configurations

• Six prior methods are adapted to the LoRA setting, using official implementations where available and an independent reproduction for PAIA.The evaluation uses methods from concept-agnostic difference discovery and concept-specified verification.Source: §B.4 · pp. 19–20
• Concept-agnostic baselines expose LoRA–base differences through token drift, prompt search, fixed-set descriptions, or discrete prompt optimization.DriftScope, CompCon, VisDiff, and DPO-Diff-EPO supply candidate prompts using these approaches.Source: §B.4 · p. 20
• PEZ recovers prompts from supplied concept images, whereas PAIA directly verifies whether a prespecified concept is encoded rather than discovering an unknown concept.PEZ does not access the adapter, compatible base model, or training trigger; PAIA uses candidate and irrelevant concept images.Source: §B.4 · p. 20

C.1 Retained Subspace Dimension

• Two retained directions perform best among tested k values, while the mean varies by at most 0.014; effects of k differ across real identities, synthetic identities, and anime characters.For k = 1, real-identity performance is 0.080 versus 0.078 at k = 2; k = 4 primarily hurts real identities and anime.Source: Table 11 · p. 20, §C.1 · p. 20

D.1 Threat Model and Controls

• The controlled attack changes only the regularizer while keeping the matched clean adapter’s training data, trigger, model, modules, and training configuration fixed.The attacker is assumed to preserve the adapter’s ability to generate its dominant hidden concept.Source: §D.1 · p. 20

D.2 Subspace-Dispersion Objective

• The subspace-dispersion objective minimizes the module-wise mean spectral concentration, for which flatter spectra have smaller values.The regularizer is computed without forming the full update matrix and preserves gradients.Source: §D.2 · p. 21

D.3 Fixed Training Schedule

• The attack strength is fixed: its coefficient is zero before step 300, rises to 0.1 by step 600, and remains 0.1 thereafter.The warm-up avoids evaluating the concentration ratio when the LoRA update is near zero; the coefficient is not tuned after training.Source: §D.3 · p. 21

D.4 Evaluation Protocol

• The fixed human-readable trigger and carrier prompt are rendered with the attacked LoRA-enabled model and evaluated against the same held-out target-concept references.The trigger is set before attack training and unavailable to the moderator; training uses neither the references nor discovery results.Source: §D.4 · p. 21