HimmBO: High-Dimensional Interactive Model-Merging Bayesian Optimization for Image Generation

1. Introduction

• Existing preference-based optimization handles only roughly 20–30 preselected adapters and requires costly image generation each round, limiting exploration of large collections.Adapter collections exceed 100K, while operating many coefficients is difficult even with a handful of sliders.Source: §1 · p. 2
• HimmBO screens for the active adapters using level-set estimation over a fixed image pool, then refines merge coefficients in the recovered lower-dimensional space.Using pre-generated gallery images eliminates image-generation wait time during screening.Source: §1 · p. 2
• A style kernel lets preference information transfer between adapters that produce visually similar results.This addresses the fact that adapter identity does not itself encode visual similarity.Source: §1 · p. 2
• At N = 1080, HimmBO reaches 0.889 similarity, compared with 0.828 for the best alternative and 0.762 for GimmBO; it reaches 0.80 similarity with 6.8% of GimmBO’s image generations.Its similarity at N = 40 is 0.907, so the drop at N = 1080 is 0.018.Source: §1 · p. 2

2. Related Work

• Adapter merging combines weight updates, while prior image-generation methods generally use a small, fixed set with predetermined or target-fitted coefficients; GimmBO instead searches coefficients using preference comparisons.GimmBO still starts with a small candidate set chosen beforehand.Source: §2 · p. 3
• Retrieval methods select adapters from a query specified before search, which can be difficult when users cannot yet express their preferences explicitly.Stylus retrieves from a text query, while LoRAverse also promotes collection coverage.Source: §2 · p. 3, Figure 3 · p. 3
• Human-in-the-loop systems use sliders or galleries, and precomputed galleries can reuse expensive results during interaction; direct slider control becomes difficult beyond roughly five to ten parameters.Many BO methods instead propose new settings each round, requiring outputs to be generated during interaction.Source: §2 · p. 3
• Level-set estimation suits screening because it identifies a set of promising candidates rather than seeking only one optimum, while HimmBO models feedback in a fixed-dimensional style space.This contrasts with approaches that ease high-dimensional search but retain a coefficient-space dimension that grows with the adapter count.Source: §2 · p. 3

3. From a curated set to a library

• The standard linear merge adds scaled adapter weight updates to the base model, producing images that mix their visual effects.The paper uses this community-standard rule but says the method can also be applied to alternative merging rules.Source: §3 · p. 3
• GimmBO searches coefficients interactively but typically uses only 20–30 adapters chosen beforehand, leaving library-scale adapter selection unresolved.With public collections numbering in the thousands, users must decide which adapters to include before tuning coefficients.Source: §3 · pp. 3–4
• Preference may have several peaks, so focusing on one tentative optimum can miss styles needed for a final merge spanning multiple promising regions.A style the user never sees cannot be requested later in the search.Source: §3 · p. 4
• Coefficient-space distance treats all adapter pairs equally and prevents feedback from transferring between visually similar adapters; a style-space preference model addresses this mismatch.Independently trained adapters can produce nearly indistinguishable appearances.Source: §3 · p. 4, Figure 4 · p. 4

4. Method

• The method targets the unreliable active-set recovery that arises when a single tentative optimum can miss adapters from other promising regions.It searches the full library for an active set, then optimizes merges within that set using a shared posterior that carries screening feedback into refinement.Source: §4 · p. 4

4.1. Style representation

• A style map converts gallery images into fixed-size style features, which are measured directly during screening and predicted from coefficients for refinement acquisition.The gallery uses images generated from each adapter on the same four prompts.Source: §4.1 · p. 5, Figure 5 · p. 5

4.2. Preferential surrogate

• HimmBO represents a candidate using both its visual style and merge coefficients, letting preference feedback transfer between visually similar adapters while preserving distinctions between merges.With the style kernel, feedback cost to recover half the active adapters remains nearly flat and is about 130× lower at N=1080 than with a coefficient-only kernel.Source: §4.2 · p. 5

4.3.1. Precomputed gallery

• The precomputed gallery associates coefficient samples with generated images and style features, allowing interaction to show selected samples without generating images during screening.Source: §4.3.1 · p. 5

4.3.2. Screening

• LSE recovers active adapters more accurately than BO as the library grows, performs best among the compared screening objectives, and retains promising candidates across multiple peaks.Screening uses precomputed galleries, so it requires no image generation.Source: Figure 6 · p. 6, §4.3.2 · p. 6
• Screening selects uncertain candidates near a preference level, estimates the above-level set, and allocates refinement slots across its potentially multiple peaks.The level is re-estimated each round; candidates within a style-space radius are kept from appearing as near-duplicate queries.Source: §4.3.2 · p. 6

4.3.3. Refinement

• Refinement optimizes coefficients for the screened adapters using batch BO, with a style-diversity term to reduce visually redundant candidates.Its variable stick length lets the search explore merge totals up to cap B; the length is sampled from public-merge coefficient totals and becomes an optimization variable.Source: §4.3.3 · pp. 6–7

4.4. Implementation details

• Search computation adds negligible interaction delay: screening takes about 0.3 s per round and refinement about 1 s for posterior updating and acquisition optimization.Source: §4.4 · p. 7

5.1. Quantitative experiments

• The simulated-user matching study tested 24 targets over five seeds, using a 1,080-adapter SDXL library and nested subsets against BO, retrieval, and coordinate-descent baselines.The main performance analysis used N=360.Source: §5.1 · pp. 7–8

5.1.1. Simulated-user comparison

• 8 inferences brought HimmBO to 0.80 similarity, versus 53 for Stylus+GimmBO and 117 for GimmBO, while HimmBO finished with the highest similarity.Screening requires no image generation, clarifying HimmBO’s advantage on the inference axis.Source: Figure 7 · p. 8, §5.1.1 · p. 8

5.1.2. Library scale

• Across nested libraries, HimmBO maintained similar similarity and the strongest adapter recovery, though recovery declined at the two largest sizes.Similarity dropped less because larger libraries offered closer visual substitutes for missed components.Source: Figure 7 · p. 8, §5.1.2 · p. 9

5.1.3. Ablations

• At N=360, straddle overtook UCB and ALM before the 12-round screening budget ended; its advantage grew with library size.At N=40, all three acquisition rules performed similarly.Source: §5.1.3 · p. 9, Figure 8 · p. 9
• With screening fixed, fixed-stick length caused the largest similarity loss; removing the style kernel or screening feedback also reduced similarity, while DPP mainly prevented visually redundant proposals.All refinement variants started from the same active set.Source: §5.1.3 · p. 9, Figure 8 · p. 9

5.2. Extensions

• When a ground-truth component was absent, HimmBO substituted a visually similar adapter, reaching rough matches above 0.9 DreamSim when the substitute was sufficiently similar.Larger libraries bring more distractors but also make closer substitutes more likely.Source: §5.2 · p. 9, Figure 9 · p. 10
• For personalization, reference images initialized the preference model and informed early screening, while the full adapter library remained available for exploration.Users provided three screening-feedback rounds from references, followed by nine normal screening rounds.Source: §5.2 · p. 10
• Adding user-vetted refinement merges to the screening pool increased mean final similarity from 0.923 to 0.944 in a preliminary experiment.These known merges can be added without training another adapter or generating another image.Source: A gallery that grows with use · p. 10

• The evaluation tests linear merging only; alternative merging rules would affect refinement, while screening remains independent because its gallery contains single-adapter images.Source: §6 · p. 10
• The simulated user gives noise-free, internally consistent feedback, so a larger user study is needed to assess whether real-user inconsistency or preference drift harms screening or refinement.Source: §6 · p. 10
• The method uses only part of standard level-set estimation because comparison-based utilities and per-round posterior refits make intervals across rounds incomparable.Source: §6 · pp. 10–11
• Reference images can steer initial screening toward preferred styles without restricting later exploration, and completed sessions can add preferred merges to the screening pool without further image generation.Source: Figure 10 · p. 11, Figure 11 · p. 11
• The style map suppresses but does not eliminate shared image content, so references with different content may retrieve wrong style neighbors; more explicit style-content separation could improve reliability across subjects.Source: §6 · p. 11
• The one-way transition means omitted adapters cannot return during refinement, while retaining delivered merges makes the gallery grow indefinitely; revisiting the library and managing retention are proposed remedies.Source: §6 · p. 11
• Unlike Midjourney’s proprietary style-space position, HimmBO returns the adapters and coefficients composing a result and supports an extensible library.Source: §6 · p. 11
• The authors flag risks including nonconsensual use of photographs and imitation of living artists, and plan to release a manifest and curation pipeline rather than adapter weights.Source: §6 · p. 11

A.1. Surrogate hyperparameters and priors

• Kernel hyperparameters are refit each round with log-normal priors centered on offline-calibrated values; shared channel length scales avoid per-dimension sparsity priors, and refinement uses less exploration than GimmBO.Without the priors, consistent comparisons can drive utility scales to extremes; fixed calibrated values also perform worse.Source: §A.1 · p. 15

A.2. Offline precomputation

• The offline gallery generates each adapter alone at unit strength on four fixed prompts, then maps images to style coordinates stored alongside the screening images.The four prompts cover a person, animal, still life, and landscape; building the gallery requires four inferences per adapter, paid once per library.Source: §A.2 · p. 15
• The style map centers features within each prompt, normalizes them, and applies PCA; retaining 80% of library variance gives d=90 at N=360.Centering emphasizes adapter contributions, while normalization makes distances reflect style rather than feature magnitude.Source: §A.2 · pp. 15–16

A.3. Screening

• Screening rebuilds above-level, below-level, and uncertain sets from posterior intervals each round, selecting queries from the uncertain set and allowing earlier classifications to change.The method omits standard LSE interval intersection because comparison-based posterior intervals are not comparable across rounds, and stops after a fixed interaction budget.Source: §A.3 · p. 16
• The active-set readout allocates m slots across promising style-space peaks rather than taking only the highest posterior-mean adapters; peak readout raises active-set F1 by about 0.04 over plain top-m.Figure S1 contrasts slot allocation, and the separation radius of 1.5ℓs gives the highest F1.Source: §A.3 · p. 16, Figure S1 · p. 16

A.4. Refinement

• Refinement predicts a proposed merge’s style as a coefficient-weighted sum for acquisition, then uses its measured style feature after generation.The prediction keeps acquisition differentiable; prediction errors affect proposed merges, not what the posterior learns from generated images.Source: §A.4 · p. 16
• Refinement varies total stick length to sample realistic community merge strengths while keeping other totals reachable, and enforces threshold τ inside the recursion so coefficients are zero or at least τ.Figure S2 contrasts this with post-hoc thresholding, which can violate the cap or leave small nonzero residuals.Source: §A.4 · p. 17, Figure S2 · p. 17

A.5. Image generation

• Images use SDXL 1.0 at 1024 × 1280; refinement merges the selected adapters’ weight updates, while SDEdit conditions session images on an anchor generated from the prompt.The anchor is noised to t0 = 0.8, leaving 24 of 30 denoising steps.Source: §A.5 · p. 17

• Screening needs no GPU and takes about 0.3 s per round; scoring scales linearly with library size, while the style-kernel matrix and median-distance calculation are computed once per library build.Source: §A.6 · p. 17
• 15 minutes per HimmBO session versus 37 minutes for GimmBO, with most of the difference coming from HimmBO’s 12 screening rounds requiring no image generation.Source: §A.6 · p. 17
• 4.4 s per image in HimmBO versus 5.7 s in GimmBO, because HimmBO refines over eight selected adapters kept in memory.Source: §A.6 · p. 17

B.1. The library

• The full SDXL-Style-1K library contains 1,080 adapters, and the five nested evaluation libraries range from 40 to 1,080 adapters.Curation begins with 9,909 style-tagged SDXL adapters and filters for style relevance, availability, and visual similarity.Source: §B.1 · p. 18

B.2. Targets and simulated user

• The 24 ground-truth targets cover four subject types, with half formed from hidden two-adapter merges and half from three-adapter merges.The simulated user converts screening selections and refinement rankings into pairwise comparisons.Source: §B.2 · p. 18

B.3. Baselines

• Baselines include published GimmBO, Stylus retrieval and Stylus+GimmBO, and random coordinate descent over adapter coefficients.Stylus retrieves from 150 adapters, while random coordinate descent sweeps each coefficient over eight values in [0,1].Source: §B.3 · p. 18

Appendix C: Choice of perceptual encoder

• Each encoder comparison replaces the encoder in both the style map and simulated user, then scores each final merge with all four encoders.The comparison therefore evaluates complete pipelines rather than isolating the style kernel.Source: Appendix C · pp. 18–19
• DreamSim leads under three of four evaluation encoders; TPIPS scores higher under TPIPS (0.923 versus 0.921).The table compares search encoders by scoring encoders.Source: Appendix C · p. 19, Table S6 · p. 19

Appendix D: How much feedback each kernel needs

• The kernel comparison measures rounds until at least half the target components are recovered, using pure posterior variance and eight gallery images per round.The curves differ only in using the full kernel or its coefficient channel; each round yields 16 comparisons.Source: Appendix D · p. 19

Appendix E: A gallery that grows with use

• The expanded gallery adds 54 related merges to six target sessions, with each added merge sharing a component but not matching a target’s component set.The merges come from 12 widely downloaded adapters with distinct styles.Source: Appendix E · p. 19
• Mean final similarity rises from 0.923 to 0.944, and active-set recovery improves for half of the six targets.The six targets were selected for rough matches that were not near-exact.Source: Appendix E · p. 19
• Popular community merges and merges from earlier sessions could densify frequently preferred regions without additional image generation, but confirming this at scale requires real usage logs.Source: Appendix E · p. 19

Appendix F: Additional qualitative results

• Figure S5 shows five open-ended sessions in which participants screen the library, write prompts, and rank merges by preference without a target.Source: Appendix F · p. 19, Figure S5 · p. 21
• Figure S4 displays eight library adapters generated alone at unit strength across four gallery prompts.Source: Appendix F · p. 19, Figure S4 · p. 20