Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

• Text-only prompt optimization leaves MLLMs underutilized by overlooking richer contextual signals available through multimodal inputs.A single image can convey characteristics more directly than long or ambiguous textual descriptions.Source: §1 · p. 1
• The paper defines multimodal prompt optimization as expanding prompt optimization beyond text to multiple modalities.A multimodal prompt is a pair of discrete textual and non-textual prompts.Source: §1 · p. 1
• The enlarged multimodal space creates challenges in maintaining cross-modal consistency and efficiently selecting sparse high-quality candidates.Optimization must account for both modality-specific effectiveness and alignment across modalities.Source: §1 · p. 2
• MPO jointly updates textual and non-textual prompts through alignment-preserving feedback and diversifies exploration with generation, editing, and mixing operations.Its prior-inherited Bayesian-UCB selection uses parent performance as a prior to favor promising regions of the multimodal space.Source: §1 · p. 2
• Across 10 datasets spanning images, videos, and molecules, MPO consistently improves over leading text-only optimization methods.The prior-inherited Bayesian-UCB strategy reduces evaluation budget by 42% compared with a prior-free baseline.Source: §1 · p. 2

• MLLMs extend LLMs to diverse non-textual modalities and support tasks ranging from classification and captioning to medical and pharmacological applications.Source: §2 · pp. 2–3
• MPO’s overview combines alignment-preserving exploration with prior-inherited Bayesian-UCB selection for multimodal prompts.Source: Figure 2 · p. 3
• Automatic prompt optimization includes gradient-based soft prompts and gradient-free generation, evaluation, and refinement of textual candidates.Prior work also uses failure analysis and evolutionary operations, but remains limited to text-only settings.Source: §2 · p. 3
• MPO differs from instance-specific prompting by discovering a single reusable prompt for performance across an entire task rather than improving one query at a time.Source: §2 · p. 3

3.1 PROBLEM DEFINITION

• A multimodal prompt is p = (t, m) ∈ T × M, and optimization seeks the pair (t*, m*) maximizing a task metric over query–answer data.The evaluation function may use metrics such as accuracy or F1.Source: §3.1 · p. 4
• The joint search space raises two core problems: preserving semantic consistency across modalities and selecting promising prompts when high-quality candidates are sparse.Source: §3.1 · p. 4

3.2 ALIGNMENT-PRESERVING EXPLORATION OF MULTIMODAL PROMPT SPACE

• MPO uses cohesive backpropagation to derive one textual feedback signal from the failure set, guiding both textual and non-textual updates.This shared signal reduces the risk of overfitting updates to one modality.Source: §3.2 · p. 4
• The joint multimodal update refines the text prompt and generates modality-specific conditions for generators that revise the non-textual prompt consistently.Source: §3.2 · p. 4
• Generation, edit, and mix operators respectively expand, refine, and recombine non-textual prompts to explore diverse multimodal regions.Generation creates from scratch, editing preserves useful structures for targeted refinement, and mixing synthesizes complementary candidates.Source: §3.2 · p. 5

3.3 EFFECTIVE PROMPT SELECTION BY PRIOR-INHERITED BAYESIAN UCB

• Parent prompts provide informative priors because their scores correlate positively with child scores, with Pearson’s r = 0.88.Figure 3 reports the parent–child score relationship used to motivate warm-started selection.Source: §3.3 · p. 5, Figure 3 · p. 5
• Prior-inherited Bayesian UCB initializes each child’s Beta score distribution from its parent posterior, then selects candidates by upper-confidence scores under a fixed evaluation budget.The mechanism supplies S pseudo-observations and updates posteriors after evaluating small data batches.Source: §3.3 · pp. 5–6
• Under an informative prior, the proposed Bayesian-UCB strategy has nonincreasing best-arm identification cost and reduces wasted evaluations on low-potential candidates.Source: §3.3 · p. 6

4.2 EXPERIMENTAL RESULTS AND ANALYSES

• MPO consistently outperforms manual, few-shot, and text-only optimization baselines across image, video, and molecular benchmarks.The evaluation covers classification, question answering, video tasks, and molecular property prediction, with results averaged over three independent runs.Source: §4.1 · p. 6, §4.2 · p. 7, Table 1 · p. 7
• MPO remains effective when varying base models, optimizer models, and modality-specific generators, including larger and lightweight open-source backbones.The reported comparisons include Qwen2.5-VL (72B) and SANA1.5 (1.6B), while MPO continues to outperform text-only methods.Source: Table 2 · p. 7, §4.2 · p. 8
• MPO’s alignment-preserving joint refinement achieves the highest cross-modal alignment and largest performance gains among the evaluated prompt variants.Sequential optimization and random image prompts follow MPO, while in-distribution and out-of-distribution image queries lag significantly behind.Source: Figure 4 · p. 7, §4.2 · p. 8
• The full multimodal prompt substantially outperforms single-modality variants, indicating that textual and non-textual components provide mutually reinforcing signals.Each modality alone surpasses the Human baseline, but the complete prompt achieves higher performance than either isolated component.Source: Table 3 · p. 8, §4.2 · p. 9
• Combining generation, edit, and mix operators produces the best performance, with each operator contributing a distinct form of multimodal exploration.Generation introduces visual compositions, edit refines local features, and mix recombines broader attributes; similar behavior is observed for molecular prompts.Source: Table 4 · p. 8, Figure 6 · p. 8, §4.2 · p. 9
• Prior-inherited Bayesian-UCB selection matches Uniform performance using 30% of its evaluation budget, reducing resource cost by 70%.MPO also reaches the performance levels of UCB and the no-prior variant with 52% and 42% less budget, respectively.Source: Figure 5 · p. 8, §4.2 · p. 9
• On CUB, MPO continues improving after ProTeGi plateaus, gaining an additional 6.4 points beyond the third iteration versus ProTeGi’s marginal 1.1-point gain.The comparison suggests that multimodal optimization continues navigating the search space after text-only optimization has reached a performance ceiling.Source: §4.2 · p. 9, Figure 7 · p. 9
• Prior strength has an intermediate optimum: weak priors provide insufficient guidance, whereas overly strong priors limit adaptation to child-prompt observations.The study also visualizes hidden states, finding that full multimodal prompts shift representations into a distinct region relative to text-only methods.Source: §4.2 · pp. 9–10, Figure 9 · p. 9

• MPO consistently surpasses leading text-only prompt optimization methods across diverse real-world multimodal problems.The evaluation covers images, videos, and molecules.Source: §5 · p. 10

• The experiments involve no human subjects, personally identifiable data, or sensitive information.All experiments use public datasets and models under research-permissive licenses.Source: ETHICS STATEMENT · p. 10

A.1 DETAILS ON DATASETS

• The benchmark suite covers image, video, and molecular classification or question-answering tasks using official or constructed train/test splits.Image and video tasks generally use 300 test examples, while molecule tasks use the entire test set.Source: §A.1 · p. 15
• PlantVillage is adapted into fine-grained plant-disease subtasks across four crop species with a 50/50 train/test split.The source dataset contains 54,306 images, 38 disease categories, and 14 crop species.Source: §A.1 · p. 15
• CUB-200-2011 is organized into 12 fine-grained bird-classification subtasks grouped by shared family names and split 50/50 with at least 80 instances per partition.The grouping targets visually similar species and balanced task difficulty.Source: §A.1 · p. 15
• SLAKE, DrivingVQA, and RSVQA evaluate visual question answering across radiology, driving, remote sensing, geospatial reasoning, and object counting.DrivingVQA is filtered to single-answer questions, while SLAKE is partitioned by CT, MRI, and X-Ray modalities.Source: §A.1 · p. 15
• Video benchmarks include driver-behavior classification and video anomaly detection using Drive&Act and VANE-Bench.Drive&Act follows its official split, while VANE-Bench uses a 60/40 train/test split.Source: §A.1 · p. 15

A.2 DETAILS ON BASELINES

• The baselines include manual prompts and text-only optimizers based on reverse engineering, score-guided optimization, evolutionary search, meta-prompt tuning, textual gradients, and cohesive prompt-example optimization.The listed methods include APE, OPRO, EvoPrompt, PE2, ProTeGi, and SEE.Source: §A.2 · p. 16

A.3 ADDITIONAL IMPLEMENTATION DETAILS

• MPO uses temperature 0 for the base model and 0.7 for the optimizer, with a failure set of 3 and an evaluation budget generally set to 100.Special handling is applied for small CUB subtasks, videos, molecules, objectives, and task-specific answer evaluation.Source: §A.3 · p. 16

A.4 META PROMPTS TO IMPLEMENT MPO

• The meta-prompt appendix supplies prompts for cohesive backpropagation, generation, editing, and mixing operators.These components correspond to Figures 10–13.Source: Figure 10 · p. 17, Figure 11 · p. 17, Figure 12 · p. 18

A.5 FULL ALGORITHM OF MPO

• Algorithm 1 initializes prompts, identifies failures, obtains feedback, generates multimodal candidates, and iteratively selects beam prompts over T optimization iterations.Joint updates use generation, edit, or mix operations for textual and non-textual components.Source: §A.5 · p. 19
• Algorithm 2 initializes child-prompt Beta priors from parent performance and allocates evaluation batches using the highest Bayesian UCB score.Posterior parameters are updated after each batch, and the top prompts by posterior mean are returned.Source: §A.5 · p. 19

B THEORETICAL ANALYSIS ON PRIOR-INHERITED BAYESIAN UCB

• The inherited Beta prior initializes each child with α0,i = ˆµpar(i) S + 1 and β0,i = (1 − ˆµpar(i)) S + 1, while the uniform baseline uses Beta(1, 1).Posterior parameters then incorporate observed successes and pulls.Source: B THEORETICAL ANALYSIS ON… · p. 20
• Prior inheritance is analyzed under the assumption that parent estimates are, on average, KL-closer to child-arm means than the uninformative prior.The assumption is empirically supported by a strong positive correlation between parent and child scores.Source: B THEORETICAL ANALYSIS ON… · p. 20

B.1 TWO AUXILIARY LEMMAS

• The prior strength S acts as an additive effective sample size, shrinking credible-interval widths as if S additional observations had been collected.The resulting width scales with 1/√(Ni(t) + S), up to the stated logarithmic factor.Source: §B.1 · p. 20
• Under the KL-closeness assumption, inherited posteriors are better centered around the true child-arm means on average than posteriors from uninformative priors.The corresponding quantiles shift favorably in expectation, raising the optimal arm’s lower quantile and lowering suboptimal arms’ upper quantiles.Source: §B.1 · pp. 20–21

B.2 SUFFICIENT CONDITION FOR OPTIMAL ARM IDENTIFICATION

• Optimal-arm identification is guaranteed when the optimal arm’s lower quantile exceeds every suboptimal arm’s upper quantile, with credible widths sufficiently small relative to the suboptimality gaps.Because inherited priors improve quantile estimates, this separation condition is achieved no later than with an uninformative prior.Source: §B.2 · p. 21

B.3 PROOF OF PROPOSITION 3.1

• Prior inheritance improves Bayesian UCB through two mechanisms: tighter credible intervals at fixed counts and more efficient allocation of pulls over time.The prior strength behaves as an additive effective sample size, while favorable quantile shifts guide sampling toward the optimal arm.Source: §B.3 · p. 21
• The tighter intervals and improved allocation make the separation condition easier and faster to satisfy, reducing expected pulls on suboptimal arms for sufficiently large budgets.The expected identification cost is non-increasing and strictly decreases when the average KL-closeness assumption holds.Source: §B.3 · p. 21

C.1 COMPARISON OF COMPUTATIONAL COSTS

• MPO incurs manageable additional computation for modality-specific generation while achieving a substantial performance improvement over text-only prompt optimization methods.The generator can use lightweight open-source models such as SANA1.5 (1.6B).Source: §C.1 · p. 22, Table 5 · p. 22

C.2 FULL MAIN RESULTS

• Tables 6 and 7 report full experimental results for image and molecule modality benchmarks, respectively, with scores averaged over three independent experiments.The tables include individual subtasks.Source: §C.2 · p. 22, Table 6 · p. 22, Table 7 · p. 22

C.3 QUALITATIVE RESULTS

• MPO’s multimodal prompts are illustrated for image, video, and molecular domains through qualitative examples and molecular optimization visualizations.The molecular analysis shows the optimization process, while additional tables present optimized prompt examples.Source: §C.3 · p. 23, Figure 14 · p. 23, Table 9 · p. 25
• Compared with SEE, MPO pairs concise textual instructions with direct visual references, producing more specific and visually grounded reasoning.SEE often relies on verbose, generic descriptions that can lead to coarse reasoning and misclassification.Source: §C.3 · p. 23, Table 13 · p. 28, Table 14 · p. 28
• 66.6% of SEE’s misclassifications on the Auklet subtask are fixed by MPO’s multimodal approach.The reported examples attribute the correction to concrete visual cues that text-only prompts fail to encode precisely.Source: §C.3 · p. 23, Table 15 · p. 29, Table 16 · p. 30

D USE OF LARGE LANGUAGE MODELS (LLMS)

• The appendix defines generation, edit, and mix conditions for modality-specific prompt operations, including creating, enhancing, and combining labeled grosbeak reference images.These conditions emphasize comparable poses, distinctive visual features, clear labels, and reduced background clutter.Source: D USE OF LARGE LANGUAGE MODELS (LLMS) · p. 24