PaperlayerView exact revision on arXiv

Source-linked AI summary

When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges

Parth Darshan, Abhishek Divekar

arXiv:2605.26046v2cs.CLcs.AIcs.LGcs.MAcs.SE

TL;DR

Multi-objective prompt optimization for LLM judges is understudied, and numerical gradient conflict-resolution methods do not directly apply to textual feedback. The paper extends TextGrad with four decomposition modes and finds gradient specificity drops 59%, while combining independently optimized instructions lowers Spearman ρ from 0.305 to 0.220.

  • Problem

    Whether single-objective textual-gradient prompt optimization extends to judges scoring multiple evaluation dimensions remains understudied, and numerical conflict-resolution tools do not directly apply.

  • Method

    The paper extends TextGrad with four decomposition modes that vary whether loss, gradient, and optimizer stages process objectives separately or jointly.

  • Results

    Across configurations, gradient specificity drops 59% from 9.0 to 3.7 when the gradient LLM processes all tasks jointly, while combined instructions reduce Spearman ρ from 0.305 to 0.220.

  • Takeaways & Limitations

    The study identifies optimization-time gradient dilution and inference-time instruction interference as distinct failure points in multi-criteria textual-gradient judge optimization.

  • Takeaways & Limitations

    Experiments use N = 3 runs on SUMMEVAL, and LLM-scored diagnostics may introduce a confound.

Abstract

from arXiv · show

Customizing an LLM judge to a specific problem or domain often involves optimizing its prompt across multiple evaluation criteria simultaneously. Textual gradient methods automate this for a single judge criterion, however they produce natural-language critiques, not numerical vectors. Thus, the conflict-resolution toolkit of multi-task learning (PCGrad, MGDA) does not apply to this multi-objective textual gradient setting. We extend TextGrad to the multi-objective setting and test four decomposition modes of textual gradient optimizers by varying how much cross-objective information the loss, gradient and optimizer LLMs share. We find the gradient's task-focus drops by 59% (9.0 to 3.7 out of 10) when the gradient LLM must provide feedback on multiple criteria jointly. Separately, we observe that naively combining single-objective optimized instructions into a single prompt degrades Spearman rho from 0.305 to 0.220 (-0.085). These results identify two separable failure modes: optimization-time gradient dilution and inference-time instruction interference, which together constrain the design space for multi-objective judge optimization using textual feedback.

1 Introduction

The paper studies multi-objective textual prompt optimization for LLM judges, where numerical multi-task conflict-resolution methods do not directly apply to natural-language gradients. It tests four decomposition modes and diagnoses optimization-time gradient dilution alongside inference-time instruction interference.

  • Motivation: Existing prompt optimizers improve LLM-judge prompts for single objectives, but their extension to judges scoring several dimensions simultaneously remains less studied.The motivating setting includes judges evaluating multiple quality dimensions at once.
  • Motivation: Numerical multi-task conflict-resolution methods such as PCGrad and MGDA do not directly apply because textual gradients are natural-language strings without vector-space structure.This structural mismatch prevents direct use of projection- or constrained-optimization-based conflict resolution.
  • Approach: The study tests four decomposition modes—SSS, SSC, SCC, and CCC—based on whether the loss, gradient, and optimizer stages process tasks separately or jointly.It also introduces gradient specificity and feedback adherence as process-level diagnostics.
  • Findings: 59%: gradient specificity falls from 9.0 to 3.7 when the gradient LLM processes all tasks jointly, while feedback adherence remains high from 7.8 to 8.8.This localizes the optimization problem to gradient quality rather than optimizer compliance.
  • Findings: In 6 of 10 Qwen3 configurations, optimization never exceeds the initial generic prompt; only SINGLE-TASK with val=mae improves, by +0.031 Spearman at step 2.The study separately uses an oracle-instruction experiment to isolate inference-time instruction interference from optimization-time dilution.

2 Related Work

Prior prompt-optimization methods use scalar signals or textual gradients, while multi-objective approaches remain nascent and largely population-based. Existing judge-optimization methods do not jointly optimize across evaluation criteria while preserving per-task gradient observability.

  • Textual Gradient Methods: Scalar prompt optimizers such as OPRO and APE preceded textual-gradient methods such as PROTEGI and TextGrad.OPRO conditions rewrites on prompt-score histories, APE uses demonstrations and selects the highest-scoring variant, and PROTEGI replaces scalar signals with natural-language critiques.
  • Textual Gradient Methods: Subsequent studies found that gradient decomposition and reflection do not consistently improve prompt optimization or fully explain APO success.GPO reported that adding reflection hurts performance, while Melcer et al. found that the chain-rule gradient decomposition does not consistently help and that the gradient metaphor is inaccurate.
  • LLM-as-a-Judge Evaluation: No prior method jointly optimizes a judge prompt across multiple evaluation criteria while preserving per-task gradient observability.The setting studied traces how each criterion’s feedback shapes each rewrite.
  • Multi-Objective Prompt Optimization: Multi-objective prompt optimization is nascent, with population-based methods maintaining Pareto fronts and decomposing objectives into scalarized subproblems.MOPO applies NSGA-II with LLM-based mutation, while ParetoPrompt decomposes objectives into scalarized subproblems.
  • LLM-as-a-Judge Evaluation: LLM-judge research improves fixed prompts through rubric tuning, reinforcement learning, debiasing, and textual-gradient methods, but existing methods remain single-objective or use other decompositions.MPO applies section-local textual gradients to prompt components, and MAPGD coordinates multiple gradient agents using semantic similarity to resolve conflicting edits.

3 Method: Multi-objective TextGrad

The method extends TextGrad to multi-objective prompt optimization by varying whether loss, gradient, and optimizer stages process objectives separately or jointly. It evaluates four decomposition modes, a single-task baseline, and validation-gated versus ungated optimization trajectories.

  • TextGrad pipeline: TextGrad uses task, loss, gradient, and optimizer LLMs in a four-stage loop that predicts scores, critiques errors, aggregates textual gradients, and updates prompts.The loss LLM compares predictions with ground-truth annotations, while the gradient LLM converts per-example losses into structured instruction-level feedback.
  • Decomposition modes: The three-letter decomposition code specifies whether the loss, gradient, and optimizer stages operate separately per task or jointly across all tasks.This stagewise parameterization defines how much cross-objective information each optimization component receives.
  • Decomposition modes: SSS is fully separate, SSC combines only optimizer inputs, SCC combines gradient inputs, and CCC combines all stages; SINGLE-TASK independently optimizes each objective.In SCC, the gradient LLM receives critiques from all four tasks and produces one unified set of edits, whereas SINGLE-TASK uses separate objective-specific runs and forward passes.
  • Decomposition modes: The architectural boundary between SSC and SCC is whether the gradient LLM sees one task at a time or reconciles feedback from all four tasks.This boundary isolates the change in cross-objective information entering the gradient stage.
  • Validation Gating: Under val=mae, candidates are accepted only when held-out mean absolute error does not increase; without gating, every candidate is accepted.The gated strategy prevents prompt regression, while the ungated strategy exposes the optimization trajectory without filtering.

4 Experimental Setup

The experiments use SUMMEVAL to optimize a judge prompt across four summarization-quality dimensions, evaluating performance with task-averaged Spearman correlation. Main results use Qwen3 models with distinct temperatures for optimization, critique, and task scoring.

  • Datasets: 160 training pairs and 480 held-out pairs are sampled from SUMMEVAL, which contains news articles, system summaries, and expert annotations.Each pair receives human scores from multiple annotators.
  • Datasets: Four dimensions—fluency, relevance, coherence, and consistency—form the simultaneous multi-task optimization objectives.The judge prompt must produce accurate scores across all four dimensions.
  • Metric: Spearman rank correlation (ρ) between predicted and human scores is the primary metric, with results reported as the arithmetic mean across four tasks.This task-averaged ρ follows prior work on LLM-based evaluation.
  • Models: Qwen3-8B serves as the task LLM, while Qwen3-235B-A22B serves as the other LLMs in the main experiments.Optimizer temperature is T = 0.7; loss and gradient temperatures are T = 0.3; task-LLM temperature is T = 1.0.

5 Results

Results show that multi-objective prompt optimization often fails to improve judge performance, with degradation driven by cross-task coupling and mitigated—but not solved—by validation gating. The optimizer can still expand the Pareto front, while single-objective optimization remains most effective across model settings.

  • Aggregate performance: In 6 of 10 configurations, the initial generic prompt remains best because optimization fails to improve or degrades performance.The only improving multi-task modes are CCC with MAE validation and SSC without validation, both showing modest gains over 12 steps.
  • Optimization trajectories: SSC drops from 0.283 to 0.184 by step 7, illustrating degradation without validation gating.The optimizer improves training-batch loss while harming held-out generalization; MAE validation rejects some spurious updates.
  • Effect of task coupling: Without validation gating, performance degrades roughly as Single-Task > SSS, SSC, SCC, CCC, consistent with stronger cross-task gradient dilution.Under MAE validation, the pattern is non-monotonic because CCC slightly outperforms SSC, suggesting occasional complementary gradients.
  • Pareto-front behavior: Despite stagnant Spearman performance, CCC’s HVI increases continuously as the optimizer discovers diverse specialist prompts expanding the Pareto front.No single prompt necessarily dominates the initialization across all four tasks.
  • Model robustness: With DeepSeek v4 under MAE validation, optimization yields better absolute Spearman improvements, but single-objective optimization remains most effective.This reproduces the broader trend across model settings.

6 Analysis

The analysis identifies two separable failure modes in multi-objective textual-gradient prompt optimization: optimization-time gradient dilution and inference-time instruction interference. Gradient quality, rather than optimizer compliance, limits optimization, while combining individually effective instructions can still reduce judge performance.

  • Optimization-time gradient dilution: 59%: Gradient specificity falls from 9.0 (±0.3) in per-task modes to 3.7 (±0.5) in cross-task modes, with zero group overlap.Per-task modes process one task per gradient call; cross-task modes process all four tasks jointly.
  • Optimization-time gradient dilution: Gradient dilution extends prior within-criterion findings to cross-criterion optimization, with consistency most diluted while coherence retains moderate focus.The per-task breakdown indicates that combining multiple task gradients degrades the per-task optimization signal.
  • Optimization-time gradient dilution: 7.8 to 8.8: Feedback adherence remains uniformly high across modes and validation settings, ruling out optimizer non-compliance as the failure explanation.The optimizer faithfully incorporates received gradients, including generic rather than criterion-specific suggestions.
  • Inference-time instruction interference: 0.220: Oracle-combined instructions achieve average Spearman ρ of 0.220, down 0.085 from 0.305 for single-task optimized instructions.The degradation occurs despite selecting each task’s best single-task instruction.
  • Inference-time instruction interference: Instruction-length asymmetry appears to drive interference: verbose task rubrics receive disproportionate inference-time attention when packed with brief instructions.The fluency instruction expands to ∼800 tokens, whereas the relevance instruction remains at ∼4 tokens.

7 Conclusion

The conclusion identifies two systematic failure points at distinct stages of multi-criteria textual gradient optimization for LLM judges. It argues that reliable multi-objective customization requires architectural changes and diagnostics for evaluating possible remedies.

  • Failure modes: The study exposes two distinct pipeline failures: gradient dilution during optimization and instruction interference during inference.These failures affect different decomposition modes and cannot be treated as a single problem.
  • Failure modes: 59% drop in task-specificity occurs when the gradient LLM reconciles feedback from multiple criteria in one call.The loss of task-specificity is identified as an optimization-time failure.
  • Implications: Reliable multi-objective judge customization requires architectural changes, because addressing either failure mode alone is insufficient.Separate judge calls per criterion eliminate interference but multiply inference costs.
  • Implications: Gradient specificity and feedback adherence provide diagnostics for evaluating approaches to either failure mode.Conflict-aware gradient resolution is suggested as a possible way to address dilution if textual gradients can be meaningfully embedded and projected.

8 Future Work

Future work proposes statistically reliable diagnostics, synthetic aligned criteria, and multi-objective critics for agentic workflows. It also outlines mitigations for gradient dilution and instruction interference.

  • Statistically reliable LLM diagnostics with PPI: Prediction-Powered Inference could combine small human-judged and large LLM-judged sets to produce provably unbiased diagnostic estimates.The hierarchical PPI extension is applicable because annotations are per-gradient while the quantity of interest is the per-mode mean.
  • Synthetic task generation for aligned criteria: Synthetic task generation could create complementary optimization objectives and align them by construction.The proposed direction synthesizes criteria rather than treating them as fixed.
  • Multi-objective critics for agentic workflows: Multi-objective judge prompts could act as critics for agentic systems that track several quality dimensions of tool-use trajectories.Applying the optimization pipeline raises whether gradient dilution and instruction interference persist when criteria are tool-grounded and partially verifiable.
  • Mitigations: A specificity-aware router could fall back to per-task gradient LLMs when multi-task specificity drops below a threshold.This mitigation aims to capture CCC’s hypervolume without losing task-focus.
  • Mitigations: Two proposed mitigations for instruction interference are next-token attention masking and length-aware instruction synthesis.Masking targets per-criterion output generation, while length-aware synthesis normalizes rubric length so no criterion dominates the attention budget.

Limitations

The study’s limitations include a SUMMEVAL-only evaluation scope, limited statistical power from N = 3 runs, and LLM-based scoring that may confound gradient-specificity and feedback-adherence measures. Other textual gradient paradigms may behave differently, although the diagnostics are algorithm-agnostic.

  • Evaluation scope: The experiments are scoped to SUMMEVAL, whose expert annotations cover four clearly separable criteria.Other benchmarks are needed to validate whether the identified failure modes generalize.
  • Future validation: BRIGHTER, ASAP++, EM-SCAD, and GitBugs are proposed to test dilution scaling, cross-language transfer, gradient-specificity transfer, and heterogeneous classification criteria.These benchmarks extend evaluation beyond summarization, ordinal quality scales, and the current criterion setup.
  • Methodological scope: Other prompt optimization paradigms may exhibit different multi-task dynamics, although the diagnostics can be applied to any textual gradient approach.The limitation concerns transferability across optimization paradigms rather than the diagnostics’ stated applicability.
  • Statistical and measurement limitations: N = 3 runs limits statistical power, so claims are restricted to robust effects such as the 59% specificity drop and −0.085 cherry-pick degradation.Gradient specificity and feedback adherence are scored by an LLM evaluator, introducing a potential confound.

Ethics Statement

Optimized judge prompts can inherit biases from underlying LLMs and human annotations, so practitioners should audit them before sensitive deployment; the authors will release code and diagnostics openly for reproducibility.

  • Ethics Statement: Optimized judge prompts inherit biases from underlying LLMs and human annotations, requiring audits before deployment in sensitive contexts.Examples include content moderation and hiring.
  • Ethics Statement: The authors will release code and diagnostics under an open-source license to support reproducibility.

A Per-Task Gradient Specificity … F Diagnostic Evaluation Prompts

The paper finds that multi-objective prompt optimization suffers from task-specificity dilution in joint textual gradients and interference when combining optimized instructions. Cross-model and DeepSeek experiments preserve these patterns, while diagnostics and prompt-template details define how they are measured and implemented.

  • A Per-Task Gradient Specificity: Consistency is the most diluted dimension in combined-gradient modes, with specificity 2.5, while coherence retains moderate focus at 5.0.These averages cover SCC and CCC, indicating that simultaneous processing distributes gradient attention unevenly across tasks.
  • B Cherry-Pick Experiment: All Variants: All six cherry-pick variants degrade below the initial generic baseline, showing robust inference-time instruction interference across selection metrics and validation settings.The variants comprise three selection metrics and two validation settings.
  • C Gradient Specificity Under Gradient-Model Swap: 52% reduction: swapping in DeepSeek-V4-Pro leaves SSC at 8.82±0.10 but lowers SCC to 4.22±0.26.The task, loss, and optimizer LLMs remain in the Qwen3 family, so dilution persists under a cross-family gradient-model swap.
  • D DeepSeek v4 Results: DeepSeek v4 experiments use DeepSeek-v4-Flash as the task LLM and DeepSeek-V4-Pro for loss, gradient, and optimizer LLMs, with MAE validation and N =3 seeds.The setup provides a stronger starting point for task-following and optimization.
  • D.1 Overall Performance: Best avg ρ reaches 0.409–0.429 from multi-objective initialization near ρ ≈0.38, with SSC achieving the best avg ρ and highest HVI.Despite stronger absolute performance, single-objective optimization still produces the largest improvement from initialization.
  • D.2 Gradient Specificity: 52% reduction: DeepSeek per-task gradient modes average 8.78 specificity, whereas joint-gradient modes average 4.25.The joint modes are SCC and CCC; the per-task modes are Single, SSS, and SSC.
  • E Multi-Objective Judge Prompt Template: The judge prompt keeps a frozen evaluation skeleton and output format while optimizing only mutable per-task instructions, which expand into criterion-specific rubrics.The skeleton remains fixed through all 12 optimization steps and is specific to SUMMEVAL.
  • F Diagnostic Evaluation Prompts: Posthoc diagnostics use Claude Sonnet 4.6 to score gradient specificity and feedback adherence on 1–10 scales for each target task.Specificity measures focus versus generic or cross-task advice; adherence measures whether instruction edits address task-relevant gradient suggestions.
Loading 2605.26046v2…