PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment

Lingxiao Kong, Steffen Staab, Cong Yang, Oya Beyan, Zeyd Boukhers

arXiv:2608.29978v1cs.CLcs.LGcs.NE

TL;DR

LLM alignment requires controllable generation across competing objectives, user preferences, and prompts without retraining. Evolutionary Soups uses hidden-state-conditioned, per-layer MoE gating and evolutionary search with greedy hypervolume contribution. It reports consistent improvements over controllable baselines across three tasks and broader Pareto-front coverage.

  • Problem

    CMOG lacks a single model that can adapt to diverse user preferences and input prompts at inference time without retraining.

  • Method

    Evolutionary Soups evolves context-aware, per-layer gating networks that dynamically merge objective-specific experts using hidden-state representations.

  • Results

    Evolutionary Soups outperforms CMOG baselines in utility and Pareto optimality across multiple tasks and objectives.

  • Takeaways & Limitations

    The evolved Pareto front supports preference-based gating selection at inference time, enabling controllable generation without retraining.

  • Takeaways & Limitations

    The reachable reward region is bounded by the span of frozen experts, and ES does not close the gap to non-controllable MORLHF on Assistant.

Abstract

from arXiv · show

Large language models are increasingly required to generate responses that satisfy multiple competing objectives. Since optimal trade-offs depend on both user preferences and input prompts, controllable multi-objective generation must dynamically adapt models at inference time without retraining. To address this, we propose Evolutionary Soups, a mixture-of-experts framework for fine-grained generation control, with gating networks trained via an evolutionary algorithm. The per-layer gating networks dynamically produce expert-merging coefficients from hidden-state representations, while the evolutionary algorithm incorporates greedy hypervolume contribution for effective evolution of these gating networks, achieving consistent improvements on large and noisy training datasets and broader coverage of the non-convex Pareto front. Experiments across three tasks demonstrate the effectiveness of Evolutionary Soups over baselines: it achieves the best hypervolume, linear utility, and Tchebyshev utility (~20% improvement) among controllable methods on all tasks.

1 Introduction

Controllable multi-objective generation must adapt to user preferences and prompts while covering the Pareto front. Evolutionary Soups addresses limitations of existing merging methods with context-aware MoE gating and evolutionary search.

  • CMOG seeks a single model that maximizes utility for any inference-time user preference without retraining.
  • Existing approaches face prompt-dependent fronts, incomplete single-gating coverage, and inaccessible non-convex trade-offs.
  • Evolutionary Soups uses an MoE architecture with dynamic hidden-state merging and evolutionary learning of gating networks.
  • Its gating networks are context-aware and operate per layer, while greedy hypervolume contribution guides evolution toward broader Pareto-front coverage.
  • Experiments report gains over CMOG baselines in utility and Pareto optimality across multiple tasks and objectives.

2 Problem Analysis

Current CMOG methods restrict reachable trade-offs through fixed or shared gating and linear scalarization. Evolutionary Soups combines context-aware per-layer gating with greedy hypervolume-based evolutionary search to expand and explore the Pareto front.

  • Evolutionary Soups design: Evolutionary Soups uses a shared gating network conditioned on attention hidden states to produce per-layer expert-merging coefficients.
  • Evolutionary Soups design: Greedy HVC selection iteratively adds the candidate with the highest marginal hypervolume contribution, monotonically improving the Pareto front.
  • Prompt-dependent fronts: A fixed merging coefficient cannot remain optimal across prompts because safe and risky inputs can induce different Pareto fronts under the same preference.
  • Incomplete single-gating coverage: Single gating uniformly across layers yields a reachable region strictly contained within that of per-layer coefficients.
  • Non-convex inaccessibility: Linear scalarization confines reachable solutions to the convex hull, leaving non-convex Pareto-front regions inaccessible.
  • Non-convex inaccessibility: Evolutionary algorithms evaluate reward vectors to approximate the full Pareto front without scalarization.

3 Evolutionary Soups

Evolutionary Soups combines context-aware per-layer MoE gating with evolutionary search to expand and robustly cover controllable Pareto fronts. Frozen objective-specific experts are merged dynamically, while greedy hypervolume contribution selects diverse gating networks for preference-conditioned inference without retraining.

  • 3.1 ES Pipeline: ES trains objective-specific LoRA experts over a shared frozen base, freezes them in an MoE, and evolves the gating networks with greedy HVC selection.Evolution proceeds through nondominated selection, crossover, and mutation over fresh prompt chunks.
  • 3.1 ES Pipeline: At inference, ES selects from the evolved non-dominated gating set the network maximizing linear or Tchebyshev utility for the requested preference.The evolved set is computed during training, so selection costs O(|P̂|) and does not add MoE forward-pass overhead.
  • 3.2 MoE Gating Network: The gating network is a shared 2-layer MLP that maps attention hidden states at each layer to expert-merging coefficients applied to LoRA increments over the frozen FFN.Hidden-state aggregation loads experts once and supports context-aware, layer-wise control.
  • 3.2 MoE Gating Network: α-entmax maps gating logits to simplex coefficients, suppressing low-weight experts and permitting exact 0 or 1 coefficients that can recover individual expert outputs.This sparsity concentrates merging on context- and layer-relevant experts and allows the merged states to reach the Pareto boundary.
  • 3.2 MoE Gating Network: Per-layer context-aware gating strictly expands the reachable reward region beyond fixed-coefficient and single-gating classes.The expansion follows from strict class containment; universal approximation is needed only to ensure the non-dominated gate is realizable.
  • 3.3 Greedy HVC Selection: Greedy HVC builds the retained population additively from parent and offspring candidates, favoring underexplored regions and impactful solutions under small populations.Theoretical results state that retained sets never lose hypervolume on each chunk and achieve at least (1 − 1/e) of optimal hypervolume over the combined pool.
  • 3.3 Greedy HVC Selection: The HVC robustness guarantee is limited to in-distribution prompts and makes no claim about distribution-shifted or adversarial prompts.Test-set results measure competitive expected fitness under the training distribution.

4 Experiments

Experiments show that Evolutionary Soups achieves strong controllable multi-objective performance across three tasks, while ablations link its gains to evolutionary optimization, per-layer gating, and greedy hypervolume selection.

  • Pareto Fronts: On Beaver, ES extends the Pareto front beyond RS, HoE, and MOD and remains stable in the high-reward region.ES matches the MORLHF reference on Beaver while composing frozen experts rather than retraining per preference.
  • Main Results: ES achieves the best mean linear utility among CMOG methods on all three tasks and the lowest Tchebyshev utility on all three.On Beaver, ES reaches linear utility 0.78 and Tchebyshev distance 0.14.
  • Main Results: ES ranks first or second in hypervolume across all three tasks, indicating broad Pareto-front coverage.The hypervolume uses reference point [−0.1]n.
  • Inference Cost: ES selects a gating network from the evolved Pareto set in O(|P̂|) time without adding MoE forward-pass overhead.The evolved front is computed once and reused across preferences.
  • Ablation Study: Ablations show that per-layer gating and evolutionary optimization are decisive, while Pareto-front selection adds a further consistent gain.Gradient and Single collapse to lower near-linear fronts; ES slightly exceeds NSGAII across most regions on both backbones.

5 Related Work

Related work spans single-policy, multi-policy, and meta-policy approaches, including model soups, dynamic MoE routing, and evolutionary multi-objective optimization. Evolutionary Soups differs by evolving context-aware, per-layer MoE gates for reusable preference control.

  • Soup-like Methods: Soup-like methods train objective-specific models and merge them at inference time to trace Pareto trade-offs without retraining.Rewarded Soups interpolates expert weights, while Bone Soups trains experts with multi-objective rewards.
  • MoE Routing: MoE approaches dynamically route among experts using context or preference information for multi-objective control.ArmoRM-MoE uses context-conditioned reward scalarization, while HoE routes among single-objective LoRA experts with gradient-trained routers.
  • Evolutionary Optimization: Evolutionary optimization approximates Pareto fronts through decomposition-based or hypervolume-based selection.Evolutionary Soups applies this paradigm to learned gating networks rather than static checkpoints, prompts, or LoRA expert searches.
  • Evolutionary Soups: Evolutionary Soups evolves gates that compute context-aware, per-layer merging coefficients and reuses the evolved Pareto front for fine-grained preference control without retraining.This contrasts with related work that evolves checkpoints, searches LoRA experts, or evolves discrete prompts.

6 Conclusions

The paper concludes that Evolutionary Soups addresses controllable multi-objective generation through dynamic per-layer gating and evolutionary Pareto-front search. Experiments and ablations support broader controllable coverage without retraining, while performance approaches the full-training reference.

  • Core Design: ES combines context-aware per-layer gating with greedy HVC selection to address coverage and non-convexity limitations in CMOG.The conclusion attributes fixed-front recovery to per-layer gating and non-convex exploration to evolutionary search.
  • Controllability: The Pareto front is computed once during training, then ES selects the utility-maximizing gating network for each preference at inference time.This provides CMOG without retraining.
  • Evidence: Ablations confirm the individual design choices, and experiments across multiple tasks show ES outperforming other CMOG approaches.The paper reports that ES approaches the full-training upper bound.

Limitations

The paper identifies limitations involving the gap to full model training, many-objective exploration, evaluation cost, and experimental scope. These constraints motivate larger populations, more efficient evaluation, broader ablations, and fuller human validation.

  • Full Model Training: ES does not close the gap to MORLHF on Assistant because frozen-expert merging bounds the reachable reward region.The paper suggests jointly fine-tuning experts and gating networks as one possible direction.
  • Many-Objective Tasks: A fixed-size population may undersample the Pareto front as the number of objectives grows.Scaling the population and extending ES to many-objective settings remain future work.
  • Evaluation Cost: Evaluating LLM generations for each population member remains the dominant evolutionary-search bottleneck and constrains population size and generations.The cost is incurred once and amortized across all served preferences.
  • Experimental Scope: Compute constraints limit the ablation scope, multi-seed comparisons, and human evaluation.Only Beaver receives ablations, only ES and RS use three seeds, and reward-model metrics raise an overfitting-to-the-judge concern partially checked on Beaver.

Ethical Considerations

Evolutionary Soups is presented as a research framework rather than a deployment recipe because controllable trade-offs can be dual-use, reward models are imperfect proxies, and safety is bounded by frozen experts.

  • Inference-time control over helpfulness versus harmlessness can turn a safety-motivated refusal into a substantive response when preferences shift toward helpfulness.The authors identify this controllability as dual-use because selection requires no retraining.
  • The authors characterize Evolutionary Soups as a framework for studying Pareto-optimal trade-offs rather than a deployment recipe.
  • Reward-model-based Pareto optimality may not reflect true alignment because proxies can be biased, miscalibrated, or gamed.The paper makes no guarantee for adversarial or distribution-shifted prompts.
  • Evolutionary Soups navigates the trade-off space represented by frozen experts rather than introducing safety capabilities beyond them.Expert safety therefore upper-bounds what any evolved front can achieve.

A Preference-Coefficient Gap

The section shows that optimal expert-merging coefficients vary across prompts under the same user preference, motivating prompt-aware and per-layer gating rather than fixed coefficients.

  • Under identical preferences, safe and risky prompts can require different helpfulness and harmlessness merging coefficients.A safe prompt may need more helpfulness weighting, whereas a risky prompt requires stronger harmlessness weighting.
  • Up to +13.1% on Beaver and +24.7% on Summary, oracle coefficients outperform naive λ = µ scalarization across the reported datasets.The oracle densely samples the coefficient simplex and selects the best coefficient for each prompt under a given preference.
  • Because λ∗ varies across prompts under identical µ, no fixed coefficient can realize optimal routing for every input.This establishes the non-constancy premise used to motivate prompt-conditioned adaptation.
  • Per-layer, context-aware gating is designed to enlarge the reachable objective space beyond fixed-coefficient or single-gating mechanisms.The gating class assumes hidden states contain sufficient routing information and can approximate the target routing function.
  • Greedy hypervolume contribution selection provides at least (1 − 1/e) of the optimal size-P subset in general and is exact for n ≤ 3 objectives.The guarantee follows from hypervolume being monotone and submodular under a fixed reference point.

B.3 Proofs of Main Theorems

The proofs establish utility-optimal selection from an evolved Pareto front, strict expansion from context-aware per-layer gating, and reliable hypervolume-based evolutionary retention.

  • Given the evolved Pareto-optimal set, selecting the gating network maximizing preference utility is at least as good as any candidate in that set.If the evolved set equals the true Pareto-optimal set, the selected network is globally utility-optimal.
  • The inference-time selection rule requires O(| ˆP|) lookup and adds no per-token overhead to the MoE forward pass.
  • The per-layer context-aware gating class contains objective-space points not dominated by any fixed-coefficient or single-gating solution.Universal approximation makes the target routing realizable to arbitrary precision under the stated assumptions.
  • Greedy hypervolume contribution selection retains subsets with at least (1 − 1/e) of the optimal hypervolume over each combined candidate pool.The analysis also gives geometric decay for noise-driven survival across independent fresh chunks.
  • Together, the theorems cover preference selection, reachable-region expansion, and robust Pareto-front evolution across the ES pipeline.The reported monotonicity is in expectation across generations because different chunks introduce fluctuations.

C Implementation Detail

Experiments evaluate Evolutionary Soups on three multi-objective tasks using pretrained reward models, shared-backbone LoRA experts, and evolutionary procedures whose stability and cost shape implementation choices.

  • Datasets and Reward Models: All datasets are tokenized and length-filtered to 8–512 tokens before prompts generate responses scored by pretrained reward models.
  • Datasets and Reward Models: Three tasks cover safety-critical helpfulness versus safety, dialogue harmlessness/helpfulness/humor, and summary quality/faithfulness/deberta score.The datasets are BeaverTails, Anthropic HH-RLHF-derived Assistant data, and Summarization data.
  • Training Process: Experts use SFT and PPO LoRA adapters over one shared frozen backbone, while per-layer gating networks dynamically merge expert increments.For LLaMA-2-7B, the shared base is approximately 14 GB, each expert about 0.19 GB, and each gating network about 2 MB.
  • Computational Considerations: Evolutionary search cost is incurred once and amortized across preferences, but LLM-generation evaluation remains the dominant computational bottleneck.
  • Evolutionary Stability: Evolutionary evaluation scores one dataset chunk per generation because full-dataset fitness evaluation is prohibitively expensive, introducing distributional noise.This noise motivates stability mechanisms intended to distinguish robust candidates from favorable-chunk survivors.
  • Evolutionary Stability: Dual-front retention degraded the Pareto front relative to standard ES because intersection-based retention was overly conservative toward long-history incumbents.Promising offspring evaluated on one chunk could be filtered out before demonstrating their potential.

D.2 Gradient-Based Refinement

The paper tests whether evolved nondominated gating networks can be further improved by preference-directed gradient refinement. Refinement produces mixed outcomes, whereas greedy hypervolume-contribution selection preserves the front and suggests near-convergence.

  • Refinement evaluation: Figure 7 compares gradient-based refinement and dual-front retention against the standard Evolutionary Soups Pareto front on Beaver.The figure is specifically used to inspect whether additional refinement improves the evolved solutions.
  • Preference-directed refinement: Gradient refinement first identifies each gating network’s strongest preference direction over the preference simplex.The direction maximizes the utility attained by that network; for linear utility, all preference mass goes to its strongest normalized objective.
  • Preference-directed refinement: Each network is then optimized with PPO using single-policy utility scalarization under its selected preference.The procedure pushes individuals along their own directions in objective space and also serves as a convergence check.
  • Refinement outcome: Refinement has mixed effects: some solutions become stronger, while others fall below their pre-refinement objective values.Thus, gradient updates do not uniformly improve the evolved individuals.
  • Selection guarantee: Greedy hypervolume-contribution selection retains candidates only when they do not reduce hypervolume on the scored chunk, preventing front regression by construction.This contrasts with gradient refinement, which can degrade some solutions.
  • Convergence interpretation: The failure of refinement to uniformly improve solutions indicates that the standard Evolutionary Soups front has approached convergence.Individuals are described as lying near locally attainable optima in their respective directions.

E Detailed Results

Detailed results report three-objective Pareto fronts and utility tables for the Assistant and Summary tasks, alongside preference-based evaluations across Beaver, Summary, and Assistant. The evaluation uses task-level normalization and compares controllable methods with a separately trained reference.

  • Pareto fronts: Three-objective Pareto fronts are reported for Assistant’s harmless–helpful–humor task and Summary’s summary–faithful–deberta task.These fronts visualize trade-offs among the three named objectives.
  • Utility results: Tables 3 and 4 report linear utility and Tchebyshev utility results, respectively.The table captions identify the two utility summaries used in the detailed results.
  • Evaluation protocol: Preference-based evaluations cover Beaver, Summary, and Assistant, selecting the best-matching evolved candidate for each Evolutionary Soups preference.Other methods are evaluated directly at their sampled preferences, while MORLHF trains a separate model per preference.
  • Evaluation protocol: Objective values are min–max normalized per task using bounds pooled over every method and seed.Reported values are seed means, with three seeds for Evolutionary Soups and Rewarded Soups and single runs for the other methods.
  • Compared methods: The detailed comparison includes Rewarded Soups, MOD, RiC, HoE, Evolutionary Soups, and the non-controllable MORLHF reference.The reference trains a separate model for each preference and provides an upper-bound comparison without inference-time controllability.
Loading 2608.29978v1…