PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation

Zeyuan Ma, Hongshu Guo, Jiacheng Chen, Guojun Peng, Zhiguang Cao, Yining Ma, Yue-Jiao Gong

arXiv:2403.01131v2math.OCcs.AIcs.CLcs.LGcs.NEcs.SE

TL;DR

LLM-based optimization approaches face limits in efficiency, prompt sensitivity, and domain-specific knowledge. LLaMoCo fine-tunes models on optimization problem-code examples using a two-phase strategy. The authors report that tuned models outperform existing methods, with a relatively small model surpassing GPT-4.

  • Problem

    Existing LLM optimization approaches face efficiency and prompt-design limitations, and pretrained models lack domain-specific expertise for customized optimizers.

    Source: §1 · pp. 1–2

  • Method

    LLaMoCo fine-tunes general LLMs on optimization problem-code instructions, using contrastive learning before instruction tuning.

    Source: §1 · p. 2

  • Results

    The authors report that LLaMoCo models consistently outperform existing approaches, and that a relatively small tuned LLM can surpass GPT-4 as an optimization code generator.

    Source: §5 · p. 9

  • Takeaways & Limitations

    A stipulated prompt protocol lets users focus on the optimization problem while reducing the domain knowledge and effort needed for prompt design.

    Source: §1 · p. 2

    The stated formulation assumes minimization problems with specified objective functions and inequality or equality constraints.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Recent research explores optimization using large language models (LLMs) by either iteratively seeking next-step solutions from LLMs or directly prompting LLMs for an optimizer. However, these approaches exhibit inherent limitations, including low operational efficiency, high sensitivity to prompt design, and a lack of domain-specific knowledge. We introduce LLaMoCo, the first instruction-tuning framework designed to adapt LLMs for solving optimization problems in a code-to-code manner. Specifically, we establish a comprehensive instruction set containing well-described problem prompts and effective optimization codes. We then develop a novel two-phase learning strategy that incorporates a contrastive learning-based warm-up procedure before the instruction-tuning phase to enhance the convergence behavior during model fine-tuning. The experiment results demonstrate that a CodeGen (350M) model fine-tuned by our LLaMoCo achieves superior optimization performance compared to GPT-4 Turbo and the other competitors across both synthetic and realistic problem sets. The fine-tuned model and the usage instructions are available at https://anonymous.4open.science/r/LLaMoCo-722A.

1. Introduction

Existing LLM optimization approaches face limits in efficiency, prompt design, and domain-specific expertise. LLaMoCo fine-tunes general LLMs to generate optimization code and reports strong benchmark performance, including results from a small model that surpasses GPT-4.

1. Introduction

  • Iterative solution-to-solution methods are constrained by context windows, resource-intensive rounds of conversation, and sensitivity to prompt design.Directly prompting for optimization programs can require fewer conversations, but still depends on carefully crafted prompts and optimizer expertise.Source: §1 · p. 1
  • LLaMoCo fine-tunes general-purpose LLMs on code-to-code pairs of problem prompts and executable optimization programs.After training, the model generates an optimizer based on an unseen problem's structure.Source: §1 · p. 2
  • The framework pairs a large-scale optimization instruction set with contrastive warm-up before instruction tuning to reinforce prompt representations and improve convergence.The authors also describe a stipulated prompt protocol that lets users focus on the optimization problem rather than constructing prompts manually.Source: §1 · pp. 2–3
  • Fine-tuned CodeGen-350M shows robust optimization performance, surpassing existing methods and even GPT-4 in the reported benchmarks.The authors attribute substantial gains to tuning a relatively small model on domain-specific tasks.Source: §1 · p. 2

2. Related Works

The related work covers fine-tuning strategies, specialized code-generation models, and prompt-based LLM optimizers. It notes that optimization-domain fine-tuning of general LLMs remains unexplored in the cited optimizer work.

2.1. Fine-tuning LLMs

  • Instruction tuning uses formatted examples and sequence-to-sequence loss to inject task knowledge; alignment tuning targets human preferences, with DPO described as a more efficient RLHF alternative.RLHF estimates a reward model from human-preference data and uses it to post-train LLMs with PPO.Source: §2.1 · p. 3

2.2. LLMs for Code Generation

  • Code-generation models are trained on code or fine-tuned from general LLMs, but general-purpose models may still have limited performance on particular tasks.The subsection describes models including AlphaCode, StarCoder, Codex, CodeGen, and Code Llama.Source: §2.2 · p. 3

2.3. LLMs as Optimizers

  • Most LLM optimizer work uses prompt engineering to iteratively improve candidate solutions or directly prompts for optimization programs; fine-tuning general LLMs with optimization knowledge remains unexplored.Iterative approaches can incur efficiency limits because they require extensive rounds.Source: §2.3 · pp. 3–4

3. LLaMoCo

LLaMoCo adapts general-purpose LLMs into optimization-code generators using an instruction set of problem prompts and effective optimizer implementations, followed by contrastive warm-up and instruction tuning.

3.1. Construction of Instruction Set

  • They synthesize 6,000 problems from benchmark-derived function and constraint sets, varying dimension, bounds, function composition, and optional constraints.The set includes constrained and unconstrained instances with varied global and local landscape properties.Source: §3.1 · p. 4
  • For each problem, benchmarking 23 optimizers and grid-searching their configurations selects the best-performing optimizer implementation as training knowledge.Configurations are tested multiple times to reduce variance.Source: §3.1 · pp. 4–5
  • Rephrased Python- or LaTeX-style descriptions from 500 computer-science students expand problems into 32,570 prompt-and-optimizer-code examples.The prompt is the input; selected optimizer code with configurations is the output.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2. Two-Phase Instruction Tuning

  • Contrastive warm-up pulls together prompt representations that share a desired optimizer and separates those assigned different optimizers before next-token instruction tuning.The warm-up uses similar and dissimilar prompts within a mini-batch and does not require context generation.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
  • Optimizer-proportional sampling rebalances both training phases so each optimizer contributes approximately equal numbers of sampled pairs per epoch.The strategy is intended to help models learn from minority instances.Source: §3.2 · p. 6

4. Results and Discussions

Across synthetic tests, fine-tuned LLaMoCo models outperform competitors, and realistic-problem tests show best generalization for LLaMoCo-S. Ablations report gains from contrastive warm-up, varied descriptions, and balanced sampling.

4.1. Experimental Setup

  • CodeGen-Mono (350M), Phi-2 (2.7B), and Code Llama (7B) were fine-tuned; larger models used LoRA, while CodeGen used full-parameter tuning.The experiments used 30k training pairs and 5 warm-up epochs followed by 20 instruction-tuning epochs.Source: §4.1 · p. 6

4.2. Performance Analysis

  • LLaMoCo models consistently outperform competitors; LLaMoCo-L raises optimization performance from 29.717% to 81.843% on Ieval.Table 1 covers unconstrained, constrained, and combined test problems.Source: §4.2 · p. 7, Table 1 · p. 7
  • OPRO and LMEA perform unsatisfactorily on complex tasks and consume tremendous tokens through repeated solution prompting.The authors identify both efficacy and efficiency as needing further improvement.Source: §4.2 · p. 7, Table 1 · p. 7
  • LLaMoCo-S achieves the best reported generalization on realistic problems from varied real-world scenarios.The test set includes industrial chemical-process, mechanical-engineering, and power-system problems, many with high-dimensional spaces and complicated constraints.Source: Table 2 · p. 7, §4.2 · p. 8

4.3. Ablation study

  • Multilingual problem descriptions boost generalization, while rephrasing them in multiple writing styles further improves final training results.Figure 5 compares Python, LaTeX, and combined descriptions, with and without rephrasing.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Contrastive warm-up accelerates subsequent instruction-tuning convergence and supports accurate code generation and stronger optimization performance.Figure 4 presents training curves and performance-gain curves on Ieval.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Balanced example-proportional sampling improves performance on majority, minority, and overall test instances.Figure 5 compares models trained with and without data balancing.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4. Open-Ended Discussion: Is GPT-4 a True Optimization Expert?

  • GPT-4 usually generates SLSQP, but a Vanilla DE hint produces competitive results on problems where SLSQP underperforms.The authors interpret this pattern as sufficient general optimization knowledge but underfitting in choosing an optimizer for a particular problem.Source: §4.4 · p. 8

5. Conclusion

LLaMoCo adapts general-purpose LLMs into optimization systems using more than 30k demonstrations and a two-phase instruction-tuning strategy. The models consistently outperform existing approaches, while the conclusion identifies adding more instances as a need for better generalization.

5. Conclusion

  • LLaMoCo models consistently outperform existing approaches, and a relatively small model can exceed GPT-4 as an optimization code generator.The conclusion also describes LLaMoCo as a preliminary exploratory research effort.Source: §5 · p. 9
  • The framework combines more than 30k demonstration examples with a novel two-phase instruction-tuning strategy.The authors describe the strategy as fine-tuning a series of LLMs.Source: §5 · p. 9
  • The authors identify augmenting the instruction set with more instances as a need to enhance generalization performance.They also consider further alignment tuning a promising future direction.Source: §5 · p. 9

A.1. Optimizer Pool and The Used Assets

The optimizer pool contains 23 selected optimizers split into unconstrained and constrained groups. The implementation uses open-source libraries, with their codebases and licenses listed in Table 3.

A.1. Optimizer Pool and The Used Assets

  • The optimizer pool contains 23 well-performing optimizers divided into unconstrained and constrained groups.The unconstrained group lists 16 optimizers and the constrained group lists 7.Source: §A.1 · p. 14
  • The unconstrained group includes simulated annealing, PSO, DE, CMA-ES, Bayesian optimization, and other optimizer families.The listed methods also include LA-MCTS, MadDE, and NSA.Source: §A.1 · p. 14
  • The constrained group includes SLSQP, Trust-Constr, COBYLA, L-BFGS-B, HECO-DE, DTPSO, and GA-TDX.Source: §A.1 · p. 14
  • The implementations use open-source libraries, and Table 3 lists adopted codebases and their licenses.The authors state that development and deployment follow those licenses.Source: §A.1 · p. 14, Table 3 · p. 14

A.2. Benchmarking Process

For each problem instance, the benchmarking process searches optimizer-specific hyperparameter configurations and selects a configured optimizer with its code implementation. It estimates performance across five independent runs and assigns zero to configurations that fail to execute.

A.2. Benchmarking Process

  • The process spans a grid-search configuration space for each optimizer based on its tunable hyperparameters.DEAP-DE’s three hyperparameters, each with five predefined values, yield 125 configurations.Source: §A.2 · p. 14, Table 4 · p. 15
  • Expected optimization performance is unbiased-estimated using five independent runs.Source: §A.2 · p. 14
  • When an optimizer fails to execute on a problem, its corresponding evaluation is set to zero.Examples include constraint incompatibility, non-convex problems, and time-consuming Bayesian optimization on high-dimensional problems.Source: §A.2 · p. 15
  • Benchmarking selects a configured optimizer and its code implementation for each problem instance after evaluating optimizer-specific configurations.Table 4 presents the optimizers’ configurations and hyperparameter-tuning settings.Source: §A.2 · pp. 14–15, Table 4 · p. 15

B. Details of Data Augmentation

The authors diversify prompts by collecting alternative Python and LaTeX formulations from 500 computer science students. They use the observed writing patterns to create rephrased versions of each problem instance’s objective and constraints.

B. Details of Data Augmentation

C. Details in Training Process

The instruction-tuning phase uses homogeneous batches to align rephrased prompts for the same problem, while batch sizes vary by model under single-GPU resource limits.

Homogeneous batch sampling

  • Mini-batches pair prompts from the same problem instance to reinforce alignment across rephrased versions.The strategy samples data pairs from the same instance during instruction tuning.Source: C. Details in Training Process · p. 17
  • Homogeneous sampling yields consistent training boosts for LLaMoCo-S, LLaMoCo-M, and LLaMoCo-L.The authors attribute the improvement to more effective learning of patterns in cross-modal prompt data.Source: C. Details in Training Process · p. 17

Batch size

  • On one NVIDIA A800, batch size is 4 for CodeGen-Mono (350M), 3 and 2 for Phi-2 (2.7B), and 3 and 1 for Code Llama (7B) across the two phases.The larger models could not fit a batch of 4, even with LoRA.Source: C. Details in Training Process · p. 17

D. Calculation of Experimental Statistics

The evaluation averages four performance metrics across five independent runs per problem instance, using separate measures for program errors, repair effort, optimization quality, and computational overhead.

D. Calculation of Experimental Statistics

  • Each approach is evaluated on a group of problem instances using its generated optimizers over 5 independent runs, with four metrics calculated for overall performance.The metrics are error rate, recovery cost, optimization performance, and computational cost.Source: D. Calculation of Experimental… · p. 18
  • Error rate measures the proportion of generated programs that encounter compilation or runtime errors during execution.Each run on each instance counts toward the error-program total.Source: D. Calculation of Experimental… · p. 18
  • Recovery cost measures the repair effort for error programs using the number of program lines that need repair relative to program length.The passage describes the line count for each program and the count of lines requiring repair.Source: D. Calculation of Experimental… · p. 18
  • Optimization performance is measured by min-max normalized objective-value descent relative to an estimated optimal objective value for each instance.The authors use the best objective value achieved during benchmarking as the estimate.Source: D. Calculation of Experimental… · p. 18
  • Computational overhead is measured by the average input-plus-output tokens consumed per problem instance, rather than wall-time complexity.The authors avoid wall-time comparisons because API communication budgets could bias the results.Source: D. Calculation of Experimental… · p. 18

E.1. Synthetic unconstrained example

The examples show LLaMoCo-S generating optimizers for synthetic unconstrained and constrained problems and for a realistic out-of-distribution constrained problem, with each example paired with its prompt and generated code.

E.1. Synthetic unconstrained example

  • For a 16-dimensional synthetic unconstrained problem composed of two basic functions, LLaMoCo-S outputs BIPOP-CMA-ES within a 40,000-evaluation budget.The prompt and generated optimization program are shown in Figure 8.Source: Figure 8 · p. 19, §E.1 · p. 19

E.2. Synthetic constrained example

  • For a 23-dimensional synthetic constrained problem with two added quality constraints, LLaMoCo-S outputs SLSQP within a 20,000-evaluation budget; GPT-4 Turbo selects the same algorithm.LLaMoCo-S's configurations achieve higher optimization performance than GPT-4 Turbo's default configurations; Figure 9 presents the prompt and generated program.Source: Figure 9 · p. 20, §E.2 · p. 20

E.3. Realistic example

  • For a 9-dimensional realistic problem with many constraints and an out-of-distribution structure, LLaMoCo-S outputs GA-TDX within a 10,000-evaluation budget, while GPT-4 Turbo suggests DE.The passage characterizes DE as hardly adopted for constrained problems; Figure 10 shows the prompt and generated program.Source: Figure 10 · p. 21, §E.3 · p. 21
Loading 2403.01131v2…