PaperlayerView exact revision on arXiv

Source-linked AI summary

FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema

Junru Lu, Siyu An, Min Zhang, Yulan He, Di Yin, Xing Sun

arXiv:2402.11811v4cs.CL

TL;DR

Existing ad-hoc prompt optimization can expose sensitive information and generalize poorly beyond its in-box generator. FIPO trains a local optimizer using a modular template and preference data, and reports superior results to online ad-hoc APO across benchmarks and generators.

  • Problem

    Existing online ad-hoc APO relies on external services and in-box responses, creating privacy risk and risking degraded performance on out-of-box generators.

    Source: §1 · pp. 1–2

  • Method

    FIPO uses a modular APO template and preference data to fine-tune a local optimizer that generates optimized task instructions.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Results

    FIPO achieves superior results to existing online ad-hoc APO across five downstream benchmarks and three diverse generators.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study evaluates a locally trained optimizer across diverse downstream generators rather than tailoring optimization to one in-box generator.

    Source: §1 · p. 2, §5 · p. 9

    FIPO leaves in-context examples unoptimized and evaluates primarily with accuracy, not interpretability, fairness, or ethical implications.

    Source: Limitations · p. 9

Abstract

from arXiv · show

When the quality of naive prompts is carefully optimized by human experts, the task performance of large language models (LLMs) can be significantly improved. However, expert-based prompt optimizations are expensive. Herein, some works have proposed Automatic Prompt Optimization (APO), to optimize naive prompts according to task outputs of given in-box testing models, with the help of advanced LLMs (e.g., GPT-4) in an ad-hoc way. Although effective, existing schemes suffer from poor generalization ability and privacy risk. To this end, we collect the first large-scale Prompt Optimization Preference dataset (POP), fine-tune offline local LLM-based optimizers, then fairly test with various downstream models. Our method allows accurate optimization of the core task instruction part within the naive prompt in a model-agnostic manner, and thus is named Free-from Instruction-oriented Prompt Optimization (FIPO). In specific, FIPO uses a modular APO template that dynamically integrate the naive task instruction, optional instruction responses, and optional ground truth to produce finely optimized prompts. The POP dataset is meticulously constructed using advanced LLMs, undergoing rigorous cross-validation by human experts and analytical models. Leveraging insights from the data with Tulu2 models and diverse fine-tuning strategies, we validate the efficacy of FIPO framework across five public benchmarks and six testing models. Check codes and data here: https://github.com/LuJunru/FIPO_Project.

1 Introduction

The introduction frames online ad-hoc APO as privacy-risky and model-specific, then presents FIPO as a locally trained alternative. It also introduces POP preference data and reports superior results across benchmarks and generators.

1 Introduction

  • Online ad-hoc APO risks exposing sensitive information to external services and can degrade on out-of-box generators because it depends on in-box responses.Source: §1 · pp. 1–2
  • FIPO fine-tunes a local optimizer with a generic APO template, allowing prompt optimization without pre-built testing generators.The framework is designed to apply across testing generators.Source: §1 · p. 2
  • The authors compile a 30,000-example Prompt Optimization Preference dataset and explore SFT, DPO, IPO, and Iterative Preference Learning.Source: §1 · p. 2
  • FIPO achieves superior results to existing online ad-hoc APO across five downstream benchmarks and three diverse generators.Source: §1 · p. 2

2 Methodology

FIPO formulates prompt optimization as local end-to-end generation and uses a modular template to collect preference data and train optimizers. Its methodology combines response-aware dataset diversification, several fine-tuning strategies, and cross-validation of the 30,000-example POP dataset.

2.1 Task Formulation

  • FIPO trains a local optimizer to generate an optimized prompt from a naive instruction, with naive responses and ground truth as optional inputs.At testing, the optimized prompt can be applied with a testing generator that may differ from the generator used to produce training responses.Source: §2.1 · p. 3

2.2 Modular Template

  • The modular template keeps the naive task instruction mandatory while allowing responses and ground truth to be included or omitted across data collection, training, and testing.The method keeps both responses during data collection, varies their presence during training, and removes them during testing to address exposure bias.Source: §2.2 · p. 3

2.3 Prompt Optimization Preference Data

  • POP contains 30,000 contrastive prompt-optimization examples collected by presenting naive prompts, responses, and GPT-4 ground truth to GPT-3.5-turbo and GPT-4.The naive prompts come from Alpaca; GPT-4 responses serve as ground truth because the dataset lacks official human ground-truth responses.Source: §2.3 · p. 4
  • GPT-4-generated responses and optimized prompts each achieved average cross-validation win rates above 85%.Validation used UltraRM critiques, GPT-4 self-judgment, and human expert checks.Source: §2.3 · p. 4, Table 1 · p. 4

2.4 Fine-tuning Strategies

  • Dataset diversification splits examples into eight types by response availability and instruction format, including generation and multiple-choice formats.Multiple-choice examples convert responses into candidates associated with the naive prompt.Source: §2.4 · pp. 4–5
  • Fine-tuning includes chosen-prompt-only SFT, DPO and IPO on chosen–rejected prompt pairs, and IPL that iteratively replaces inferior prompts and responses after self-judgment.IPO regularizes DPO by limiting the optimization range with squares.Source: §2.4 · p. 5

3 Experiments

The experiments compare offline FIPO with online APO baselines across downstream generators and benchmarks, then examine fine-tuning strategies and prompt examples. FIPO improves performance across benchmarks and generators, while results vary by optimizer size, training strategy, and prompt case.

3 Experiments

  • The evaluation first generates optimized prompts with the fine-tuned optimizer, then compares optimized and naive testing responses for answer quality.The experiments proceed from settings and baseline comparisons to fine-tuning analysis and case analysis.Source: §3 · p. 5

3.1.1 Baselines

  • The experiments compare offline FIPO with APE and PromptAgent, using GPT-3.5-turbo as the in-box generator and GPT-4 as optimizer for both baselines.FIPO optimizes the task instruction while retaining in-context examples; evaluation uses five benchmarks, with comparisons to APE and PromptAgent aligned to their six BBH tasks.Source: §3.1.1 · p. 5, §3.1.2 · p. 6, Table 3 · p. 7

3.1.2 Evaluation Benchmarks

  • Evaluation covers GSM8k, BBH, PiQA, CosmosQA, and MMLU, with accuracy reported for generation and multiple-choice benchmarks.FIPO is tested across all five benchmarks, while comparisons with APE and PromptAgent use six BBH tasks to match their evaluation settings.Source: §3.1.2 · pp. 5–6

3.2.1 Efficacy of FIPO

  • FIPO-optimized prompts improve performance across five benchmarks and downstream generators; average gains are 6.37% for Llama2-7B, 2.26% for Tulu2-13B, and 1.99% for Baichuan2-13B.Against online APO methods, FIPO leads in downstream tests except the in-box GPT-3.5-turbo; gains are larger on open-source 7B models than on 70B models, while GPT-3.5 gains up to 22% and GPT-4o about 8%.Source: §3.2.1 · pp. 6–7, Table 2 · p. 6, Figure 4 · p. 6

3.2.2 Fine-tuning Analysis

  • Tulu2-13B is too small for the task, and its optimized prompts score below human-written naive prompts on average.Preference supervision yields improvements of 0.31% to 4.34%; IPO outperforms DPO in standalone and IPL-integrated fine-tuning, while reduced dataset diversification weakens results on all benchmarks except BBH.Source: §3.2.2 · p. 7, Table 3 · p. 7

3.2.3 Case Analysis.

  • FIPO prompts add task-specific cues, including a leap-year fact for BBH, emphasis on “NOT” and an answer-letter instruction for MMLU, and Signal Detection Theory definitions for CosmosQA.For a GSM8K average problem, the prompt breaks the calculation into steps but also includes a cheating note with the final answer.Source: §3.2.3 · p. 7

4 Related Work

Related work describes prompt optimization as discrete or continuous search, alongside preference-optimization methods for aligning language models. The cited approaches include both prompt-search techniques and training strategies based on preference data.

  • The section characterizes APO as a technique for improving LLM capabilities in downstream settings and distinguishes discrete from continuous approaches.The supplied footnote states that answer-leaking notes occur mostly in GSM8K and BBH math questions, happening less than 10%.Source: §4 · p. 7
  • Discrete APO searches for prompts by selecting combinations of discrete tokens, while continuous APO searches in embedding space.Prefix Tuning and Prompt Tuning insert and update prefix vectors; hybrid methods combine continuous embeddings with discrete templates.Source: §4 · p. 8
  • Preference optimization aligns LLMs using preference data; PPO uses a reward model and reinforcement learning, while DPO uses implicit modeling to avoid explicit reward models.The passage notes PPO's training instability and cost as drawbacks.Source: §4 · p. 8

5 Conclusion

FIPO frames prompt optimization as end-to-end text generation using a modular template that can take a naive prompt, response, and ground truth as inputs. The paper collects preference data, applies fine-tuning strategies, and evaluates the framework across benchmarks and downstream generators.

  • FIPO's modular template treats prompt optimization as end-to-end text generation, flexibly taking a naive prompt, response, and ground truth as inputs to produce an optimized prompt.The paper collects a prompt-optimization preference dataset, applies multiple fine-tuning strategies, and evaluates the framework with downstream generators.Source: §5 · pp. 8–9

Limitations

The authors identify three limitations: optimized prompts can contain overly detailed “cheating notes,” evaluation focuses primarily on accuracy, and in-context examples are not optimized.

  • FIPO can produce overly detailed “cheating notes,” particularly for mathematical tasks, which may not align with the intended use of prompt optimization.The paper notes that this can enhance performance.Source: Limitations · p. 9
  • Evaluation primarily measures accuracy, leaving interpretability, fairness, and ethical implications for future consideration.Source: Limitations · p. 9
  • FIPO optimizes task instructions but does not optimize in-context examples.Source: Limitations · p. 9

A Dataset Diversification

The diversification process creates eight data formats from combinations of the naive prompt, optional responses, and response format. For IPL, it reuses collected data to add instruction-following and discrimination examples.

A Dataset Diversification

  • Eight data formats result from combining the naive prompt with optional naive responses, ground-truth responses, and generation or multi-choice formats.Source: A Dataset Diversification · p. 12
  • Type 3 contains only the mandatory naive prompt and is also used during testing, when responses are unavailable.Source: A Dataset Diversification · p. 12
  • Type 6 converts a generation-format example to multi-choice and adds a binary naive response while retaining the same task description.Source: A Dataset Diversification · p. 12
  • IPL reuses the raw data to add 15k instruction-following examples and 15k discrimination examples.The instruction-following examples pair naive prompts with ground-truth responses; a new meta-prompt supports inner discrimination.Source: A Dataset Diversification · pp. 12–13

B HyperParameters and Training Cost

The appendix reports training settings and additional IPL details. IPL uses self-generated prompt updates after a warm-up epoch, while its additional training data increases training time per epoch.

B HyperParameters and Training Cost

C Additional details of IPL

  • IPL begins self-rewarding updates after one warm-up epoch, generating a new prompt, judging it against the naive prompt, and using the better prompt to generate the next response.Source: C Additional details of IPL · p. 13
  • The optimizer achieved 100% classification accuracy using 5% of the training data as validation, and updated prompts conservatively.Source: C Additional details of IPL · p. 14
  • Only 2.4% of naive prompt-response samples are upgraded, while the self-game approach significantly promotes downstream test prompt optimization.Source: C Additional details of IPL · p. 14

D Entire fine-tuning results

The fine-tuning results distinguish SFT, basic preference optimization, and IPL: SFT models perform below the naive baseline, while full IPL-IPO training shows an upward trend. Results also vary with model size and the preference-learning strategy.

D Entire fine-tuning results

E Detailed results of 6 BBH subsets

FIPO generally leads across downstream testers, while PromptAgent and GPT-4 perform well on some individual tasks but are less comprehensive overall. Reported task-level wins span multiple models and BBH subtasks.

Overall Performance

  • FIPO generally leads across downstream testers and average scores, while PromptAgent and GPT-4 are less consistent overall.Naive prompts and APE often lag behind; on GPT-4o, APE averages 0.797 versus FIPO's 0.844.Source: E Detailed results of 6 BBH subsets · pp. 15–16

Optimizer Comparison

Task-specific Performance

  • FIPO records task highs of 0.683 on Llama2-7B Temporal and 0.616 on Llama2-7B Causal Judgment.The passage also reports highs of 0.525 on Tulu2-7B Penguins and 0.891 on Tulu2-7B Causal Judgment.Source: E Detailed results of 6 BBH subsets · p. 15
  • FIPO records highs of 0.810 on GPT-4o Geometry, 0.840 on GPT-4o Causal Judgment, and 0.792 on Qwen2-72B-Instruct Epistemic.These are task-specific scores reported in the results analysis.Source: E Detailed results of 6 BBH subsets · p. 15

F Specific ablation discussion on BBH

BBH behaves differently from other datasets in the ablation experiments, yet FIPO retains an advantage on six representative BBH subtasks. The section also reports cost comparisons, prompt-cheating patterns, and the roles of its meta-prompts.

F Specific ablation discussion on BBH

  • FIPO retains a clear advantage over other leading prompt optimizers across six representative BBH subtasks.The ablated optimizers otherwise have suboptimal performance across benchmarks, except on BBH; the authors hypothesize that BBH's symbolic reasoning tasks explain the difference, and suspect further discrete-instruction diversification may affect such tasks.Source: F Specific ablation discussion on BBH · p. 16

G Cost comparison: FIPO vs. others

  • $300 in one-time data collection and 10 hours of training equal 75 GPT-4 usages; local optimization lowers total costs when optimizing massive prompt sets.The cost estimate's inference comparison uses the 149 test samples in the BBH “penguins_in_a_table” subtask; deployment took under three minutes in practice.Source: Table 9 · p. 16, G Cost comparison: FIPO vs. others · p. 16

H Overwhelmed Cheating Notes

  • Mathematical questions in GSM8K and BBH are more likely to receive cheating notes, but accounting for this does not change the reported overall conclusion.The analysis manually checked 256 random optimized prompts per benchmark.Source: Figure 6 · p. 16, H Overwhelmed Cheating Notes · p. 16

I Employed Meta Prompts

  • Tables 10–12 provide meta-prompts for data collection, data diversification, and discrimination, respectively.Table 11's diversification prompt can include naive responses and ground-truth responses optionally; Table 12 covers discrimination-data creation and discrimination during IPL training.Source: I Employed Meta Prompts · p. 16, Table 10 · p. 17, Table 11 · p. 18
Loading 2402.11811v4…