PaperlayerView exact revision on arXiv

Source-linked AI summary

MODP: Multi Objective Directional Prompting

Aashutosh Nema, Samaksh Gulati, Evangelos Giakoumakis, Bipana Thapaliya

arXiv:2504.18722v1cs.CCcs.AI

TL;DR

Prompt engineering remains subjective, while much research optimizes task performance without accounting for the LLM’s behavior. MODP adds LLM-intrinsic objectives to metrics-driven prompt development; on a sample dataset, it reports a 26% improvement over basic prompts.

  • Problem

    Prompt engineering remains subjective, and task-specific optimization often neglects the behavior of the LLM being prompted.

    Source: §1 · p. 1

  • Method

    MODP optimizes prompts across task-specific and LLM-intrinsic objectives using a metrics-driven framework.

    Source: §1 · p. 2

  • Results

    26% improvement from basic prompts on a sample dataset.

    Source: §8 · p. 8

  • Takeaways & Limitations

    MODP is designed to develop robust prompts while ensuring test performance translates effectively to production data.

    Source: §8 · p. 8

    As objectives grow, optimization becomes more complex; quantifying objectives can be difficult, and identifying LLM-intrinsic objectives may require prior model experience.

    Source: §8 · p. 8

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have led to their popularity across multiple use-cases. However, prompt engineering, the process for optimally utilizing such models, remains approximation-driven and subjective. Most of the current research on prompt engineering focuses on task-specific optimization, while neglecting the behavior of the LLM under consideration during prompt development. This paper introduces MODP -- Multi Objective Directional Prompting, a framework based on two key concepts: 1) multi-objectivity: the importance of considering an LLM's intrinsic behavior as an additional objective in prompt development, and 2) directional prompting: a metrics-driven method for prompt engineering to ensure development of robust and high-precision prompts. We demonstrate the effectiveness of our proposed ideas on a summarization task, using a synthetically created dataset, achieving a 26% performance gain over initial prompts. Finally, we apply MODP to develop prompts for Dell's Next Best Action support tool, which is now in production and is used by more than 10,000 internal support agents and serving millions of customers worldwide.

1 Introduction

Prompt engineering offers a more accessible alternative to costly fine-tuning, but remains subjective and inefficient without systematic guidance. MODP frames prompt development as optimization across task-specific and LLM-intrinsic objectives.

1 Introduction

  • Fine-tuning can be computationally expensive and infrastructure-intensive, while prompt engineering is more accessible but can remain time-consuming without systematic guidance.Prompting leverages knowledge embedded in the model without a weight-based learning approach.Source: §1 · p. 1
  • Prompt design remains subjective and trial-and-error-driven, creating inconsistencies across users, tasks, and domains.Model and prompt choices are also frequently based on anecdotal evidence or heuristics.Source: §1 · pp. 1–2
  • MODP treats prompt engineering as optimization across task-specific performance requirements and objectives arising from the LLM’s intrinsic behavior.The framework aims to directionally improve prompt iterations and can compare different LLMs and prompting methods.Source: §1 · p. 2

2 Related Work

Prior prompt-engineering methods offer heuristics and automated optimization, but the paper identifies gaps in empirical guidance, reliable evaluation, and attention to LLM behavior. It proposes a generalized, multi-objective approach with weighted scoring to guide prompt selection.

2 Related Work

  • Existing prompting approaches provide heuristics for prompt design but lack empirical guidance for judging whether iterative changes improve outcomes.The paper also notes limited understanding of how different prompting strategies perform across scenarios.Source: §2 · p. 2

2.1.1 LLM Hallucinations.

  • Hallucinations can make outputs factually or logically inconsistent, with errors potentially propagating in decision-making and multi-step tasks.The paper highlights particular concern for applications selecting answers from context, such as RAGs.Source: §2.1.1 · p. 2
  • Long inputs can cause models to retain less information from the middle, risking overlooked instructions and incomplete or erroneous responses.The paper describes a trade-off between detailed prompts and retaining their full contents.Source: §2.1.3 · p. 2

2.1.3 The Guesswork of Prompt Design:

  • Prompt design remains context-dependent guesswork: concise prompts may increase hallucinations, while longer prompts risk information loss and noncompliance.Questions about optimal example counts and how many instructions a prompt can follow remain open.Source: §2.1.4 · p. 2

2.1.5 Addressing the guesswork and importance of the human in loop:

  • Automated optimizers face black-box LLMs, unreliable model-generated self-assessment, and continued need for human oversight.The paper also identifies a lack of consistent, directional measures of holistic prompt improvement.Source: §2.1.5 · p. 3

2.1.6 Task-Specific Optimization:

  • Task-focused prompt optimization can overlook production needs to balance task dimensions with LLM behaviors such as hallucination, toxicity, and prompt injection.The paper proposes a generalized framework combining heuristic iterations with analytical multi-objective optimization.Source: §2.1.6 · p. 3
  • The proposed scoring procedure weights task-specific and LLM-intrinsic objectives to select the prompt with the highest overall score.The algorithm describes assigning objective weights and comparing candidate prompts across models.Source: §2.1.6 · p. 4

3 Multi-Objective Directional Prompting

MODP operationalizes prompt development as a weighted, multi-objective process: define measurable goals, test them on a representative sample, iterate, and validate the selected prompt on held-out data.

3 Multi-Objective Directional Prompting

  • MODP organizes objective identification and balancing into a four-step process: sample data, define objectives and metrics, prioritize objectives, and iterate with evaluation.The framework is presented as a unified method for systematically identifying and balancing objectives.Source: §3 · p. 4

3.1 Defining Objectives

  • Prompt development is treated as multi-objective optimization, with measurable, incremental improvements across task-relevant factors such as accuracy, toxicity, and format adherence.Which objectives matter depends on the task.Source: §3.1 · p. 4

3.2 Finding a Representative Sample

  • Strategically selecting approximately 20% of the data provides a representative sample for testing objectives before full-scale evaluation.The authors report that targeted sampling improved consistency relative to simple random sampling in their experiments.Source: §3.2 · p. 4

3.3 Weighting of Objectives

  • Objective weights encode application priorities, allowing penalties such as hallucination or toxicity to count more heavily when required.The weighted score guides selection of prompts that fit the desired balance.Source: §3.3 · p. 4

3.4 Iterating Through Prompts

  • Prompt iterations are assessed using overall and category-specific accuracy alongside adherence to objectives such as toxicity and format constraints.Weighted scoring is used to compare prompts that may perform differently across objectives.Source: §3.4 · p. 4
  • The practical stopping point is when performance gains become negligible or core objectives reach satisfactory levels, rather than a separate formal stopping function.The authors say this cutoff saved computational resources and avoided over-tuning for minimal gain.Source: §3.4 · p. 5
  • The selected prompt is designated optimal when it performs robustly across weighted objectives, then tested on the remaining 80% of the dataset.The held-out evaluation is intended to confirm that improvements generalize beyond the representative sample.Source: §3.4 · p. 5

4 Dataset

The paper demonstrates its methodology using ReCoRD data organized into news categories and augmented with synthetic toxic examples. Its enterprise use case relied on proprietary data that cannot be disclosed publicly.

4.1 Creating a test dataset

  • ReCoRD supplies over 120,000 commonsense-reasoning queries generated from CNN/Daily Mail articles, with answers drawn from the corresponding passages.The dataset is described as crowd-worker-validated.Source: §4.1 · p. 5
  • The enterprise dataset cannot be disclosed publicly, so the paper demonstrates its full methodology with public or synthetic data.This preserves enterprise-data confidentiality while allowing methodological transparency.Source: §4.1 · p. 5

4.2 Creating Categories

  • News examples were categorized into Crime, Health, Markets, Politics, Sports, Tech, and Misc, with categories created using Mixtral.Misc covers articles spanning multiple categories or not fitting the other listed categories.Source: §4.2 · p. 5

4.3 Synthetically Creating Toxic Dataset

  • Because the news data was relatively non-harmful, the authors added synthetic toxic examples using Toxigen as their source.The passage describes Toxigen as a machine-generated dataset of toxic and benign statements concerning minority groups.Source: §4.3 · p. 5

5 Experimentation

The experiments developed prompts incrementally, evaluated both task accuracy and response handling, and measured performance overall and by category.

5 Experimentation

  • Prompt development began with basic instructions, then added complex edge-case guidance and Mixtral-specific tags before using progressively detailed, step-by-step prompts.Correct multiple-choice answers counted even when extra text appeared; toxic-content responses also counted when their phrasing varied slightly.Source: §5 · p. 5

5.1 Overall Accuracy

  • Overall accuracy combined correct toxic-content detection with correct answers to non-toxic questions.Toxic cases counted when flagged; non-toxic answers were checked against acceptable answers after cleaning and normalization.Source: §5.1 · p. 5

5.2 Category-wise Accuracy

  • Category-wise accuracy measures correct responses divided by the test cases in each category, revealing performance differences across areas such as toxicity, Crime, and Health.The framework also allows additional criteria when a task makes response formats important.Source: §5 · p. 5, §5.2 · p. 6

6 Observations

The observations compare prompt designs across accuracy and other objectives, finding that clear, specific instructions and model-aware features accompany stronger performance. Incremental refinement adds guidance for edge cases, while excessive complexity can reduce clarity.

6 Observations

  • Prompt design was evaluated across general and targeted approaches, with attention to accuracy, toxicity handling, and adherence to specified formats.The analysis considered prompt length, clarity, model-specific instructions, and gradual refinement.Source: §6 · p. 6, §6.1 · p. 6

6.1.1 Prompt Length and Clarity.

  • Prompt9 and Prompt7, which used Mixtral-specific [INST] and <s> tags, achieved correctness percentages of 73% and 71%, respectively.The authors report that these model-specific features significantly improve comprehension and accuracy.Source: §6.1.3 · p. 6

6.1.3 Incremental Improvements through Additional Instructions.

  • Prompt1 and Prompt2 scored 48% and 44% overall correctness, while Prompt10 and Prompt11 performed better by explicitly addressing edge cases such as toxicity.The results suggest that adding detail can improve accuracy, but excessive complexity may reduce clarity.Source: §6.1.3 · p. 6, §6.1.1 · p. 6
  • Each prompt iteration added complexity or specificity to address more scenarios, and incremental structure and guidance helped the model handle core tasks and edge cases.The authors characterize prompt engineering as stepwise optimization rather than trial and error alone.Source: §6.1.3 · p. 6

6.1.1 Prompt Length and Clarity.

  • 73% accuracy: Prompt9 and Prompt11 achieved the highest scores while balancing detailed instructions with clarity.Both included explicit instructions for toxic content, which the authors say likely contributed to strong performance in ethically sensitive areas.Source: §6.1.1 · p. 6, Table 1 · p. 7

6.2 Objective-Specific Prompt Performance

Objective-specific evaluation shows that aggregate accuracy can mask category and toxicity differences; format adherence during toxic-query handling is another objective the experiments did not specifically measure.

6.2 Objective-Specific Prompt Performance

  • Objective-specific performance can reveal prompt shortcomings that overall accuracy alone misses, including handling edge cases such as toxicity and hallucination.Figure 4 is presented as illustrating this objective-specific evaluation.Source: §6.2 · p. 7

6.2.1 Category-Specific Accuracy

  • Prompt6 reached 69% overall accuracy and 90.3% on toxicity, but its accuracy was 52.5% in markets and 53.3% in politics.The authors note that optimizing for one category can compromise results in others.Source: §6.2.1 · p. 7
  • Prompts 4 and 10 explicitly instructed the model to avoid toxic queries and achieved high accuracy on those scenarios.The authors emphasize balancing category performance with ethical and safety standards.Source: §6.2.2 · p. 7

6.2.2 Handling Toxicity as an Additional Objective

  • The model sometimes failed to maintain prescribed output formats on toxic queries, and format adherence was not specifically measured in these experiments.The authors identify stricter prompt design as needed.Source: §6.2.2 · p. 7

6.2 Objective-Specific Prompt Performance

  • Prompt9 and Prompt11 each reached 73% overall accuracy, yet their politics accuracy was 65.8% and 62.5%, respectively.Prompt9 led in tech (78.3%) and sports (77.5%); Prompt11 reached 82.5% in tech and 96.3% on the toxicity objective.Source: Figure 4 · p. 7, §6.2.1 · p. 7

6.2.2 Handling Toxicity as an Additional Objective

  • More explicit and comprehensive prompts are presented as a way to balance category accuracy and toxicity management while enforcing reliable output structures.The paper frames these as distinct objectives to address together.Source: §6.2.2 · p. 7
  • A prompt that performs well in one area but misses other critical objectives may not be suitable for practical use.The paper names accuracy, ethical considerations, and strict output formats as objectives real-world applications may require.Source: §6.2.3 · p. 7

6.2.3 Multi-Objective Prompting

  • The framework incorporates prompt length, clarity, category-specific performance, and ethical handling into multi-objective prompt optimization.The authors describe this as supporting robust, flexible prompts across tasks and domains.Source: §6.2.3 · p. 7

6.3 Scaling from In-Sample to Out-of-Sample Data

The authors tested prompts on a smaller subset before applying them to the remaining data. Accuracy trends persisted in the larger sample, supporting the use of representative subsets for prompt development.

6.3 Scaling from In-Sample to Out-of-Sample Data

  • Accuracy trends from the initial ~20% sample persisted on the remaining 80% of the dataset, which the authors interpret as evidence that the subset was sufficiently representative.The same prompts were evaluated on the larger dataset after preliminary testing.Source: §6.3 · pp. 7–8
  • The authors propose testing and refining prompts on a focused subset, then extending validated findings to larger or domain-specific datasets to reduce computation and iteration time.They describe this sample-driven process as supporting scalability while maintaining accuracy and consistency.Source: §6.3 · p. 8
  • The preliminary evaluation on approximately 20% of the data indicated how prompts performed across multiple categories and objectives.Figure 5 presents prompts versus accuracy.Source: §6.3 · pp. 7–8, Figure 5 · p. 8

7 Enterprise Application of MODP

MODP was applied in Dell's Next Best Action support tool, including during a model migration. In evaluation, MODP prompts improved task-specific objectives by an average of 8%.

7 Enterprise Application of MODP

  • Dell uses MODP to develop, refine, and compare prompts in its Next Best Action tool, which assists agents and customers with product troubleshooting.The framework also facilitated quicker prompt development during an LLM migration with minimal impact on performance.Source: §7 · p. 8

7.1 Performance Metrics

  • Table 2 compares GPT, the premigration model, and target Mixtral using baseline prompts and prompts developed with MODP.The evaluation considered both task-specific and LLM-specific objectives.Source: §7.1 · p. 8, Table 2 · p. 8
  • MODP prompts produced an average 8% gain across task-specific objectives: faithfulness, answer relevancy, and precision.Response adherence and answer consistency were human-evaluated using real-time feedback, and the claim was further validated with SME validations.Source: Table 2 · p. 8, §7.1 · p. 8
  • The reported evaluation results reinforce MODP's practical impact through its successful deployment in the Next Best Action support tool.Source: §7.1 · p. 8

8 Conclusion

The conclusion presents MODP as a multi-objective, metrics-driven approach to prompt engineering, reporting a 26% improvement from basic prompts on a sample dataset. It also identifies challenges in defining objectives and optimizing prompts as their number grows.

8 Conclusion

  • The framework yielded a 26% improvement from basic prompts on a sample dataset while incorporating intrinsic LLM behavior into prompt development.The authors describe the approach as using metrics to develop robust prompts and help test performance translate to production data.Source: §8 · p. 8
  • As objectives grow, single-prompt optimization becomes more complex; quantifying objectives can be difficult, human evaluation can increase development time, and identifying LLM-specific objectives may require prior model experience.Source: §8 · p. 8
  • The authors identify multi-agent systems, long-context LLMs, advanced reasoning models, and optimal stopping criteria as directions for future work.They note Pareto front-based methods [15] as one possible approach to stopping criteria, while acknowledging iterative objective evaluations.Source: §8 · p. 8

A Appendix: Prompt used in Section 3

The appendix lists prompts for answering passage-derived fill-in-the-blank questions, with variants that add toxicity screening and specify answer formatting.

A Appendix: Prompt used in Section 3

  • The listed prompts ask the model to replace “@placeholder” in a question using the supplied passage.Several request only the direct answer, without explanation or commentary.Source: A Appendix: Prompt used in Section 3 · p. 10
  • Some prompt variants instruct the model to refuse hateful, offensive, toxic, unethical, or harmful content with a fixed response.The refusal wording and the content categories vary across prompts.Source: A Appendix: Prompt used in Section 3 · p. 10
  • The appendix includes both plain-text and [INST]-formatted prompts, with some variants adding <s> and </s> markers.One prompt also frames the model as an expert in filling blanks from passage-derived questions.Source: A Appendix: Prompt used in Section 3 · p. 10

Related papers

Loading 2504.18722v1…