PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflective Prompt Tuning through Language Model Function-Calling

Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour, Estevam Hruschka

arXiv:2605.21781v1cs.CL

TL;DR

Prompt design is labor-intensive and sensitive to superficial choices, while existing optimizers have limited mechanisms for capturing recurring failures and revision history. RPT uses function-calling diagnostics and accumulated memory to revise prompts, and across three reasoning tasks improves seed prompts by up to +12.9 points while improving calibration. Its scope is limited by evaluation on three reasoning tasks and frontier proprietary models, with effectiveness potentially differing for other models or settings.

  • Problem

    Prompt design is labor-intensive and sensitive to formatting, phrasing, and instruction ordering, while existing methods have limited feedback and memory for systematic failure-driven revisions.

  • Method

    RPT uses an LLM function-calling optimizer that generates structured diagnostics of recurring failure modes and combines them with memory of prior reports and prompt revisions.

  • Results

    Across three reasoning tasks, RPT improves over seed prompts by up to +12.4 points on LiveBench-Math and remains competitive with state-of-the-art baselines, while confidence-aware optimization improves calibration alongside task performance.

  • Takeaways & Limitations

    RPT is especially effective for recurring reasoning failures and produces targeted revisions aligned with diagnosed failures, supporting scalable and interpretable prompt tuning.

  • Takeaways & Limitations

    The evaluation covers three reasoning tasks using GPT-4.1 targets and frontier proprietary optimizers, so effectiveness may differ for other models or settings without function calling.

Abstract

from arXiv · show

Large language models (LLMs) have become increasingly capable of following instructions and complex reasoning, making prompting a flexible interface for adapting models without parameter updates. Yet prompt design remains labor-intensive and highly sensitive to formatting, phrasing, and instruction order, motivating automated prompt optimization methods that reduce manual effort while preserving inference-time flexibility. However, existing methods often search over prompt candidates or use fixed critique-refine pipelines driven by individual examples or small batches, limiting their ability to capture systematic error patterns and make targeted edits grounded in failure history. We propose Reflective Prompt Tuning (RPT), a framework that uses LLM function calling to simulate the iterative workflow of human prompt engineers. An LLM optimizer calls a diagnostic function that evaluates the target model over an entire optimization set, summarizes recurring failure modes, and returns a structured diagnostic report. The optimizer uses this report, together with an accumulated memory of prior reports, to revise the prompt for the next iteration. RPT further supports confidence-aware optimization by using calibration signals in diagnostic feedback and final prompt selection. Across three reasoning tasks, RPT improves over initial prompts by up to 12.9 points, remains competitive with state of the art, and improves confidence calibration. Our analyses show that RPT is especially effective on multi-hop and mathematical reasoning, producing targeted prompt revisions that align with diagnosed failure patterns and lead to gains in task performance and calibration.

1 Introduction

Prompting offers flexible adaptation without parameter updates, but prompt design remains manual and sensitive to wording and structure. RPT addresses these challenges with function-calling diagnostics, memory over prior revisions, and confidence-aware selection, improving performance and calibration across reasoning tasks.

  • Prompting adapts model behavior by specifying objectives, constraints, and output formats without modifying model parameters.
  • Manual trial-and-error, expertise requirements, and sensitivity to formatting, phrasing, and instruction order make prompt effectiveness unreliable across settings.
  • Existing prompt optimizers often use fixed context-updating pipelines and lack explicit memory over prior diagnostic reports and revisions, limiting cross-iteration credit assignment.
  • RPT evaluates confidence signals during diagnostic feedback and final prompt selection, extending optimization beyond task performance alone.
  • RPT uses an LLM optimizer that calls a diagnostic function to evaluate the target model, cluster recurring failure modes, and revise prompts using accumulated feedback.
  • Across three reasoning tasks, RPT improves over initial prompts by up to +12.9 points and remains competitive with state-of-the-art automated prompt-optimization baselines.

2 Reflective Prompt Tuning (RPT)

Reflective Prompt Tuning (RPT) frames prompt optimization as an iterative, diagnosis-driven process: an optimizer evaluates model behavior, identifies recurring failures, and revises prompts using structured feedback and history. It combines task-performance and calibration signals for prompt selection while retaining representative failure evidence for targeted revisions.

  • 2 Reflective Prompt Tuning (RPT): RPT automates the prompt-engineering workflow by iteratively running a prompt, inspecting outputs, diagnosing recurring failures, revising the prompt, and repeating.The framework uses LLM function calling to support this diagnosis-driven optimization process.
  • 2.1 Problem Statement: RPT selects among candidate prompts using development-set metrics that include task performance and confidence calibration error.The scalar selection function combines these metrics; prompt length alone is not a reliable indicator of development performance.
  • 2.2.2 Reflective Prompt Revision with Memory: RPT’s explicit diagnostic function and memory address limitations of fixed or performance-driven optimization by grounding revisions in systematic failures and optimization history.The approach is designed to support targeted prompt changes rather than treating each update as an independent proposal.
  • 2.2 Methodology Overview: At each iteration, a diagnostic function evaluates the current prompt on the full optimization set and returns aggregate metrics together with recurring failure information.The report captures both prompt quality and how failures recur across the dataset.
  • 2.2.1 Constructing Diagnostic Feedback: The diagnostic function records reasoning traces, final answers, and confidence, computes task and calibration metrics, identifies failed examples, and generates response-level critiques.Critiques assess reasoning, evidence use, formatting, and whether reported confidence matches response correctness and quality.
  • 2.2.1 Constructing Diagnostic Feedback: RPT aggregates up to three diagnoses per failed instance into recurring failure topics containing labels, descriptions, and representative examples.ClusterFusion compresses local critiques into a focused summary of systematic failures, with the number of topics controlling summary granularity.
  • 2.2.1 Constructing Diagnostic Feedback: The resulting structured report converts scalar evaluation into dataset-level diagnosis that helps the optimizer infer prompt shortcomings and propose targeted revisions.The report retains a focused subset of prominent recurring clusters rather than every diagnosis.
  • 2.2.2 Reflective Prompt Revision with Memory: The optimizer generates the next prompt from the current prompt, diagnostic report, and external memory of prior reports and revisions.This history lets it track persistent failures, prior edits, and performance changes when individual edits affect metrics differently.

3 Experimental Setup

RPT is evaluated on three reasoning tasks using GPT-4.1 as the target model and compared with ACE, GEPA, and MIPRO. Evaluation reports task performance and verbalized-confidence calibration.

  • RPT is evaluated on HotPotQA, LiveBench-Math, and Formula, covering multi-hop, mathematical, and domain-specific numerical reasoning.
  • GPT-4.1 serves as the target model, while RPT uses GPT-5, GPT-5-mini, Gemini-3.1-Pro, and Gemini-3.1-Flash-Lite as optimizer LLMs.
  • RPT is compared with the state-of-the-art automated prompt optimizers ACE, GEPA, and MIPRO.
  • Table 1 reports initial and optimized-prompt performance for each task and prompt-optimization method.
  • Task metrics are accuracy for HotPotQA and Formula and task score for LiveBench-Math, with Brier score measuring calibration from verbalized confidence.

4 Results and Analyses

Across three reasoning tasks, RPT improves seed prompts and performs especially strongly on recurring reasoning failures, while confidence-aware optimization improves calibration. Trace analyses link diagnosed failures to targeted patches and identify task-dependent associations between patches and later gains.

  • 4.1 RPT Is Competitive with SOTA Baselines: RPT achieves its best results on recurring reasoning failures, including the best LiveBench-Math performance across optimizer settings and gains of up to +12.4 points.
  • 4.1 RPT Is Competitive with SOTA Baselines: GPT-5 raises RPT’s Aggregate score from 68.5 to 74.3, while Gemini-3.1-Pro raises it from 67.7 to 70.1 relative to smaller family counterparts.
  • 4.2 Confidence Signals Improve Calibration: Confidence-aware GEPA improves both task performance and Brier score on HotPotQA, but gains are limited on LiveBench-Math and Formula.
  • 4.2 Confidence Signals Improve Calibration: RPT incorporates calibration into diagnostic feedback and prompt selection, improving alignment between self-reported confidence and empirical correctness alongside task performance.
  • 4.3.1 Does RPT Produce Targeted Revisions?: Failure-to-patch alignment varies by task: HotPotQA and LiveBench-Math show failure-specific revisions, whereas Formula uses broader domain safeguards and weaker fine-grained credit assignment.
  • 4.3.2 Do Prompt Patches Predict Gains?: Useful patches differ by task but commonly impose concrete controls on reasoning or output; Formula shows mixed associations for specialized safeguards.

5 Related Work

Related work spans automated prompt search, reflective textual-feedback methods, and memory-based adaptive context. RPT differs by combining full-split diagnosis, recurring-failure aggregation, revision memory, and direct calibration-aware selection.

  • Automated prompt optimization: Automated prompt optimization methods search trigger tokens, generate natural-language candidates, or revise prompts through textual gradients and evolutionary search.
  • Reflective optimization methods: Reflective methods use execution traces, evaluation feedback, or program-aware strategies to guide prompt or program optimization.
  • Reflective optimization methods: RPT evaluates the current prompt over the full optimization split, aggregates recurring failures, and uses prior reports to guide subsequent revisions.
  • Reflective optimization methods: Unlike methods that treat calibration as auxiliary feedback, RPT incorporates confidence and calibration directly into diagnostic reports and final prompt selection.
  • Memory and adaptive context: Memory-based methods accumulate reflections, strategies, or playbooks to improve later behavior or context construction, while RPT applies memory to prompt optimization.

Conclusion

RPT uses function calling, structured diagnosis, and revision memory to optimize prompts across reasoning tasks. It improves seed prompts, calibration, and failure-aligned revisions, especially for multi-hop and mathematical reasoning.

  • RPT is a diagnosis-driven framework that uses LLM function calling, structured feedback, and memory over prior revisions.
  • Across three reasoning tasks, RPT improves over seed prompts and remains competitive with state-of-the-art methods, especially on multi-hop and mathematical reasoning.
  • Confidence-aware optimization improves calibration alongside task performance, while optimization traces show revisions aligned with diagnosed failures.

Limitations

RPT’s evaluation is limited to three reasoning tasks and specific model configurations, while its computational cost and reliance on prompting and verbalized confidence constrain broader use.

  • Evaluation covers only multi-hop question answering, mathematical reasoning, and domain-specific numerical reasoning, not open-ended generation, coding, dialogue, agents, or long-horizon interaction.The experiments use GPT-4.1 as the target model and frontier proprietary LLMs as optimizers.
  • RPT’s effectiveness may differ for smaller open-source models, weaker optimizer LMs, or settings where function calling is unavailable.
  • Each iteration evaluates the full optimization set, critiques failures, clusters diagnoses, and conditions revision on prior reports, making RPT more computationally expensive than small-batch optimizers.
  • Scaling to much larger datasets or longer optimization trajectories may require more aggressive sampling, report compression, or memory retrieval.
  • Prompting alone cannot guarantee resolution of all failures, especially deeper mathematical or domain-specific convention errors that may require complementary interventions.
  • Confidence-aware optimization relies on verbalized confidence as a blackbox uncertainty proxy that should be validated carefully before high-stakes use.

7 RPT Prompts

The RPT prompts define task-specific seed instructions and structured JSON outputs, while critic prompts diagnose actionable, clusterable failure modes from model traces.

  • 7.1 Seed Prompts: Seed prompts require context-grounded reasoning, concise justification, answer verification, and valid JSON output.
  • 7.1 Seed Prompts: HotPotQA seed outputs contain justification, answer, and confidence fields, with confidence represented as a number in [0,1].
  • 7.1 Seed Prompts: LiveBench-Math prompts require step-by-step mathematical reasoning, exact requested answer formatting, confidence, and valid JSON.
  • 7.1 Seed Prompts: The math seed prompt additionally instructs the model to be concise, double-check calculations and logic, and return the required schema.
  • 7.2 Critic Prompts: Critics receive QA or math traces and produce 1-3 failure modes with labels, definitions, example-specific explanations, and evidence bases.
  • 7.2 Critic Prompts: The QA critic is explicitly given context consisting of titles and snippets for diagnosing failures.
  • 7.2 Critic Prompts: Failure-mode labels must be concrete, consistent, short, and clusterable rather than containing example-specific entities or dates.
  • 7.2 Critic Prompts: Critics focus on actionable diagnoses, return an empty list when no clear failure mode exists, and output only schema-valid JSON.

7.3 Shared Optimizer Prompt

The shared optimizer prompt enforces an evaluate-then-decide loop, prioritizes task performance with calibration as a secondary objective, and directs targeted prompt maintenance.

  • At each iteration, the controller must call evaluate_prompt once on the current PromptProgram, read its report, and then output PATCH or STOP.
  • Task performance on the training split is primary, while calibration is secondary and should improve without hurting task performance.
  • When available, current_summary metrics and deltas against previous or best results guide decisions, while history detects trajectories, regressions, and ineffective edits.
  • Edits should target diagnosed failure modes with concrete checks, comparisons, extractions, or verifications rather than vague restatements.
  • The optimizer should revise, merge, delete, or reorganize existing instructions, avoid redundancy and conflict, and keep the JSON output contract stable.
  • The controller should output STOP when training performance plateaus or further edits are unlikely to help, but only after evaluation.

7.4 Optimization Datasets

RPT is studied across three reasoning datasets, with full-set diagnostic analyses of failure persistence and actionability alongside prompt-length and performance trends.

  • 7.4 Optimization Datasets: HotPotQA evaluates multi-hop question answering that combines evidence across supporting passages to produce concise answers.
  • 7.4 Optimization Datasets: LiveBench-Math evaluates mathematical reasoning involving problem decomposition, intermediate computation, and final-answer generation.The study uses the 2024-08-31 release and splits 368 questions into train, development, and test sets.
  • 7.4 Optimization Datasets: Formula evaluates domain-specific numerical reasoning over structured XBRL financial data, requiring financial concepts and computation.
  • RPT Configuration: Each diagnostic iteration evaluates the current prompt on the optimization split, critiques incorrect responses, clusters diagnoses, and summarizes recurring failure modes with aggregate metrics.The optimizer uses the report and prior-report memory to revise or stop.
  • Failure-Mode Persistence: Failure persistence is measured by average run length, the number of consecutive iterations a topic remains active; longer runs indicate failures persisting despite revisions.
  • Failure-Mode Persistence: Persistent failures are task-specific and concentrate on deeper reasoning, computation, definitions, conventions, and domain constraints rather than generic formatting.
  • Actionability of Diagnosed Failure Modes: Actionability analysis measures next-iteration changes in task score and Brier score, but the association is not causal.
  • Prompt Length and Performance: Prompt length generally grows with added constraints, yet development performance is non-monotonic, motivating final selection by development performance rather than recency.

7.9 Example Prompt Revision

RPT transforms a general HotPotQA prompt into a targeted instruction set by using diagnostic feedback to address recurring extraction, reasoning, disambiguation, and calibration errors. Prompt length generally grows across iterations while development performance improves early before plateauing or fluctuating, motivating development-set selection.

  • Optimized prompt: RPT revises the prompt with explicit controls for question parsing, minimal span extraction, surface-form matching, multi-hop tracing, and confidence calibration under ambiguity.These controls target recurring failure patterns identified through diagnostic feedback.
  • Iteration behavior: Prompt length generally increases across RPT iterations, whereas development performance improves early and then plateaus or fluctuates, supporting development-set selection over defaulting to the final iteration.The comparison covers HotPotQA, LiveBench-Math, and Formula.
  • Original prompt: The original HotPotQA prompt asks for context-based reasoning, support verification, concise justification, and JSON output.Its output schema includes justification, answer, and confidence fields.
  • Optimized prompt: The optimized instructions require matching requested answer type, granularity, units, qualifiers, and canonical or surface forms without extraneous text.They also specify exact handling for numerical answers, dates, place or organization names, and yes/no responses.
  • Optimized prompt: For compositional questions, the prompt requires tracing all hops, verifying subject–object direction, applying temporal or comparative constraints, and returning the final answer span.A final checklist verifies type, constraints, surface form, and minimal contiguous formatting.
  • Optimized prompt: Confidence calibration maps unambiguous support to 0.85 and lower or capped confidence to ambiguity, weaker inference, or contradiction.The prompt also limits justifications to three sentences and requires confidence to reflect evidence and ambiguity.
Loading 2605.21781v1…