PaperlayerView exact revision on arXiv

Source-linked AI summary

Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison

Yoonho Lee, Joseph Boen, Chelsea Finn

arXiv:2511.07919v2cs.LG

TL;DR

Existing reward and preference signals can omit the reasons behind judgments and how to improve. Feedback Descent uses textual feedback to guide inference-time edits to text artifacts; it reports competitive or superior outcomes across three domains, including high-percentile molecules on DOCKSTRING.

  • Problem

    Scalar rewards and pairwise preferences provide at most a single bit per comparison, leaving out why one behavior is better and how to improve.

    Source: §1 · p. 1

  • Method

    Feedback Descent iteratively proposes text-artifact edits conditioned on accumulated feedback, compares each candidate with the current best, and makes no weight updates.

    Source: §1 · p. 1

  • Results

    Across visual design, prompt optimization, and molecule design, Feedback Descent is competitive with GEPA, outperforms GRPO and specialized molecular optimizers, and reaches or exceeds the 99.9th percentile on multiple DOCKSTRING targets.

    Source: §1 · p. 2, §4.4 · pp. 8–9

  • Takeaways & Limitations

    The paper concludes that text can serve as an optimizable medium for continual improvement without retraining.

    Source: §5 · p. 10

    The method relies on strong evaluators, which may be scarce, and harder tasks require reliable feedback.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce \textit{Feedback Descent}, a framework that optimizes text artifacts -- prompts, code, and molecules -- through structured textual feedback, rather than relying solely on scalar rewards. By preserving detailed critiques instead of compressing them to binary preferences, Feedback Descent widens the information bottleneck in preference learning, enabling directed optimization in text space rather than weight space. We show that in-context learning can transform structured feedback into gradient-like directional information, enabling targeted edits. Unlike prior approaches that collapse judgments into single bits, our evaluators pair each comparison with textual feedback, which functions as high-bandwidth supervision. The iteration loop is done purely at inference time, without modifying any model weights, and is task-agnostic. We evaluate Feedback Descent on three diverse domains and find that it outperforms state-of-the-art prompt optimization (GEPA), reinforcement learning methods (GRPO, REINVENT), and even specialized graph-based molecular optimizers. In the DOCKSTRING molecule discovery benchmark, Feedback Descent identifies novel drug-like molecules surpassing the $99.9$th percentile of a database with more than $260{,}000$ compounds across six protein targets.

1. Introduction

Feedback Descent uses textual critiques alongside pairwise preferences to optimize text artifacts at inference time. The paper presents it as a general framework and reports results across three domains.

Motivation

  • Scalar rewards and pairwise preferences convey at most a single bit per comparison, discarding why one behavior is better and how to improve.The introduction identifies this as an information bottleneck in existing reinforcement-learning supervision.Source: §1 · p. 1

Approach

  • At inference time, a language model uses accumulated feedback to make targeted edits to text artifacts without updating model weights.Each candidate is compared with the current best, and the evaluator supplies both a preference and textual feedback.Source: §1 · p. 1, Figure 1 · p. 2

Contributions

  • Feedback Descent works across visual design, prompt optimization, and molecule design, with competitive or superior results against specialized methods.The authors report competitive performance with GEPA, outperforming GRPO and specialized molecular optimizers, and molecules above the 99.9th percentile of a 260,000-compound database.Source: §1 · p. 2

2. Feedback Descent: Open-Ended Text Optimization

Feedback Descent optimizes text-representable artifacts by turning comparative feedback into directed semantic edits. Its iterative loop uses accumulated critiques as heuristic directional cues, not literal gradients.

2.1. Problem Setup

  • Feedback Descent targets text-representable artifacts whose quality is easier to judge than to construct.The paper uses SVG code for a unicorn as a running example: evaluators can compare renderings and explain their choice even when writing high-quality SVG from scratch is difficult.Source: §2 · p. 2

2.2. Feedback Descent

  • The loop maintains a best artifact, proposes candidates using the accumulated feedback, and compares each candidate with the current best.The language model is prompted to address prior critiques while preserving successful elements; feedback is retained as a rationale history.Source: §2.2 · p. 3

2.3. Analogy to Gradient Descent

  • Textual feedback is an approximate, sometimes contradictory directional cue, so the method lacks gradient descent's convergence guarantees.The paper describes it as higher-bandwidth supervision than a binary preference or scalar reward.Source: §2.3 · p. 3
  • Under idealized assumptions, rationale-guided updates achieve linear convergence independent of effective dimensionality, while zeroth-order baselines scale exponentially worse.The authors say these theoretical results motivate the approach, rather than provide rigorous guarantees for the discrete text domains studied empirically.Source: §2.3 · pp. 3–4

3. Related Work

The related work spans preference learning, gradient-free optimization, compound AI systems, and inference-time improvement. Feedback Descent's distinguishing feature is its use of accumulated external textual feedback to guide optimization.

Preference Learning

  • Preference-learning methods compress judgments into binary or scalar preferences, whereas Feedback Descent uses detailed textual feedback at inference time without retraining.Prior work has also used rationales to improve reward modeling or weight-based training.Source: §3 · p. 4

Evolutionary Algorithms and Gradient-Free Optimization

  • Feedback Descent resembles evolutionary optimization but operationalizes directed mutations through accumulated textual feedback.The passage contrasts its approach with gradient-free methods' dimension-related challenges and notes prior LLM molecule optimization work [71].Source: §3 · p. 4

Optimizing Compound AI Systems

  • Unlike TextGrad's pointwise updates based only on the latest artifact, Feedback Descent keeps a trajectory-level buffer of comparative feedback.The paper says this history makes Feedback Descent more scalable at long-horizon optimization than TextGrad [89].Source: §3 · p. 4

Inference-Time Optimization for LLMs

  • After five iterations, Feedback Descent matches or exceeds direct prompting's win rate in every tested From Scratch and rubric-aware condition.The table caption states all combinations are at least 50%.Source: Table 1 · p. 5

4. Experiments

The experiments evaluate Feedback Descent on visual design, prompt optimization, and molecule discovery using distinct representations and evaluation methods. It improves SVG outputs, remains competitive with specialized prompt optimization, and achieves strong molecular optimization results across six targets.

4.1. Experimental Domains

  • The evaluation spans SVG design judged by vision-language models, prompt optimization scored on benchmark accuracy, and molecular discovery evaluated with docking scores.Molecules are represented as SMILES and assessed across six protein targets; the prompt tasks use held-out test examples.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2. SVG Optimization

  • After five iterations, Feedback Descent reliably improves SVG outputs over the initial population for both GPT-4o-mini and GPT-5-mini.Qualitative examples show styles diverging across judges in line with their aesthetic criteria.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3. Prompt Optimization

  • Feedback Descent is competitive with GEPA across both models, leading on IFBench and Hover while GEPA leads on HotpotQA and PUPA.The approach jointly optimizes prompts using textual summaries of pairwise performance differences and requires minimal domain-specific engineering.Source: §4.3 · p. 7, Table 2 · p. 7

4.4. Molecule Optimization (DOCKSTRING)

  • Feedback Descent outperforms all baselines across six protein targets, matching or exceeding the 99.9th and 99.99th DOCKSTRING percentiles and sometimes surpassing its best molecule.The results cover a database of 260155 compounds.Source: §4.4 · p. 8, Table 3 · p. 8
  • Feedback Descent reaches high-scoring molecular regions with competitive trajectories and comparable or fewer oracle calls, a pattern reported across targets.For PPARG, its molecules sit on or above the DOCKSTRING frontier for docking affinity and drug-likeness.Source: §4.4 · p. 9, Figure 4 · p. 9, Figure 5 · p. 9
  • Across targets, high-scoring Feedback Descent molecules have weak or negative similarity correlations with approved drugs, indicating exploration of novel chemical regions.Reported Spearman correlations range from −0.39 to 0.40; CDK2 is excluded from this comparison under the stated inclusion criteria.Source: §4.4 · p. 9, Figure 6 · p. 10
  • Textual rationales outperform binary-only feedback by 1.74 docking score units, while randomly shuffled rationales underperform best-of-N sampling.With 50% feedback noise, scores remain above the 99.9th DOCKSTRING percentile for ADRB1 and PGR.Source: Table 4 · p. 8, §4.4 · p. 9
  • In 400 comparisons, an LLM judge preferred molecules aligned with true feedback over scrambled feedback 81% of the time.The reported binomial-test result is p<10^-10.Source: Table 5 · p. 9, §4.4 · p. 10

5. Discussion

Feedback Descent improves text artifacts through structured pairwise feedback across visual design, prompt optimization, and molecule discovery, without retraining. Its use depends on reliable evaluators, and creative tasks may require balancing refinement with exploration.

5. Discussion

  • The inference-time framework improves text artifacts through structured pairwise feedback across three domains without requiring retraining.The paper presents text as an optimizable medium rather than static data.Source: §5 · p. 10
  • The method relies on strong evaluators, which may be scarce in some domains, and reliable feedback remains a prerequisite for harder tasks.For creative domains, strictly following the gradient may be limiting; balancing refinement with exploration is a next step.Source: §5 · p. 10

A. Formal Statements and Proofs

The appendix formalizes rationale-guided convergence under smoothness and PL assumptions, then contrasts full-gradient and coordinate-sparse query regimes. Full-gradient rationales give dimension-independent query complexity, while coordinate queries require averaging to recover the stated iteration rate.

A. Formal Statements and Proofs

  • Proposition 1 states linear convergence under the PL condition for rationale-guided directions, with a smooth objective.The stated setup also assumes the projection is inactive when a constraint set is present.Source: A. Formal Statements and Proofs · p. 18
  • The proposition specifies step size η=α/(L(α^2+σ^2)) for the update under its stated assumptions.Source: A. Formal Statements and Proofs · p. 18

A.1. Query Complexity and Dimension Dependence

  • When rationales provide full gradient information at unit cost, query complexity equals T and is dimension-independent.Source: §A.1 · p. 18
  • With one uniformly random coordinate per query, the stated estimator has α=1, σ^2=d−1, κ1=d, and step size η=1/(Ld).Source: §A.1 · p. 18
  • Averaging m independent coordinate queries per iteration reduces the stated variance to σ^2=(d−1)/m.Source: §A.1 · p. 18
  • Averaging d coordinate queries recovers T=O((L/μ) log(1/ε)) iterations at d queries per iteration, so total queries remain Θ.The supplied passage does not complete the expression after Θ.Source: §A.1 · p. 18

Unrolling yields

The derivation conditions on the current iterate and chooses a step size that makes the stated bracket yield the claim.

Unrolling yields

B. Lower Bounds for Exhaustive/Random Zeroth-Order Search

The section formalizes the slowness of zeroth-order grid and random search without directional information. On the stated family, exhaustive grid search is exponential in dimension and polynomial in 1/ε with exponent d/2.

B. Lower Bounds for Exhaustive/Random Zeroth-Order Search

C.1. Implementation Details

The implementation adapts Feedback Descent to visual design, prompt optimization, and molecule optimization, with domain-specific evaluators and feedback. Prompt comparisons additionally use a statistical analysis to identify actionable patterns.

C.1. Implementation Details

  • SVG designs are compared in rendered form, and the champion updates only when both comparison orderings agree on the winner.Winning rationales are added to the generation prompt to guide later designs.Source: §C.1 · p. 19
  • IFBench prompt optimization uses Qwen3-8B and GPT-4.1-mini with different iteration, stopping, and temperature settings.Qwen3-8B runs 10 iterations without early stopping at temperature 0.6; GPT-4.1-mini runs up to 15 iterations with patience 5 at temperature 1.0.Source: §C.1 · p. 20
  • Prompt comparisons are partitioned into four outcome groups, then tested for significant hypotheses using Fisher’s exact test, minimum support, and lift thresholds.The analysis generates roughly 20 hypotheses per group and retains significant findings at p<0.1 with support of at least 3 examples and lift thresholds of 2.0 for wins or 1.5 for failures.Source: §C.1 · p. 20
  • Molecule optimization evolves SMILES from acetamide, pentane, and benzene using DOCKSTRING across six protein targets.Feedback incorporates RDKit properties, binding-site information, and similarities to approved drugs; the implementation uses batch size 8 and top-k selection of 10.Source: §C.1 · p. 20

C.2. Additional Results

Across six protein targets, discovered molecules extend beyond the DOCKSTRING baseline in docking affinity and drug-likeness. The accompanying prompt templates specify how candidate prompts and molecules are generated and how task-specific judgments are operationalized.

C.2. Additional Results

  • Across all protein targets, discovered molecules extend beyond the DOCKSTRING baseline on both docking affinity and drug-likeness.Figure 7 depicts the joint trade-off through Pareto frontiers, with highlighted discovered molecules improving both Vina score and QED.Source: §C.2 · p. 20, Figure 7 · p. 21

C.3. Prompt Templates

  • The paper says rubrics for other tasks follow a similar style, translating each aesthetic into operational rules intended to minimize ambiguity.The Anatomy SVG rubric is presented as an example.Source: §C.3 · p. 20
  • The Anatomy SVG rubric prioritizes equine proportions, connected limbs, anatomical structure, and natural horn placement, while penalizing fused limbs and implausible joints.When both images are plausible, the rubric favors better weight and grounding.Source: §C.3 · pp. 21–22

C.2. Additional Results

  • Table 6 reports full molecule-optimization results with standard deviations for six targets and identifies fragment-based methods as having structural priors unavailable to text-only methods.Its caption describes how top generative results and database populations exceeding the best generative result are marked.Source: Table 6 · p. 22

C.3. Prompt Templates

  • The prompt-optimization templates include candidate-generation and rationale-generation prompts, with sections for training signals and an improvement strategy.The system prompt asks the model to extract actionable insights from the data.Source: §C.3 · pp. 22–23
  • The prompt-improvement template asks for generalization beyond a small training sample and extraction of evidence-backed patterns for actionable edits.It directs attention to strategies, recurring errors, and essential facts or constraints.Source: §C.3 · pp. 22–23
  • Prompt instructions favor concise, precise guidance, avoid redundant or over-specified additions, and preserve effective simple baseline instructions.Added complexity should provide measurable value.Source: §C.3 · p. 23
  • The molecule-generation template requests one SMILES string and a chemical hypothesis linking structural changes to expected binding and QED improvements.Its task explanation frames the benchmark as maximizing binding and drug-likeness.Source: §C.3 · pp. 23–24

Related papers

Loading 2511.07919v2…