Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures

Introduction

• Boundary failures are coherent near misses that violate the given prompt but satisfy a related intent or constraint setting.Examples include responses that miss format, language, lexical, or structural requirements despite solving the task.Source: Introduction · p. 1
• Forward-only supervision records that a boundary-failure response lost under its original prompt, but not whether that rejection is prompt-specific.Because prompts share model parameters, repeatedly rejecting a coherent response mode may influence its relative score in other contexts where it remains useful.Source: Introduction · p. 1
• BPS keeps the conventional correction pair under the original prompt and adds a validated reverse pair under an achieved prompt, without changing the DPO objective.The same response is rejected where it is wrong and preferred where it is right.Source: Introduction · pp. 1–2
• Held-out crossed-anchor probes on Qwen3-4B-Instruct-2507 show BPS preserves original-side rankings while improving achieved-side rankings under Gemini and Kimi-K2.6 construction.A blind human audit supports the intended reverse preference direction and finds little response leakage.Source: Introduction · pp. 1–2

Related Work

• BPS is related to reverse-pair and input-side preference methods, but ties its reverse label to an observed failure, a synthesized achieved prompt, and validation.It also preserves rejection under the original prompt rather than replacing failure with success, unlike the described hindsight relabeling connection.Source: Related Work · p. 2

Problem Formulation

• BPS focuses on partial successes, with 0 < r < 1, especially near misses with r ≥ 0.5; fully successful and fully failed responses are not its reverse-anchor target.The pass ratio measures the share of prompt constraints satisfied, and reverse anchors require eligibility and validation.Source: Problem Formulation · pp. 2–3

Forward Execution Anchor

• For each candidate failure, the teacher supplies a structured correction hint and the student generates a correction that must be non-degenerate and satisfy all original-prompt constraints.The main instantiation yields 1,443 forward pairs.Source: Forward Execution Anchor · p. 3

Reverse Intent Anchor

• Reverse anchors require an eligible coherent near miss, a standalone achieved prompt matching what the response exhibits, and cycle-consistency validation of the crossed preference.The pipeline excludes factual, logical, unsafe, incomplete, or incoherent failures; 282 of 316 candidates pass validation (89.2%).Source: Reverse Intent Anchor · pp. 3–4

DPO Training on Mixed BPS Dataset

• The mixed dataset contains original-prompt forward pairs and validated achieved-prompt reverse pairs, and standard DPO trains on both with a frozen reference policy.All pairs receive equal per-sample weight; the final reverse density is ρ ≈ 0.195, giving 1,725 total pairs.Source: DPO Training on Mixed BPS Dataset · p. 4

Main Results

• BPS scores above Base on all listed IFBench and IFEval variants, has the strongest listed FollowBench scores, and exceeds both comparisons at every Multi-IF turn.The main comparison uses the matched step-100 checkpoint.Source: Main Results · p. 4, Table 1 · p. 5
• BPS improves all three Multi-IF turns over Base (+2.21, +1.08, +1.03), while the BPS–Fwd gap grows with turn depth (+0.72, +3.37, +4.34).Forward-DPO shifts from +1.49 on turn 1 to −2.29 and −3.31 on turns 2–3; the BPS pattern also appears in 23/24 language-turn averages.Source: Main Results · p. 5, Table 1 · p. 5
• BPS is at or above Base on the listed general-capability metrics, recovers Base-level scores on shown agentic tasks, and matches Base’s ToolBench OOD hallucination rate.Additional Qwen-family checks show the same IFBench pattern, improved code metrics, and stable MMLU in those settings.Source: Main Results · p. 5, Table 2 · p. 6
• A separate function-calling instantiation gives BPS a consistent directional advantage on held-out full-call accuracy and achieved-side boundary switching.This rebuild uses separate data and checkpoint selection, so it tests transfer of the construction rather than the IFBench-trained checkpoint.Source: Main Results · p. 5, Table 3 · p. 6

Diagnostic Analysis

• On held-out crossed anchors, BPS preserves original-side rankings and raises achieved-side ranking accuracy from 6.8% to 62.3%; the Kimi-K2.6 probe shows a similar shift, from 9.3% to 61.6%.The absolute Gemini achieved margin remains small, so the result is a shift in reward geometry rather than a complete preference reversal.Source: Diagnostic Analysis · pp. 5–6, Table 4 · p. 7
• A blind audit of 200 held-out anchors found 91.0% natural achieved prompts, 99.0% agreement with the reverse preference direction, and 0.5% leakage.Three annotators assessed each anchor after seeing only the achieved prompt and two shuffled responses.Source: Diagnostic Analysis · p. 6

Ablation Studies

• Shuffled-Rev partly reduces later-turn degradation but weakens IFBench, while Constraint-Rev helps later Multi-IF turns but drops IFBench below Base; BPS has the strongest tested balance.Shuffled-Rev permutes achieved prompts, whereas Constraint-Rev mechanically modifies violated constraints rather than synthesizing achieved intent.Source: Ablation Studies · p. 6, Table 5 · p. 7
• Duplicating forward pairs leaves achieved-side accuracy at 6.8%, while mechanical reverse pairs reach 32.9% versus BPS at 62.3% and weaken original-side accuracy to 95.9% from 100.0%.Reverse labels work best when the prompt captures the response mode; density checks also show ρ = 0.30 reduces IFBench, while verbosity is higher for Forward-DPO than BPS on IFBench.Source: Ablation Studies · pp. 6–7

• BPS adds validated achieved-prompt reverse anchors, encoding where the same response is wrong and where it is right without changing the DPO objective.Source: Conclusion · p. 7
• Across held-out reward probes and downstream evaluations, BPS counteracts cross-prompt rejection patterns while preserving the original preference boundary.Source: Conclusion · p. 7
• BPS’s clearest downstream separation appears in multilingual multi-turn instruction following, with capability-retention evidence from tool-use, agentic, and code checks.Source: Conclusion · p. 7

• BPS represents achieved intent as a natural-language prompt and may store schemas or constraint lists alongside it for formal action spaces.Source: Limitations · pp. 7–8
• Construction uses teacher calls for correction hints, achieved-prompt synthesis, and validation, with steps that can be batched, cached, or replaced by cheaper verifier models.Source: Limitations · p. 8
• Validation criteria vary by domain: instruction-following uses constraint-satisfaction and leakage checks, while function calling also checks parseability, schema validity, function identity, and argument values.Source: Limitations · p. 8
• Experiments use standard DPO to isolate the effect of crossed boundary-failure pairs; the construction could also be combined with other preference-optimization objectives.Source: Limitations · p. 8

Ethical Statement

• The study uses no private user conversations or deployed-system data; newly collected judgments are task-focused achieved-prompt annotations without personal or sensitive content.Reverse pairs are retained only after eligibility, safety, truthfulness, and non-leakage validation; ambiguous cases are discarded.Source: Ethical Statement · p. 8
• For multilingual, culturally sensitive, or safety-relevant data, the authors say additional audit can use their blind-review protocol.Source: Ethical Statement · p. 8
• The authors report teacher-inference and compute estimates to support reproducibility and transparent construction costs.Source: Ethical Statement · p. 8

AI Assistance Statement

• ChatGPT was used only for language polishing; Gemini-3.1pro-preview served as an offline teacher for construction, not downstream benchmark scoring.Source: AI Assistance Statement · p. 8

Artifacts, Licenses, and Intended Use

• The study uses third-party models, datasets, benchmarks, and tools for research under their respective licenses and terms, and does not redistribute benchmark test sets or model weights.Source: Artifacts, Licenses, and Intended Use · p. 8
• Derived BPS data should preserve source-artifact access conditions, and is intended for research rather than safety-critical or high-stakes deployment without further validation.Source: Artifacts, Licenses, and Intended Use · p. 8

Generalization

• The function-calling instantiation uses Gemini, two-stage validation, 342 forward pairs, and reverse density ρ = 0.15, or 51 reverse pairs.Source: Generalization · p. 8

Training Details

• Training uses standard DPO with sigmoid loss and β = 0.1; main results compare fixed step-100 checkpoints, and the main DPO run takes about 1.6 GPU-hours.The 1,536-token sequence limit applies to training, while evaluation uses benchmark decoding budgets; teacher inference required about 2,500 calls / 5M tokens and is excluded from GPU-hours.Source: Training Details · pp. 8–9
• BPS improves function-calling accuracy, τ2-bench overall, and MBPP+ over both Base and FC-Fwd, but falls below Base on τ-bench retail and slightly on its overall score.FC-Fwd scores higher on τ2-retail.Source: Table 6 · p. 9

Evaluation Implementation Details

• Evaluation follows official or commonly adopted benchmark protocols and uses temperature 0.7, top-p 0.9, and a maximum generation length of 8192 tokens.Source: Evaluation Implementation Details · p. 9

Data Construction Funnel

• Table 7 summarizes retention rates at each construction stage; its rates are relative to the preceding stage except for reverse candidates and valid reverse pairs, and sampling uses K = 3 rollouts per prompt.Source: Data Construction Funnel · p. 9, Table 7 · p. 9

Full ToolBench Results

• Table 8 presents the full ToolBench results across all splits and metrics; the supplied passage does not state their values or comparisons.Source: Full ToolBench Results · p. 9

Reverse Density Ablation

• At ρ = 0.30, IFBench falls below Fwd-DPO (32.82 vs. 34.39), while lower reverse density provides a smaller correction.The main density, ρ = 0.195, retains all validated reverse pairs.Source: Reverse Density Ablation · p. 9, Table 9 · p. 10

Budget-Matched Reward Controls

• Duplicating forward pairs leaves achieved-side ranking unchanged, while mechanical reverse pairs improve it partially but remain below BPS and weaken original-side ranking.The controls match BPS in total pair count or reverse-pair count, respectively.Source: Budget-Matched Reward Controls · p. 9, Table 10 · p. 10

A Challenging BPS Example

• A challenging example converts a coherent answer about the largest country by area into a validated achieved prompt distinct from the original clock question.The original also includes strict multilingual and formatting constraints.Source: A Challenging BPS Example · p. 10

Constraint-Rev Kimi Reward Geometry

• +0.066 (95% CI [0.043, 0.095]) is the paired achieved-margin advantage of BPS over Constraint-Rev on the Kimi-heldout subset.Constraint-Rev improves achieved-side ranking over Forward-DPO but remains below BPS; BPS also preserves stronger original-side margins.Source: Table 11 · p. 10, Constraint-Rev Kimi Reward Geometry · p. 10

Additional Model and Seed-Data Checks

• Table 16 reports additional checks using IFEval prompts as seeds on the Qwen3-4B (base checkpoint) and Qwen2.5-7B-Instruct models.Source: Additional Model and Seed-Data Checks · p. 10

Failure Type Breakdown

• BPS raises achieved-side ranking accuracy for every observed failure type, with the largest gains on common format/structure and lexical mismatches.The breakdown is descriptive rather than a claim about all possible boundary failures; most validated anchors concern surface constraints.Source: Failure Type Breakdown · pp. 10–11

Three-Seed Reward-Geometry Stability

• Across all three runs, achieved-side pairwise ranking accuracy remains above 63% for BPS and near 8% for Forward-DPO, while original-side ranking stays saturated for both methods.Source: Three-Seed Reward-Geometry Stability · pp. 11–12

Output Length Analysis

• Forward-DPO outputs are 3.7× longer than Base on IFBench and are also longer on ToolBench, while BPS keeps lengths closer to Base.Forward-DPO’s longer outputs accompany lower performance on several reported metrics.Source: Output Length Analysis · p. 10, Table 15 · p. 12

Three-Seed IFEval Stability

• Across three training seeds, the average IFEval score is consistently higher for BPS, though its gains are smaller than the crossed-anchor and Multi-IF effects.Source: Three-Seed IFEval Stability · p. 12

Human Audit of Achieved Prompts

• Annotators prefer τfail under Pachieved in 99.0% of audited anchors; 91.0% of achieved prompts are judged natural and leakage is rare (0.5%).The blind audit covers 200 held-out anchors and 600 judgments; reverse-preference agreement is 100% for Gemini-source and 98% for Kimi-source anchors.Source: Human Audit of Achieved Prompts · p. 12

Teacher Prompts and Validation Rubric

• The correction hint asks the teacher to identify violated constraints, needed changes, and retainable factual content; the student then produces τperfect conditioned on the hint.The teacher receives the original prompt, failed response, and unsatisfied constraints.Source: Teacher Prompts and Validation Rubric · p. 12
• For eligible failures, the teacher creates a standalone achieved prompt encoding τfail’s language and structure without quoting it, requesting false content, or implicitly asking for reproduction.The reverse pair is accepted only when the prompt is standalone, τfail fully satisfies it and is coherent, truthful, and safe, and τperfect is clearly worse under it.Source: Teacher Prompts and Validation Rubric · pp. 12–13

Training-Anchor Reward Probe

• 7.1% to 75.9% is the achieved-side pairwise ranking accuracy change on 282 training reverse anchors under BPS.Paired achieved margins rise by +0.44 (95% CI [0.36, 0.54]).Source: Training-Anchor Reward Probe · p. 13
• BPS raises achieved-side pairwise ranking accuracy for every observed failure type, with the largest gains on dominant format/structure and lexical mismatches.The included content-intent and language-mismatch categories are small.Source: Table 18 · p. 14
• +0.360 (95% bootstrap CI [0.320, 0.401]) is the pooled paired achieved-margin improvement across the three-seed Gemini held-out probe.BPS preserves original-side pairwise ranking and raises achieved-side ranking accuracy across seeds.Source: Table 19 · p. 14
• 87.79±0.23 to 88.83±0.59 is the three-seed change in overall IFEval average, with BPS higher.Instruction-level strict is slightly lower on average under BPS, while prompt-level strict, prompt-level loose, and instruction-level loose improve.Source: Table 20 · p. 14

Checkpoint Sensitivity of Forward-DPO

• Fwd-60 has a higher narrow correction-oriented IFBench average than Fwd-100 but remains below BPS on the listed multilingual multi-turn, code, agentic, and tool-use checks.The comparison includes matched Fwd-100 and BPS-100 checkpoints; the main paper uses step 100 to match optimizer updates.Source: Checkpoint Sensitivity of Forward-DPO · p. 13, Table 22 · p. 15