PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Planning with Reasoning for Image Styling via Offline RL

Subhojyoti Mukherjee, Stefano Petrangeli, Branislav Kveton, Trung Bui, Franck Dernoncourt, Arko Mukherjee

arXiv:2603.07148v1cs.LG

TL;DR

Complex image styling is difficult for direct prompt-based editing because vague instructions must coordinate multiple visual attributes and preservation constraints. The paper trains compact planners with compositional tools, structured context, explicit reasoning, synthetic quality-scored trajectories, and offline reward-aware methods. Across 30,000 trajectories and Simple, Regular, and Complex settings, RW and SW outperform competing baselines on most tasks, with visual grounding strengthening RW’s performance.

  • Problem

    Direct prompt-based editing often fails on complex transformations because vague prompts must coordinate multiple visual attributes and preservation constraints.

  • Method

    The framework combines orthogonal compositional tools, structured visual context, per-step reasoning, synthetic trajectories, and offline reward-aware planner training.

  • Results

    Across 30,000 synthetic trajectories and Simple, Regular, and Complex settings, RW and SW outperform competing baselines on most tasks.

  • Takeaways & Limitations

    Reward weighting preserves fine-grained quality distinctions for multi-step reasoning, while visual grounding amplifies RW’s gains.

  • Takeaways & Limitations

    The approach may be misused to generate misleading visual content or deepfakes, despite its structured and interpretable transformations.

Abstract

from arXiv · show

Direct prompt-based editing often fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image. Our core intuition is that leveraging compositional image editing tools rather than direct prompting profits from structured agent-level planning with explicit reasoning, leading to better results. This structured planning framework enables efficient offline RL post-training on quality-scored trajectories to improve performance. We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning. Our key contributions include: (1) A tool-based agentic planning methodology that combines a compositional library of orthogonal primitive transformations, structured context representation, and explicit per-step reasoning to decompose complex styling into interpretable tool sequences. (2) A synthetic data generation pipeline producing three large-scale datasets (each $\sim$10K trajectories) with reasoning chains, plans, and quality scores, as no existing datasets provide such supervision. Our datasets and code are publicly available at the HuggingFace repository. (3) Offline RL training methods for learning planners with reasoning as our core algorithmic contributions, which consistently improve over the Edit-Only baseline in visual quality and instruction following. (4) Comprehensive evaluation across 4B and 8B parameter Qwen3-VL models showing that our methods outperform other baselines in the majority of compositional tasks, validated by human evaluations.

1. Introduction

Complex image styling requires coordinated changes across multiple visual attributes, which direct prompt-based editing often handles inconsistently. The paper proposes structured tool-based planning, synthetic supervision, and reward-aware training to improve compositional editing.

  • High-level styling spans transformations such as time, season, architecture, and artistic medium across creative workflows.
  • Direct prompt-based editing struggles with imprecise instructions requiring coordinated visual changes and preservation constraints.The paper illustrates failures including inconsistent results, misaligned colors, and structural problems.
  • The framework decomposes complex styling into explicit intermediate representations using compositional tools, structured context, and per-step reasoning.Its tool library uses orthogonal primitives and typically composes 2-5 tools for complex transformations.
  • Three synthetic datasets each contain approximately 10,000 trajectories with structured context, multi-step plans, reasoning chains, and quality scores.The Simple, Regular, and Complex variants differ in action count, compositional difficulty, and theme diversity.
  • Reward-aware training methods consistently improve visual quality and instruction following over direct prompt-based editing across the reported settings.The methods include reward-filtered, reward-weighted, standardized reward-weighted, and preference-based training.

2. Problem Setup

The paper formulates image styling as sequential tool composition grounded in a structured visual-state representation. A four-stage pipeline converts vague editing goals into precise instructions rendered by a frozen editor.

  • Image styling is formulated as sequential decision-making over parameterized tools conditioned on an image, prompt, and structured visual state.The representation grounds planning in explicit attributes such as lighting and season.
  • Direct editing maps a vague prompt and base image to an edited image, whereas structured editing replaces the vague prompt with a precise instruction.The stated goal is to obtain better edited images through four structured stages.
  • The tool library covers 10 orthogonal visual dimensions whose effects are intended to combine predictably.Examples include location, architecture, time period, time of day, season, weather, mood, color grading, medium, and atmospheric effects.
  • Stage 2 generates typically 2-5 symbolic actions, with chain-of-thought reasoning produced before each action.Renaissance styling is illustrated through reasoning about historical context and medium before selecting tools.
  • Stage 3 synthesizes a precise natural-language instruction, and Stage 4 renders it with a frozen Qwen-Image-Edit editor and assigns a trajectory reward.Freezing the editor separates planning decisions from pixel rendering.
  • Each trajectory stores the prompt, image, context, actions, synthesized instruction, edited image, and reward within a dataset of 10,000 trajectories per variant.

3. Synthetic Data Generation

The authors generate synthetic teacher-guided trajectories because existing direct-editing datasets lack structured context, explicit reasoning chains, and multi-step plans. The pipeline varies task complexity while preserving common reward evaluation and split procedures.

  • A Qwen3-VL-8B-Instruct teacher generates trajectories containing context extraction, reasoning-based planning, instruction synthesis, and reward evaluation.HiDream-I1-Dev generates images and Qwen-Image-Edit performs editing in the stated pipeline.
  • Explicit reasoning precedes each action so students learn why tools are selected and how actions contribute to the styling goal.The reasoning-action interleaving is intended to support interpretable planning, action quality, and instruction following.
  • Trajectories sharing a base image remain in the same 80%/10%/10% train-validation-test split to evaluate generalization to new visual content.
  • Rewards average scores across 17 dimensions, including 11 for action-plan quality and 6 for final-image quality, on a 0-5 scale.The reward supports reward-aware methods such as RW, SW, and DPO.
  • The Simple, Regular, and Complex datasets each contain 10,000 trajectories and progress from 1-2 actions to 3-5 actions with 83 diverse themes and preservation constraints.All variants use the same four-stage generation and reward-evaluation pipeline.
  • Human evaluation of 3,000 training samples produced a 77% pass rate, with every dataset variant exceeding 70%.

4. Learning Algorithms

The paper compares supervised learning, reward filtering, reward weighting, standardized weighting, and preference optimization for offline planner post-training. These methods differ in whether they ignore, filter, weight, normalize, or contrast reward information.

  • Offline reinforcement learning provides the training setting for reward-aware post-training of planners.
  • Supervised Learning: Supervised fine-tuning maximizes likelihood over complete reasoning-action trajectories while ignoring their reward differences.Consequently, poor and excellent trajectories contribute equally under the described objective.
  • Reward-Filtered Training: Reward-filtered training retains trajectories with r_i ≥ 4.0, preserving approximately 65% of the data.It removes clearly poor samples but discards diversity and ignores continuous quality differences.
  • Preference Optimization: DPO learns from paired same-input trajectories by preferring higher-reward examples through a Bradley-Terry objective with KL regularization β = 0.1.The method requires paired data and doubles computational cost per sample.
  • Reward Weighting: Reward-weighted training uses all trajectories while assigning larger gradient contributions to higher-quality samples.This preserves data diversity while emphasizing successful behaviors.
  • Standardized Reward Weighting: Standardized Reward-Weighted training z-score normalizes rewards before using them as sample weights.Above-average trajectories receive positive weights and below-average trajectories negative weights, providing variance reduction across same-input rollouts.

5. Experiments

Experiments across synthetic datasets, model sizes, and modalities show that method effectiveness depends on task characteristics, while structured planning improves over direct editing and trained compact models often exceed GPT-4o.

  • Evaluation Setup: 3 synthetic datasets and 4B/8B text-only or vision-language Qwen3-VL models are evaluated using GPT-4o on six image-quality dimensions.The evaluation covers 200 test samples, with human validation of the evaluator reported separately.
  • Comparison with GPT-4o: Trained 4B/8B models outperform GPT-4o on image quality in 10 out of 11 configurations, with human validation reporting a 77% pass rate.GPT-4o is treated as a separate zero-shot reference because it is a much larger closed-source model.
  • Baseline Comparison: Overall gaps of 1.3-7.3 points show Edit-Only consistently underperforms the best-performing RL method, supporting structured action planning.Edit-Only has no planning metrics because it does not use planning or tool calls.
  • Results by Task and Modality: 78.77 on Regular Text-4B and 77.86 on Regular Text-8B make SW the strongest method for compositional text tasks.The figures report SW outperforming the GPT-4o zero-shot baseline in both configurations.
  • Results by Task and Modality: 79.33 makes RW strongest on Simple Vision-4B, where visual grounding favors continuous reward weighting.Vision models achieve higher absolute scores than their text-only counterparts.
  • Results by Task and Modality: 85.41 makes DPO strongest on Complex Vision-8B, followed closely by RW and SW across diverse themes.The complex setting contains 83 themes.

6. Conclusion

The paper presents an offline-RL framework for compositional image styling and finds that reward-aware methods perform differently across task complexity and modality. Its datasets and evaluation pipelines provide a blueprint for agentic systems in creative domains.

  • Conclusion: The framework evaluates 30,000 synthetic trajectories across Simple, Regular, and Complex settings and finds RW and SW outperform competing baselines on most tasks.The evaluation includes human evaluation.
  • Conclusion: Reward weighting preserves fine-grained quality distinctions for multi-step reasoning, with visual grounding amplifying RW’s advantage on vision-4B models.The framework uses a compositional library of 10 primitives with per-step reasoning and a 5-stage synthetic data pipeline.
  • Conclusion: The released data-generation and ground-truth-free evaluation pipelines are presented as a general blueprint for efficient agentic systems in creative domains.Future work includes video editing with temporal consistency and larger tool libraries.

Impact Statement

The paper frames structured, interpretable image-editing agents as useful for creative workflows while acknowledging potential misuse for misleading visual content and deepfakes. Its methods trade data retention, quality weighting, preference comparison, and computational simplicity.

  • Impact: The framework’s explicit reasoning and documented transformation steps may improve transparency compared with black-box editing methods.The paper connects this interpretability to potential content provenance and authenticity verification efforts.
  • Risks: Potential misuse includes generating misleading visual content or deepfakes, despite the approach’s stated interpretability benefits.This boundary is raised in the impact statement rather than as an evaluated failure mode.
  • Method Trade-offs: RW and SW preserve all training data while weighting trajectories by continuous reward, unlike R, which discards 35% of trajectories.RW and SW emphasize quality through each trajectory’s contribution to parameter updates.
  • Method Trade-offs: DPO uses chosen-rejected trajectory pairs to capture fine-grained quality distinctions, whereas RW is advantageous when trajectories are broadly moderate-to-high quality.DPO requires paired preferences, while RW uses continuous reward weighting.
  • Method Trade-offs: R retains 65% of data after applying an r ≥4.0 threshold and offers computational simplicity while maintaining training signal.The paper describes this balance as effective across diverse task types and complexity levels.
  • Qualitative Comparisons: Qualitative comparisons show RW, SW, DPO, and R excelling in different examples across text-only, vision-language, simple, regular, and complex settings.The comparisons include 9-way outputs spanning Original, Baseline, Edit-Only, trained methods, and GPT-4o Planner.

A. Visual Method Comparisons

The visual comparisons examine how training methods perform across image-styling settings and relate the framework to agentic planning, generative-model alignment, and reward-weighted offline RL.

  • Visual Comparisons: Each qualitative comparison uses nine outputs: Original, Baseline, Edit-Only, Standard SL, R, RW, SW, DPO, and GPT-4o Planner.The comparisons are intended to show when specific training methods excel.
  • Related Work: The related-work discussion positions the framework between controllable image synthesis, agentic reasoning, and offline reinforcement learning.It distinguishes direct editing, planner-based decomposition, preference optimization, and reward-weighted fine-tuning.
  • Related Work: Reward-weighted offline-RL methods are extended from language-model conversation optimization to vision-language compositional planning tasks.The cited lineage includes Reward-Weighted Regression, Advantage-Weighted Regression, and standardized reward-weighted fine-tuning.

C. Complete Problem Formulation Details

The appendix specifies structured context extraction, action libraries, reward evaluation, and teacher-generated trajectories for compositional image styling. Examples show that coherent transformations can score highly, while preservation constraints remain difficult when aesthetic requirements conflict.

  • C.1. Context Representation Details: The framework extracts a structured JSON representation of 10 visual attributes from each image using Qwen3-VL-8B-Instruct.The representation provides explicit state awareness for planning across location, architecture, era, time, season, weather, lighting, color, medium, and atmospheric effects.
  • C.2. Action Space Specification: The action space uses parameterized transformations, with 10 core actions for normal data and 20 actions for complex data.Actions include location, architecture, era, and compositional constraints such as preserving attributes or excluding regions.
  • C.3. Reward Function Details: The reward function assigns scores in [0, 5] using six weighted criteria to define quality tiers for training.Reward thresholds support methods including continuous reward weighting and preference learning.
  • C.4. Synthetic Data Generation Details: The synthetic-data pipeline generates trajectories with context extraction, reasoning-based planning, instruction synthesis, and reward evaluation.The teacher trajectory procedure samples actions and reasoning iteratively, updates the current context, and stops when the goal is satisfied or aSTOP is selected.
  • Stage 3: Action Planning with Teacher Model: The autumn-vineyard example achieves 5/5 adherence to both its plan and prompt, with an overall quality score of 4/5.The transformation creates a spring tulip field with emerging technology, while technical metrics indicate some image distortion.
  • Stage 3: Action Planning with Teacher Model: The cyberpunk example scores 3/5 across planning, reasoning, adherence, and overall quality because preservation constraints are only partially satisfied.Neon lighting creates the intended atmosphere, but wooden elements and plants are not fully preserved or integrated.
  • Key Takeaways: Preservation-constrained transformations require careful balancing of action intensity and coverage, which discrete parameterization handles poorly.The paper suggests continuous parameter spaces or iterative refinement as future directions.
  • Key Takeaways: The reward model distinguishes fully successful transformations from partially successful ones, providing meaningful downstream training signal.Successful samples receive 4-5/5 scores, while the constrained cyberpunk example receives 3/5.

D.5.1. KEY INSIGHTS FROM THREE-DATASET COMPARISON •

The comparison describes how dataset design, action complexity, and training objectives shape planner behavior. It also contrasts reward-aware methods with standard supervised learning and details the supervised baseline implementation.

  • D.5.1. KEY INSIGHTS FROM THREE-DATASET COMPARISON: The datasets differ in action-library design, with orthogonal atomic actions expanded by compositional and constraint actions for harder settings.Normal uses 10 actions, while regular and complex variants introduce broader compositional and constraint capabilities.
  • D.5.1. KEY INSIGHTS FROM THREE-DATASET COMPARISON: RW performs best on simple vision tasks, SW on regular text tasks, and DPO on regular vision tasks.Reported overall scores include 79.33 for Vision-4B, 78.77 for Text-4B, and 85.41 for Vision-8B.
  • D.5.1. KEY INSIGHTS FROM THREE-DATASET COMPARISON: Standard supervised learning predicts actions and per-step reasoning from the image, prompt, structured context, and action history while ignoring rewards.The baseline maximizes trajectory likelihood using the synthetic action and reasoning sequences.

E.1.4. LIMITATIONS OF STANDARD SL

Standard supervised learning treats all synthetic trajectories equally, wasting reward information and lacking preference signals. Reward-aware alternatives instead use trajectory quality to prioritize or compare training examples.

  • E.1.4. LIMITATIONS OF STANDARD SL: Standard supervised learning is quality-blind: trajectories with rewards of 3.0 and 5.0 contribute equally.This treats poor and excellent synthetic behavior as having identical training influence.
  • E.1.4. LIMITATIONS OF STANDARD SL: Low-quality trajectories can cause degradation by teaching suboptimal behaviors and reducing alignment with teacher performance.The risk is greatest when poor-quality trajectories are prevalent.
  • E.1.4. LIMITATIONS OF STANDARD SL: Standard supervised learning provides no preference signal when multiple plans exist for the same image-prompt pair.It cannot indicate which of several candidate trajectories is better.
  • E.1.4. LIMITATIONS OF STANDARD SL: The baseline completely ignores the reward evaluations computed during synthetic data generation.Reward-aware methods are introduced to use this otherwise discarded information.
  • E.1.4. LIMITATIONS OF STANDARD SL: RW assigns continuous weights w(r) = max{r − 3.0, 0}, giving higher-quality trajectories greater influence and zeroing rewards below 3.0.Its normalized weighted loss preserves a comparable loss scale while scaling gradient contributions by quality.
  • E.1.4. LIMITATIONS OF STANDARD SL: Reward-aware methods improve over SL, with DPO and RW reporting 12-15% gains and RW reporting 18-22% gains on complex compositional reasoning.A 4B model trained with DPO can match or exceed an 8B model trained with standard SL on several metrics.

F.8. Edit-Only Baseline Detailed Analysis

Edit-Only bypasses explicit action planning, producing competitive visual quality on simpler atomic edits but weaker instruction following and inconsistent complex transformations.

  • Edit-Only bypasses action planning by directly applying ground-truth edit instructions to images.
  • Complex Text-4B: 71.49 overall on Complex Text-4B trails SW’s 78.77 by 7.28 points, while planning metrics are unavailable.Visual quality is integrated into Overall and not separately evaluated.
  • Complex Text-8B: 71.24 overall on Complex Text-8B trails SW’s 77.86 by 6.62 points, with significantly lower instruction following.The same pattern suggests that model scale alone does not compensate for absent structured planning.
  • When Edit-Only Can Be Competitive: Edit-Only is more competitive on atomic tasks, where the gap narrows to 1.29 points, but it still lacks explicit reasoning and trails on instruction following.The evidence attributes complex-edit failures to missing decomposition into atomic actions.
  • Complex Vision-8B: 83.38 overall on Complex Vision-8B trails DPO’s 85.41 by 2.03 points, despite Edit-Only achieving the highest visual quality at 84.07.Its instruction-following score is 83.81 versus DPO’s 87.03.

F.9.1. COMPLEX TEXT-4B DETAILED RESULTS

On Complex Text-4B, SW achieves the highest Overall score and leads most planning metrics, while R narrowly leads Visual Quality and RW ranks second overall.

  • 78.77 overall makes SW the winner on Complex Text-4B.SW also leads Semantic Accuracy, Coherence, Technical Execution, Instruction Following, and Transformation Strength.
  • 76.58 Semantic Accuracy is SW’s lead over RW’s 73.61 and R’s 73.61.
  • 81.55 Coherence is SW’s highest score, ahead of R’s 81.42 and RW’s 80.00.
  • 80.19 Technical Execution is SW’s highest score, exceeding R’s 79.61 and RW’s 78.84.
  • 77.55 Instruction Following and 73.94 Transformation Strength are SW’s leading scores on those metrics.The corresponding comparisons are RW 76.84 and R 76.26 for Instruction Following, and RW 72.45 and R 71.87 for Transformation Strength.
  • 83.03 Visual Quality makes R the leader, narrowly ahead of SW’s 82.84, while RW places second overall at 77.18.R’s filtering strategy uses r_i ≥4.0 to select visually appealing examples.

F.9.2. COMPLEX TEXT-8B DETAILED RESULTS

Across the larger-model configurations, overall winners vary by task and modality: SW leads text-only Complex Text-8B, RW leads Simple Vision-4B, and DPO leads Complex Vision-8B.

  • Complex Text-8B: 77.86 overall makes SW the winner on Complex Text-8B, but its lead over R is only 0.24 points and over RW 0.52.
  • Complex Text-8B: At Complex Text-8B scale, RW wins Visual Quality at 83.00, Coherence at 81.93, and Technical Execution at 79.67.SW wins Semantic Accuracy at 74.53 and Instruction Following at 77.00; R wins Transformation Strength at 73.80.
  • Simple Vision-4B: 79.33 overall makes RW the winner on Simple Vision-4B, leading Visual Quality at 83.95 and Instruction Following at 78.06.SW leads Coherence at 83.57, while RW and SW tie on Technical Execution at 81.86.
  • Simple Vision-4B: Edit-Only reaches 78.04 overall on Simple Vision-4B, only 1.29 points behind RW, reflecting the relative simplicity of single-action tasks.It still trails the best methods and has no planning-metric scores.
  • Complex Vision-8B: 85.41 overall makes DPO the winner on Complex Vision-8B, leading all listed planning and transformation metrics.DPO’s highest scores include Semantic Accuracy 87.12, Coherence 85.51, Technical Execution 83.98, Instruction Following 87.03, and Transformation Strength 85.68.
  • Complex Vision-8B: 84.07 Visual Quality makes Edit-Only the visual-quality leader on Complex Vision-8B, ahead of DPO’s 82.97, despite weaker instruction following.DPO’s diverse 83-theme dataset is associated with its stronger preference-learning performance.
  • Cross-configuration comparison: No single method dominates all configurations: SW leads Regular Text, RW leads Simple Vision-4B, and DPO leads Regular Vision-8B.

G.1.1. REGULAR DATASET: TEXT-8B MODELS

Across evaluated configurations, trained planners outperform direct Edit-Only editing, while the strongest method depends on model scale, visual grounding, and dataset complexity. For Regular Text-8B, SW leads overall quality, whereas RW leads visual-quality dimensions.

  • 77.86 Overall score makes SW the strongest trained method on Regular Text-8B, followed by R at 77.62 and RW at 77.34.
  • RW leads Visual Quality at 83.00 and Coherence at 81.93, while SW leads Semantic Accuracy at 74.53 and Instruction Following at 77.00.
  • 71.24 Overall score for Edit-Only is substantially below the other trained methods, highlighting the importance of structured action planning for complex edits.
  • 79.33 Overall score makes RW the strongest method on Simple Vision-4B, where it leads multiple visual-grounded dimensions.
  • 79.62 Overall score makes R the strongest method on Simple Vision-8B, while RW remains strongest on Visual Quality and Coherence.
  • 85.41 Overall score makes DPO the strongest method on Complex Vision-8B, indicating that method effectiveness varies with model capacity, visual grounding, and task complexity.

H.2. Key Findings on Reasoning Quality

Reward-aware training improves reasoning and action-plan quality, with reasoning quality strongly aligned with action quality. Qualitative examples further show that SW produces more specific, compositional, and constraint-aware plans than baseline reasoning.

  • 79.15 Overall Reasoning Quality for RW on Simple Vision-4B exceeds Baseline at 77.93 and Standard at 76.95.
  • Pearson r > 0.85 links Overall Reasoning Quality with Overall Action Quality across the evaluated configurations.
  • 82.09 Overall Score makes RW strongest on Simple Vision-4B, while SW leads Regular Vision-4B at 81.57 and Regular Vision-8B at 83.08.
  • Vision grounding raises Overall Action Quality by 2-5 points and Overall Reasoning Quality by 1-3 points relative to text-only configurations.
  • SW provides more specific visual observations and balances competing constraints instead of refusing difficult edits.
  • SW reduces the church-to-Angkor-Wat transformation from four baseline actions to two by treating location setting as encompassing architectural change.

I.7. Evaluation Infrastructure

Evaluation combines GPT-4o-based, traditional, and human assessments of image-editing trajectories. Human ratings indicate generally high synthetic-data quality and low rates of fundamental annotator disagreement, while exact agreement remains moderate.

  • GPT-4o evaluation uses temperature 0.0 and costs approximately $0.02 per trajectory evaluation.
  • GPT-4o scores six image-quality dimensions and eleven action-plan dimensions on a 0-100 scale, then averages the dimension scores.
  • Human evaluation uses three annotators, four quality dimensions, and a Pass/Partial/Fail scale over stratified trajectory samples.
  • 77% of 873 rated samples receive an overall pass, supporting the quality of the synthetically generated training data.
  • All dataset variants exceed 70% pass rates, with Complex reaching 79.4% despite strict preservation constraints and adversarial prompts.
  • 62.8%-66.2% exact agreement and below 11% complete disagreement indicate moderate rating alignment with rare fundamental Pass-versus-Fail conflicts.

J.4. Validation of Dataset Quality

Human evaluation supports the quality of the synthetic training data and shows that advanced training methods generally perform similarly, with SW and D among the strongest methods. GPT-4o rankings align only weakly with human judgments, so human validation remains important.

  • Dataset Quality: All three dataset variants exceeded 70% pass rates, indicating consistent quality across task complexities.Complex achieved the highest pass rate at 79.4%, despite adversarial prompts and strict preservation constraints.
  • Dataset Quality: Less than 11% of annotations showed complete disagreement, indicating consistent identification of unsuitable training samples.Annotators could differ on borderline cases while still agreeing on fundamentally unsuitable samples.
  • Dataset Quality: 77% of synthetic samples passed human quality assessment, with another 14.9% rated Partial.The evaluation covered the outputs of the four-stage synthetic data generation pipeline.
  • GPT-4o Validation: GPT-4o’s mean Spearman correlation with human rankings was approximately 0.10, while winner accuracy was 46.9%.Top-2 accuracy was higher at 78.3%, but correlation varied by method from -0.16 to +0.30.
  • Method Ranking: SW and D were top-performing methods, with win rates of 20–23% across datasets, while RW and R achieved 16–18%.Win-rate differences were typically 5–10 percentage points, indicating similar performance among advanced methods.
  • GPT-4o Validation: GPT-4o should not serve as the sole quality metric, although it can support relative comparisons and large-scale screening.The study recommends validating critical findings with human evaluation.
Loading 2603.07148v1…