Source-linked AI summary
Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models
Andrej Andrejev
TL;DR
The paper asks whether iterative diff-based generation is a better training regime than direct whole-file generation for code editing. It trains two models in both regimes on shared Flutter/Dart data and finds direct generation performs better overall, while diff-based methods remain competitive on short, localized edits.
Problem
Evidence is limited on whether diff-based output is superior as a training objective rather than merely as an inference-time interface for code editing.
Method
Two architecturally different models are trained in direct and iterative diff-based regimes on shared Flutter/Dart data and evaluated with controlled comparisons.
Results
Direct generation outperforms diff-based generation across all measured metrics, with the gap persisting under matched-ID and jointly compiling-code comparisons.
Takeaways & Limitations
Diff-based generation is most competitive for short, spatially localized edits, linking its wins to task locality and short trajectories.
Takeaways & Limitations
The regimes are not token-matched, with steps-mode fine-tuning using roughly 10× more tokens than direct-mode fine-tuning.
Abstract
from arXiv · showhide
Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.
1 Introduction
The paper compares direct whole-file generation with iterative diff-based editing as training regimes for Flutter/Dart code models. It finds direct generation is broadly stronger, while diff-based generation is competitive for short, localized tasks.
- Direct generation emits the complete modified file, whereas diff-based generation applies a sequence of localized search/replace edits to the original.
- The training-objective evidence is mixed: edit-sequence training can improve synthesis quality, while practical diff formats can increase malformed edits.
- The study trains two architecturally different models in both regimes on shared data and holds tokenization, data, and evaluation fixed across arms.
- Direct generation beats iterative diff-based generation by a wide margin on every measured metric across both architectures.
- The aggregate gap persists after matched-ID controls and when comparisons are restricted to code compiling under both regimes.
- Diff-based wins concentrate in short edit trajectories, especially refactoring and error-handling/edge-case fixes, the dataset’s two lowest-step categories.
2 Related Work
Related work treats edit format as a meaningful design choice spanning inference interfaces, training curricula, adaptive routing, and iterative repair. This paper extends those comparisons to controlled Flutter/Dart editing while explicitly scoping its steps mode as a simpler single-pass loop.
- Edit formats as an inference-time interface: Diff-style interfaces reduce token cost relative to whole-file replacement but increase mechanically unapplyable edits, especially for weaker models.
- Diff/edit sequences as a training curriculum: LintSeq reports improved downstream synthesis from synthetic edit-sequence training, whereas this paper studies editing existing files rather than blank-slate synthesis.
- Adaptive and hybrid formats: Adaptive-format work argues that per-edit format selection can outperform committing a model to either diff or whole-file output.
- Adaptive and hybrid formats: The paper’s task-locality finding independently aligns with adaptive-format claims by associating diff competitiveness with short, spatially localized edits.
- Iterative program repair and multi-step editing agents: Unlike feedback-driven repair agents, the paper’s steps mode is single-pass, receives no execution feedback, and targets self-contained Flutter/Dart snippets.
- Instruction-tuned code editing: Instruction-tuning research likewise finds that response-format choices materially affect code-editing quality.
3 Methodology
The methodology compares four controlled Flutter/Dart editing arms built from two backbones and two output regimes. It also defines step-based training, application, and matched-ID evaluation procedures while documenting important training asymmetries and safeguards.
- Backbones: The experiment uses a from-scratch Rainbow-Pony-100M backbone and a pretrained Qwen2.5-Coder-0.5B backbone.
- Output regimes: Both backbones are fine-tuned in direct and steps regimes, yielding four arms evaluated on the same held-out Flutter/Dart task set.
- Output regimes: Direct mode generates the complete modified file in one pass, while steps mode sequentially generates mechanically applied search/replace actions until stopping or reaching 20 steps.
- Training data: Both regimes derive from the same pool of 14,600 hand-designed Flutter/Dart tasks, but steps training expands examples into decomposed step-level rows.
- Training configuration: Steps-mode fine-tuning receives roughly 10× more tokens than direct-mode fine-tuning, so the regimes are not token-matched.
- Action format: Steps prompts include goal, current code, edit history, and tagged action/change output, whereas direct prompts produce the complete modified file without history or action structures.
- Edit application: The application heuristic may silently edit the wrong repeated span because first- or last-occurrence resolution is not reliably correct.
- Matched-ID comparison: The matched-ID comparison restricts steps results to normally completed trajectories without fallback or budget exhaustion, then compares the same sample IDs against direct outputs.
4 Results
Direct generation outperforms steps-mode generation across aggregate metrics for both architectures. Most steps-mode failures arise from normally completed trajectories, with ambiguity fallbacks accounting for a large share of failures within that group.
- Aggregate performance: The evaluation compares four arms using shared held-out sets for each architecture.Table 2 summarizes the three core metrics across direct and steps modes.
- Aggregate performance: Direct generation beats steps-mode generation on dart_pass, bits_per_byte, and similarity_ratio for both architectures.The dart_pass gaps are 45.5 percentage points for Rainbow-Pony and 39.9 points for Qwen.
- Failure attribution: Most steps-mode trajectories complete normally, with 81–85% stopping because the model emitted an explicit done action.Table 3 categorizes trajectory outcomes by stop_reason.
- Failure attribution: Normally completed trajectories account for approximately 84% of Rainbow-Pony and 70% of Qwen steps-mode failures.Thus, the bulk of failures is not attributable to step-budget exhaustion or unrecoverable apply failures.
- The role of fallback edits: Fallback-affected trajectories account for roughly 65% of Rainbow-Pony and 71% of Qwen failures within the done bucket.Trajectories requiring fallback resolution occur at less than a quarter the rate of trajectories needing none.
4.4 Correcting for task-selection bias: matched-ID comparison
Matched-ID analysis shows that task-selection bias does not explain steps mode’s disadvantage: direct mode remains stronger on identical sample IDs, including among outputs that compile on both sides.
- Matched-ID comparison: 792 clean steps IDs were matched against direct-mode outputs on the identical sample IDs.The clean subset consisted of trajectories that were done, required no fallback, and used fewer than 20 steps.
- Matched-ID comparison: The matched direct subset was easier than direct mode’s full population, confirming that clean steps trajectories were not a random task sample.This selection effect should favor steps mode in the comparison rather than disadvantage it.
- Matched-ID comparison: After accounting for selection, the residual gap between steps and direct was larger rather than smaller than the naive full-population gap.The matched-ID result therefore strengthens, rather than weakens, the aggregate comparison.
- Compiling outputs: Among matched rows where both outputs passed dart_pass, direct-mode output still received higher blinded LLM-judge ratings.Table 7 covers 390 paired Rainbow-Pony rows and 688 paired Qwen rows; all six pairwise differences were significant at p < 0.001.
- Compiling outputs: The matched comparisons therefore separate the raw compile-rate gap from a residual quality difference among code that compiles successfully either way.The paper describes roughly half of the raw dart_pass gap as a judge-confirmed residual quality difference.
4.6 Where does steps mode win? A paired analysis
A paired analysis finds a limited but reproducible region where steps mode wins: short trajectories, especially in refactoring and error-handling or edge-case tasks, while some metrics are unreliable indicators of editing quality.
- Paired analysis: Per-row paired comparisons flag whether steps strictly improves over direct on each metric for the same sample ID.The analysis compares rows present in both a model’s direct and steps outputs.
- Metric caveats: Bits_per_byte’s near-zero win rate is uninformative about editing quality, while similarity_ratio’s high win rate is considered a metric artifact.The paper notes that bits_per_byte is structurally decoupled from the steps trajectory and that similarity_ratio can be inflated by edit-base reuse.
- Additional predictors: Initial file length does not explain steps-mode wins: effects are small and have opposite signs across architectures.Rainbow-Pony wins skew toward 6% longer files, whereas Qwen wins skew toward 11% shorter files.
- Category analysis: Refactoring and error-handling or edge-case fixes are the only categories consistently overrepresented among steps-mode wins in both architectures.Their win counts are small, with 63–86 total wins per architecture, so other category differences are treated as noise.
- Trajectory length: Steps-mode wins are associated with significantly shorter trajectories in all six architecture-by-judge-dimension tests.The composite judge-majority comparison and each individual judge dimension show the same association.
- Trajectory length: 10–18% win rates at 1–6 steps decline to approximately 0% by 14–19 steps across the examined category and architecture breakdowns.Steps-mode judge scores decline as trajectories lengthen, while direct-mode scores on the same rows remain roughly flat.
4.7 Unifying the findings: task locality
Category and trajectory findings converge on task locality: steps mode is competitive when the required change is spatially narrow and self-contained, rather than because of category label or architecture alone.
- Task locality: Task locality unifies the category and trajectory findings as one underlying variable.The paper defines locality as how spatially narrow and self-contained the required change is.
- Category and trajectory link: Refactoring and error-handling or edge-case tasks have the two lowest mean num_steps among nine categories for both architectures.These are the same categories overrepresented among steps-mode wins.
- Task locality: Win rates and judge scores both deteriorate as trajectory length increases, while direct-mode scores stay roughly flat across the same range.The trajectory-length pattern is therefore consistent with the category pattern because the winning categories require fewer edit steps.
- Task locality: Task locality, rather than category label or architecture per se, predicts when iterative diff-based generation is competitive with direct whole-file generation.This is the paper’s unified interpretation of the matched category and trajectory analyses.
5 Discussion
On this Flutter/Dart task, direct whole-file generation is the better default, while diff-based generation remains competitive for short, localized edits. Task locality explains this pattern and supports choosing the format per edit rather than per model.
- Discussion: Whole-file direct generation is the better default because steps-mode costs grow with trajectory length faster than token-efficiency benefits beyond a handful of steps.Longer trajectories also create more opportunities for wrong-span selection, compounded errors, and lost file-state tracking.
- Discussion: Diff-based generation is specifically competitive on short, spatially localized edits.
- Discussion: The categories where diff-based generation wins are independently the dataset’s two lowest mean-edit-step-count categories for both architectures.These categories are refactoring and error-handling/edge-case fixes.
- Discussion: Task locality reconciles the two regimes as complementary tools for different edit types rather than universally conflicting approaches.This framing is consistent with the adaptive-format argument.
6 Limitations •
The study’s conclusions are bounded by its token-budget imbalance, narrow evaluation setting, decoding procedure, feedback-free steps mode, apply heuristic, judge validation, subgroup sizes, and one training-scheduler error.
- Training setup: Steps-mode fine-tuning used roughly 10× more tokens than direct-mode fine-tuning, 50M versus 5M, yet still underperformed in aggregate.The unequal budget cuts against explaining the central finding through insufficient steps-mode training.
- Scope: All results concern Flutter/Dart editing, with no claim of generalization to other languages or larger multi-file repositories.
- Evaluation: Evaluation used greedy decoding and one generation per task, so pass@k and sampling-temperature effects were not measured.
- Evaluation: Steps mode received no compiler or test feedback between edits and should not be extrapolated to feedback-driven, tool-using agents.
- Evaluation harness: The first-occurrence ambiguity fallback can silently introduce errors, and a stricter harness might reduce the observed gap.
- Evaluation: Blinded LLM-judge scores were not independently validated against human raters.A human-agreement spot-check is left for future work.
- Subgroup analyses: Category-level win analyses contain only 63–86 total wins per architecture and report no category-level significance tests.
- Training setup: qwen-direct used a non-decaying learning rate for the back half of training because of a scheduler configuration error.Its validation loss plateaued rather than continuing to settle.
7 Conclusion
Across two architecturally distinct models trained on the same Flutter/Dart editing data, direct whole-file generation outperformed iterative diff-based generation on every measured metric. Diff-based generation remained competitive on short, spatially local edits, motivating task locality as a criterion for choosing between formats per edit.
- Conclusion: Direct whole-file generation outperformed iterative diff-based generation on every measured metric across both model architectures.This included blinded LLM-judge assessments of code compiling successfully under both regimes.
- Conclusion: Most of the performance gap came from normally completed trajectories that drifted semantically, especially when search targets were ambiguous.
- Conclusion: Diff-based generation was specifically competitive on short, spatially local edits.
- Conclusion: Task locality is proposed as the axis for deciding per edit whether whole-file or diff-based generation is appropriate.The conclusion adds an independent empirical data point supporting adaptive-format proposals.
8 Data and Code Availability
The paper releases evaluation datasets, fine-tuning data, model checkpoints, and training and evaluation harness code. The datasets include task inputs, references, model outputs, automated metrics, trajectory metadata, and blinded judge scores.
- Datasets: Four evaluation datasets are released on the Hugging Face Hub, one for each architecture/mode arm, joined by a shared sample_id.
- Dataset contents: Each dataset includes the task instruction, initial file, reference final file, model output, and automated metrics.Metrics include dart_pass, Dart analysis counts, bits_per_byte, and similarity_ratio.
- Dataset contents: Steps-mode datasets include stop reasons, step counts, fallback counts, action lists, and trajectory logs.
- Dataset contents: Datasets include blinded LLM-judge scores for goal fulfillment, correctness, and code quality using gpt-4.1.
- Models: The four fine-tuned checkpoints and the shared Rainbow-Pony pretrained-only base checkpoint are released.
- Datasets: The two fine-tuning datasets are released, including flutter-full-examples-v1 with 14,600 rows and flutter-diff-steps-v1.
- Licensing: Licenses differ by artifact type: evaluation datasets and model checkpoints use MIT, while fine-tuning datasets use Apache-2.0.
- Code: Training and evaluation harness code, including decomposition, static analysis, and judge scripts, is released on GitHub.