Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

• Direct generation emits the complete modified file, whereas diff-based generation applies a sequence of localized search/replace edits to the original.Source: §1 · p. 1
• The training-objective evidence is mixed: edit-sequence training can improve synthesis quality, while practical diff formats can increase malformed edits.Source: §1 · p. 2
• The study trains two architecturally different models in both regimes on shared data and holds tokenization, data, and evaluation fixed across arms.Source: §1 · p. 2
• Direct generation beats iterative diff-based generation by a wide margin on every measured metric across both architectures.Source: §1 · p. 2
• The aggregate gap persists after matched-ID controls and when comparisons are restricted to code compiling under both regimes.Source: §1 · p. 2
• Diff-based wins concentrate in short edit trajectories, especially refactoring and error-handling/edge-case fixes, the dataset’s two lowest-step categories.Source: §1 · p. 2

Edit formats as an inference-time interface

• Diff-style interfaces reduce token cost relative to whole-file replacement but increase mechanically unapplyable edits, especially for weaker models.Source: §2 · p. 2

Diff/edit sequences as a training curriculum

• LintSeq reports improved downstream synthesis from synthetic edit-sequence training, whereas this paper studies editing existing files rather than blank-slate synthesis.Source: §2 · p. 3

Adaptive and hybrid formats

• Adaptive-format work argues that per-edit format selection can outperform committing a model to either diff or whole-file output.Source: §2 · p. 3
• The paper’s task-locality finding independently aligns with adaptive-format claims by associating diff competitiveness with short, spatially localized edits.Source: §2 · p. 3

Iterative program repair and multi-step editing agents

• Unlike feedback-driven repair agents, the paper’s steps mode is single-pass, receives no execution feedback, and targets self-contained Flutter/Dart snippets.Source: §2 · p. 3

Instruction-tuned code editing

• Instruction-tuning research likewise finds that response-format choices materially affect code-editing quality.Source: §2 · p. 3

Backbones

• The experiment uses a from-scratch Rainbow-Pony-100M backbone and a pretrained Qwen2.5-Coder-0.5B backbone.Source: §3.1 · pp. 3–4

Output regimes

• Both backbones are fine-tuned in direct and steps regimes, yielding four arms evaluated on the same held-out Flutter/Dart task set.Source: §3.1 · p. 4
• Direct mode generates the complete modified file in one pass, while steps mode sequentially generates mechanically applied search/replace actions until stopping or reaching 20 steps.Source: §3.1 · p. 4

Training data

• Both regimes derive from the same pool of 14,600 hand-designed Flutter/Dart tasks, but steps training expands examples into decomposed step-level rows.Source: §3.1.1 · p. 4

Training configuration

• Steps-mode fine-tuning receives roughly 10× more tokens than direct-mode fine-tuning, so the regimes are not token-matched.Source: §3.1.1 · p. 4

Action format

• Steps prompts include goal, current code, edit history, and tagged action/change output, whereas direct prompts produce the complete modified file without history or action structures.Source: §3.1.2 · pp. 5–6

Edit application

• The application heuristic may silently edit the wrong repeated span because first- or last-occurrence resolution is not reliably correct.Source: §3.1.3 · p. 6

Matched-ID comparison

• The matched-ID comparison restricts steps results to normally completed trajectories without fallback or budget exhaustion, then compares the same sample IDs against direct outputs.Source: §3.3 · p. 7

Aggregate performance

• The evaluation compares four arms using shared held-out sets for each architecture.Table 2 summarizes the three core metrics across direct and steps modes.Source: §4.1 · p. 7, Table 2 · p. 7
• Direct generation beats steps-mode generation on dart_pass, bits_per_byte, and similarity_ratio for both architectures.The dart_pass gaps are 45.5 percentage points for Rainbow-Pony and 39.9 points for Qwen.Source: §4.1 · p. 8

Failure attribution

• Most steps-mode trajectories complete normally, with 81–85% stopping because the model emitted an explicit done action.Table 3 categorizes trajectory outcomes by stop_reason.Source: §4.2 · p. 8, Table 3 · p. 8
• Normally completed trajectories account for approximately 84% of Rainbow-Pony and 70% of Qwen steps-mode failures.Thus, the bulk of failures is not attributable to step-budget exhaustion or unrecoverable apply failures.Source: §4.2 · p. 8

The role of fallback edits

• Fallback-affected trajectories account for roughly 65% of Rainbow-Pony and 71% of Qwen failures within the done bucket.Trajectories requiring fallback resolution occur at less than a quarter the rate of trajectories needing none.Source: §4.3 · p. 8, Table 5 · p. 10

Matched-ID comparison

• 792 clean steps IDs were matched against direct-mode outputs on the identical sample IDs.The clean subset consisted of trajectories that were done, required no fallback, and used fewer than 20 steps.Source: §4.4 · p. 10, Table 6 · p. 10
• The matched direct subset was easier than direct mode’s full population, confirming that clean steps trajectories were not a random task sample.This selection effect should favor steps mode in the comparison rather than disadvantage it.Source: §4.4 · p. 10
• After accounting for selection, the residual gap between steps and direct was larger rather than smaller than the naive full-population gap.The matched-ID result therefore strengthens, rather than weakens, the aggregate comparison.Source: §4.4 · p. 10

Compiling outputs

• Among matched rows where both outputs passed dart_pass, direct-mode output still received higher blinded LLM-judge ratings.Table 7 covers 390 paired Rainbow-Pony rows and 688 paired Qwen rows; all six pairwise differences were significant at p < 0.001.Source: §4.5 · pp. 10–11, Table 7 · p. 10
• The matched comparisons therefore separate the raw compile-rate gap from a residual quality difference among code that compiles successfully either way.The paper describes roughly half of the raw dart_pass gap as a judge-confirmed residual quality difference.Source: §4.5 · pp. 10–11

Paired analysis

• Per-row paired comparisons flag whether steps strictly improves over direct on each metric for the same sample ID.The analysis compares rows present in both a model’s direct and steps outputs.Source: §4.6 · p. 11, §4.6.1 · p. 11

Metric caveats

• Bits_per_byte’s near-zero win rate is uninformative about editing quality, while similarity_ratio’s high win rate is considered a metric artifact.The paper notes that bits_per_byte is structurally decoupled from the steps trajectory and that similarity_ratio can be inflated by edit-base reuse.Source: §4.6.1 · p. 12

Additional predictors

• Initial file length does not explain steps-mode wins: effects are small and have opposite signs across architectures.Rainbow-Pony wins skew toward 6% longer files, whereas Qwen wins skew toward 11% shorter files.Source: §4.6.1 · p. 12

Category analysis

• Refactoring and error-handling or edge-case fixes are the only categories consistently overrepresented among steps-mode wins in both architectures.Their win counts are small, with 63–86 total wins per architecture, so other category differences are treated as noise.Source: §4.6.2 · p. 12, Table 9 · p. 12

Trajectory length

• Steps-mode wins are associated with significantly shorter trajectories in all six architecture-by-judge-dimension tests.The composite judge-majority comparison and each individual judge dimension show the same association.Source: §4.6.3 · p. 12, Table 10 · p. 13, Table 11 · p. 13
• 10–18% win rates at 1–6 steps decline to approximately 0% by 14–19 steps across the examined category and architecture breakdowns.Steps-mode judge scores decline as trajectories lengthen, while direct-mode scores on the same rows remain roughly flat.Source: §4.6.3 · p. 13

Task locality

• Task locality unifies the category and trajectory findings as one underlying variable.The paper defines locality as how spatially narrow and self-contained the required change is.Source: §4.7 · p. 13

Category and trajectory link

• Refactoring and error-handling or edge-case tasks have the two lowest mean num_steps among nine categories for both architectures.These are the same categories overrepresented among steps-mode wins.Source: §4.7 · p. 13, Table 12 · p. 13

Task locality

• Win rates and judge scores both deteriorate as trajectory length increases, while direct-mode scores stay roughly flat across the same range.The trajectory-length pattern is therefore consistent with the category pattern because the winning categories require fewer edit steps.Source: §4.7 · p. 13, Figure 4 · p. 14, Figure 5 · p. 14
• Task locality, rather than category label or architecture per se, predicts when iterative diff-based generation is competitive with direct whole-file generation.This is the paper’s unified interpretation of the matched category and trajectory analyses.Source: §4.7 · p. 16

Discussion

• Whole-file direct generation is the better default because steps-mode costs grow with trajectory length faster than token-efficiency benefits beyond a handful of steps.Longer trajectories also create more opportunities for wrong-span selection, compounded errors, and lost file-state tracking.Source: §5 · p. 16
• Diff-based generation is specifically competitive on short, spatially localized edits.Source: §5 · p. 16
• The categories where diff-based generation wins are independently the dataset’s two lowest mean-edit-step-count categories for both architectures.These categories are refactoring and error-handling/edge-case fixes.Source: §5 · p. 16
• Task locality reconciles the two regimes as complementary tools for different edit types rather than universally conflicting approaches.This framing is consistent with the adaptive-format argument.Source: §5 · p. 16

Training setup

• Steps-mode fine-tuning used roughly 10× more tokens than direct-mode fine-tuning, 50M versus 5M, yet still underperformed in aggregate.The unequal budget cuts against explaining the central finding through insufficient steps-mode training.Source: §6 · p. 16

Scope

• All results concern Flutter/Dart editing, with no claim of generalization to other languages or larger multi-file repositories.Source: §6 · p. 16

Evaluation

• Evaluation used greedy decoding and one generation per task, so pass@k and sampling-temperature effects were not measured.Source: §6 · p. 16
• Steps mode received no compiler or test feedback between edits and should not be extrapolated to feedback-driven, tool-using agents.Source: §6 · p. 16

Evaluation harness

• The first-occurrence ambiguity fallback can silently introduce errors, and a stricter harness might reduce the observed gap.Source: §6 · p. 16

Evaluation

• Blinded LLM-judge scores were not independently validated against human raters.A human-agreement spot-check is left for future work.Source: §6 · p. 17

Subgroup analyses

• Category-level win analyses contain only 63–86 total wins per architecture and report no category-level significance tests.Source: §6 · p. 17

Training setup

• qwen-direct used a non-decaying learning rate for the back half of training because of a scheduler configuration error.Its validation loss plateaued rather than continuing to settle.Source: §6 · p. 17

Conclusion

• Direct whole-file generation outperformed iterative diff-based generation on every measured metric across both model architectures.This included blinded LLM-judge assessments of code compiling successfully under both regimes.Source: §7 · p. 17
• Most of the performance gap came from normally completed trajectories that drifted semantically, especially when search targets were ambiguous.Source: §7 · p. 17
• Diff-based generation was specifically competitive on short, spatially local edits.Source: §7 · p. 17
• Task locality is proposed as the axis for deciding per edit whether whole-file or diff-based generation is appropriate.The conclusion adds an independent empirical data point supporting adaptive-format proposals.Source: §7 · p. 17

Datasets

• Four evaluation datasets are released on the Hugging Face Hub, one for each architecture/mode arm, joined by a shared sample_id.Source: §8 · p. 17

Dataset contents

• Each dataset includes the task instruction, initial file, reference final file, model output, and automated metrics.Metrics include dart_pass, Dart analysis counts, bits_per_byte, and similarity_ratio.Source: §8 · p. 18
• Steps-mode datasets include stop reasons, step counts, fallback counts, action lists, and trajectory logs.Source: §8 · p. 18
• Datasets include blinded LLM-judge scores for goal fulfillment, correctness, and code quality using gpt-4.1.Source: §8 · p. 18

Models

• The four fine-tuned checkpoints and the shared Rainbow-Pony pretrained-only base checkpoint are released.Source: §8 · p. 18

Datasets

• The two fine-tuning datasets are released, including flutter-full-examples-v1 with 14,600 rows and flutter-diff-steps-v1.Source: §8 · p. 18

Licensing

• Licenses differ by artifact type: evaluation datasets and model checkpoints use MIT, while fine-tuning datasets use Apache-2.0.Source: §8 · p. 18

Code

• Training and evaluation harness code, including decomposition, static analysis, and judge scripts, is released on GitHub.Source: §8 · p. 18