PaperlayerView exact revision on arXiv

Source-linked AI summary

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

Yubo Wang, Jiarong Liang, Yuxuan Zhang, Xuye Liu, Cong Wei, Yuyu Zhang, Ping Nie, Wenhu Chen

arXiv:2607.12463v3cs.AIcs.CL

TL;DR

Coding-agent post-training starts from models not specifically optimized for conditioning on external tool returns and continuing from them. The paper introduces function-aware FIM mid-training, selecting functions by program structure and complexity–inferability, then reports consistent coding-agent gains and broader capability recovery after post-training. Its evidence is scoped by Python-only data and limited cross-base validation.

  • Problem

    Conventional code pretraining under-exposes the function-level conditioning structure that coding agents need when continuing after externally produced tool returns.

  • Method

    Function-aware FIM mid-training masks functions or connected function groups selected by dependency-graph analysis and a complexity–inferability criterion, embedding chain-of-thought rationales before code.

  • Results

    +2.8 and +3.0 points on SWE-Bench-Verified for Qwen2.5-Coder-Instruct 7B and 14B, with gains across post-training pipelines, Qwen3-8B, and non-target benchmarks.

  • Takeaways & Limitations

    The same Python-only mid-training corpus transfers a function-call inductive bias across coding-agent, non-agent coding, and non-coding tool-use evaluations after post-training.

  • Takeaways & Limitations

    The corpus and in-domain agent benchmarks are Python-only, while cross-base evidence comes from one Qwen3-8B configuration that also changes the post-training pipeline.

Abstract

from arXiv · show

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.

1 Introduction

The paper identifies a training-time gap between conventional code pretraining and the conditioning structure required by coding agents. It addresses this gap with function-aware FIM mid-training and finds consistent coding-agent gains plus recovery of broader capabilities after agentic post-training.

  • Motivation: Coding agents need to continue reasoning from histories, actions, and externally produced observations, a structure that ordinary left-to-right code pretraining under-exposes.Function call sites provide the analogous context, call, return, and downstream-consumption structure at internet scale.
  • Motivation: Random-span FIM is poorly aligned because its boundaries are syntactically arbitrary, provide no reasoning supervision, and dilute the structural signal during general pretraining.The proposed recipe targets functions, embeds chain-of-thought rationales, and places the objective in dedicated mid-training.
  • Approach: Function-aware FIM selects masking targets through program dependency graphs and a complexity–inferability criterion, with reasoning embedded inside the masked span.The objective is applied immediately before agentic post-training.
  • Results: +2.8 and +3.0 points on SWE-Bench-Verified for Qwen2.5-Coder-Instruct 7B and 14B, respectively, after mid-training.The gains hold across two post-training pipelines, with +5.3 points for the SWE-Smith pairing on the same 7B base.
  • Results: +3.2 points on SWE-Bench-Verified for Qwen3-8B with SWE-Lego, indicating the result is not confined to one Qwen2.5-Coder configuration.This comparison jointly varies the base model and post-training pipeline, so it is evidence rather than a guarantee across model families.
  • Robustness: +11.1 on LiveCodeBench, +2.4 on BFCL, and +3.9 on τ-bench after mid-training before agentic post-training.The Python-only corpus produced recovery on non-target coding and non-coding tool-use benchmarks after post-training regressions.

2 Method

The method builds structurally selected function-level FIM targets from Python repositories using dependency analysis, complexity, inferability, and difficulty-aware scoring. It then trains the model to generate a rationale followed by code for selected functions or connected function groups.

  • Data: The corpus contains approximately 400K FIM samples and 2.6B tokens from 968 decontaminated Python repositories.Repositories overlapping SWE-Bench sources and commits after the benchmark base dates are removed.
  • Target Selection: Each source file yields single-function or connected 2–3-function mask targets through dependency-graph construction, complexity scoring, inferability scoring, and threshold selection.The multi-function variant accounts for patches spanning related functions and recomputes inferability under joint masking.
  • Dependency Graph: AST analysis constructs call edges between callers and callees and sibling edges between methods of the same class.Sibling edges capture intra-class coupling through shared instance state rather than direct calls.
  • Complexity Score: The complexity score combines normalized lines of code, cyclomatic complexity, and maximum control-flow nesting depth.The components are weighted and normalized by soft caps.
  • Inferability Score: The inferability score aggregates caller specificity, callee count, signatures, documentation, and class-sibling signals as proxies for contextual predictability.Hand-designed proxies avoid coupling target selection to a particular reference model.
  • Target Scoring: The single-function score combines complexity and inferability in a harmonic-mean-like form and applies a one-sided difficulty penalty.This favors targets that are both complex and recoverable while down-weighting targets that remain hard even with full context.
  • Sample Construction: Gemini-3-Flash generates a rationale and candidate body from the masked file, a separate judge filters the pair, and retained samples place rationale before body in the FIM span.The ground-truth body is used as a filtering anchor, not as part of the training target.

3 Experiments

Experiments show that function-aware FIM mid-training consistently improves coding-agent benchmarks across model sizes, post-training pipelines, and a different base model, while recovering capabilities eroded by agentic post-training. Ablations further identify function selection and multi-function masking as important design choices, with rationale supervision contributing additional gains.

  • Main coding-agent results: +5.30 points on SWE-Bench-Verified with SWE-Smith versus +2.80 with R2E-Gym on the same 7B base.SWE-Bench-Lite gains are +0.50 with SWE-Smith and +3.67 with R2E-Gym, so benefit magnitude depends on the pipeline.
  • Main coding-agent results: +3.00 points on SWE-Bench-Verified for Qwen2.5-Coder-14B-Instruct with R2E-Gym, alongside a +4.00-point SWE-Bench-Lite gain.The corresponding 7B gains are +2.80 on Verified and +3.67 on Lite.
  • Main coding-agent results: +3.20 Verified and +5.40 Lite points on Qwen3-8B with SWE-Lego show transfer beyond the Qwen2.5-Coder-Instruct configurations.This comparison varies the base model and post-training pipeline jointly.
  • Capability preservation and transfer: +11.10 on LiveCodeBench and +2.40 on BFCL restore capabilities after R2E-Gym post-training, while the six-benchmark average rises from 16.04 to 19.56.The same mid-training run preserves the SWE-Bench-Verified improvement.
  • Ablations: Removing CoT still lifts the ablation average by +1.18, while self-CoT recovers 1.68 of the +2.43-point Gemini-3 gain.These results indicate that FIM structure contributes independently of frontier-teacher rationale distillation.
  • Ablations: The full complexity–inferability selection reaches 15.60, and the 80%/15%/5% single/pair/triple mixture reaches 16.10.Pair masking improves the average from 15.60 to 15.90, whereas substituting triple targets is nearly neutral at 15.70.

4 Analysis

The analysis compares post-training-only and mid-training-plus-post-training agents, finding better recovery from negative tool feedback and concentrated gains on multi-function tasks within a file.

  • 4.1 Recovery from Negative Observations: 28.8% of error-containing trajectories reach a passing patch with mid-training, versus 24.8% for the baseline.+4.0 percentage points and +16% relative recovery, despite similar negative-feedback exposure: 91.8% versus 88.8%.
  • 4.1 Recovery from Negative Observations: Mid-training increases edits per solved task from 3.3 to 7.4 and trajectory length from 15.1 to 23.6 steps.This shift is described as an iterate-and-verify policy.
  • 4.2 The Gain Concentrates on Multi-Function Reasoning: 22.7% of multi-function single-file tasks are solved with mid-training, compared with 13.6% for the baseline.The absolute gain is +9.1 percentage points, more than four times the +2.1-point gain on single-function tasks.
  • 4.2 The Gain Concentrates on Multi-Function Reasoning: Multi-file tasks are not differentially helped, which the analysis attributes to function-aware FIM operating within files.The strongest gains occur where tasks require control- and data-dependency reasoning among functions inside one file.

5 Related Work

The paper builds on mid-training, structure-aware FIM, coding-agent foundation models, and teacher distillation, while combining these ideas into a dedicated function-aware objective.

  • Mid-training and continued pretraining: Mid-training adds a stage between pretraining and post-training for installing inductive biases that generic web text or fine-tuning may not provide.The paper follows this staging philosophy while targeting an agent-oriented structural prior.
  • Fill-in-the-middle and structure-aware code objectives: Structure-aware FIM methods move beyond random spans through AST masking, planning signals, or instruction-aware slots.The paper instead selects function-level targets with PDG analysis and a complexity–inferability criterion, embeds CoT, and uses dedicated mid-training.
  • Fill-in-the-middle and structure-aware code objectives: Repository-level methods such as GraphCoder and DRACO exploit program dependencies at inference time, unlike this recipe's mid-training objective.The distinction is part of the paper's positioning relative to dependency-aware code methods.
  • Coding agent foundation models: Coding-agent research spans broader benchmarks, varied scaffolds, and trajectory-centric post-training pipelines.The paper evaluates its recipe across representative pipelines including R2E-Gym, SWE-Smith, SWE-Lego, and related systems.
  • Distillation from frontier models: The use of Gemini-3-Flash for rationale generation places the recipe within teacher and CoT distillation, but the paper reports it is not distillation-bound.The CoT-source ablation supports this positioning.

6 Limitations and Discussion

The paper scopes its conclusions through limitations involving language coverage, teacher dependence, cross-base validation, and assumptions about code modularity.

  • Scope boundaries: The corpus and in-domain evaluation are Python-only, leaving transfer to Java, C++, and Rust for future work.Cross-language evidence is only indirect through FullStackBench-EN.
  • Scope boundaries: The default CoT recipe depends on Gemini-3-Flash, while self-generated rationales recover most of the gain.A fully open-source replication would require a comparably strong open teacher.
  • Scope boundaries: Cross-base validation uses one Qwen3-8B with SWE-Lego configuration, which also changes the post-training pipeline.This indicates the recipe is not tied to one combination but does not guarantee transfer across all base families.
  • Scope boundaries: Function-aware FIM presupposes modular code and may provide fewer eligible targets for monolithic scripts, generated code, or notebooks.These regimes are not studied systematically.

7 Conclusion

The conclusion frames coding-agent steps and function calls as structurally aligned, then uses function-aware FIM mid-training to install an agent-relevant prior.

  • 7 Conclusion: A coding-agent step and a function call share context, action, externally produced return, and continuation.This alignment motivates using ordinary source code as an internet-scale supply of agent-relevant signal.
  • 7 Conclusion: Function-aware FIM mid-training masks functions selected by PDG analysis and a complexity–inferability criterion, with CoT embedded in the middle span.The method is presented as a self-supervised stage for installing the corresponding structural prior.

A.1 License Inventory

The corpus covers 968 GitHub repositories, with licenses and repository metadata released alongside the corpus. Its selection pipeline includes quality filtering, dependency-based target selection, and fixed threshold parameters.

  • Every corpus license permits at least non-commercial research use, and small license categories are aggregated as “Other research-permissive licenses.”
  • 968 repositories span ten topic categories, with reference implementations, scientific computing, and small frameworks dominating while compiler and networking/security tails preserve coverage diversity.
  • The corpus statistics are reported after decontamination and quality filtering, using the Qwen2.5-Coder tokenizer for token counts.
  • Algorithm 1 combines program dependency graphs, complexity and inferability scores, a difficulty penalty, and hard filters to select FIM targets.
  • The selection pipeline uses fixed file-length, function-length, complexity, and FIM-score thresholds throughout.

B.2 Program Dependency Graph: Call Resolution

The method constructs Python program dependency graphs by resolving function calls and sibling relationships, then selects structurally complex yet contextually inferable FIM targets. Inferability combines caller, callee, signature, documentation, and class-context signals, while residual difficulty is penalized asymmetrically.

  • Call Resolution: Function-call edges resolve direct module calls, class instantiation, and self/cls method calls within the qualified-name index.
  • Call Resolution: Unresolved calls use short-name matching against the qualified-name index, recovering cross-module references with controlled false positives.
  • Target Selection: Algorithm 1 filters files and functions by size, excludes dunder methods and low-complexity functions, then ranks surviving targets by FIM score.
  • Call Resolution: Sibling edges connect every pair of methods in the same class, capturing structural relationships beyond direct calls.
  • Inferability Score: Inferability combines caller specificity, intra-file callees, signature annotations and name descriptiveness, docstrings, and class context.
  • Difficulty Penalty: The difficulty penalty leaves targets with residual difficulty ∆(v) ≤ τd unchanged and applies Gaussian damping above the threshold.
  • Multi-Function Groups: For groups of connected functions, the score combines coupling, group complexity and inferability, and a difficulty penalty; coupling reflects intra-group dependencies and shared state.

C Training Hyperparameters

The experiments apply FIM mid-training and existing agentic post-training pipelines across multiple models, with evaluations using repeated runs. Mid-training improves target benchmarks and changes agent behavior toward more active iteration, while reducing empty-patch failures.

  • Training Setup: FIM mid-training uses LLAMAFACTORY, while R2E-Gym, SWE-Smith, and SWE-Lego provide the agentic post-training pipelines.
  • Training Setup: The FIM hyperparameters are applied uniformly to Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-14B-Instruct, and Qwen3-8B.
  • Training Setup: SWE-Lego uses 2 epochs instead of its official 4 to prevent overfitting in this setup.
  • Compute: Reproducing the full experiments requires approximately 30 days, or 5,760 GPU-hours, on an eight-H100-GPU node.
  • Evaluation: SWE-Bench-Verified evaluations report run-means over three independent runs, with 500 instances per run.
  • Evaluation Results: +9.1 pp is the largest gain on multi-function single-file tasks, compared with +2.1 pp on single-function tasks; multi-file tasks remain at approximately 11.3% for both checkpoints.
  • Behavioral Analysis: Search actions fall from 15.3% to 11.0%, while execute_bash actions rise from 19.5% to 24.6% after mid-training.
  • Behavioral Analysis: Mid-training reduces no-patch failures from approximately 11 to approximately 1 trajectory per run, while localization and patch-error counts remain nearly unchanged.

D.6 Multi-File Tasks Are Not Differentially Helped

Function-aware FIM mid-training does not differentially improve tasks whose gold patches span multiple files. The paper attributes this boundary to training at function level within individual files, leaving cross-file coordination directly untrained.

  • On 71 Verified tasks with gold patches spanning at least two files, both checkpoints solve approximately 11.3% on average and are balanced head-to-head.
  • The paper attributes the multi-file boundary to function-level, within-file FIM training and proposes extending selection to cross-file function pairs.
  • A representative premature-termination case shows the baseline finishing after one step with an empty edit, while the mid-trained agent iterates through tool feedback and reaches a passing patch.

E Behavioral Analysis on SWE-Bench-Lite

On SWE-Bench-Lite, mid-training improves recovery and encourages more iterative editing while eliminating no-patch failures. Gains are concentrated in single-function, single-file tasks; the multi-function bucket shows no improvement.

  • Trajectory-level behavior: 18.8% to 22.1%: mid-training raises the recovery rate on SWE-Bench-Lite.The analysis uses the same trajectory-parsing pipeline and failure taxonomy as Verified, with run-means over three evaluation runs.
  • Trajectory-level behavior: 4.4 to 7.8 str_replace operations per task: solved trajectories undergo more edit iterations after mid-training.This behavioral change accompanies the higher recovery rate on Lite.
  • Failure modes: ∼8 to 0 trajectories per run: mid-training eliminates the no-patch failure mode.Localization errors fall by about 11, while patch errors rise by about 7, indicating a shift toward iteration and verification.
  • Gold-patch stratification: +4.0 pp: the end-task improvement comes entirely from the single-function single-file bucket, which gains +4.9 pp.The multi-function single-file bucket is small (n=54) and shows no gain; the single-function bucket contains n=246 tasks.
  • Per-instance comparison: ∼1.8× overall and ∼2.1× on single-function single-file tasks: per-instance head-to-head outcomes favor mid-training on Lite.The multi-function single-file bucket is balanced, consistent with its small absolute size.
Loading 2607.12463v3…