Source-linked AI summary
Learning to Commit: Generating Organic Pull Requests via Online Repository Memory
Mo Li, L. H. Xu, Qitai Tan, Ting Cao, Yunxin Liu
TL;DR
LLM coding agents can pass functional benchmarks yet produce repository-inorganic pull requests because they lack project-specific historical knowledge. Learning to Commit builds Online Repository Memory through oracle-supervised reflection on past commits, then conditions future-task generation on the resulting skills. On held-out future tasks, the framework improves organicity-related outcomes, while its current validation remains limited to one expert-maintained repository and LLM-based evaluation.
Problem
Coding agents often produce functionally correct but repository-inorganic patches because current methods do not capture project-specific conventions, internal APIs, and historical architectural context.
Method
Learning to Commit uses a strict chronological split and supervised contrastive reflection to compare blind historical-task attempts with oracle diffs, accumulating reusable repository skills for future patch generation.
Results
The skill-conditioned agent improves organicity on held-out future tasks, including file localisation, patch bloat, logic similarity, and redundancy-related judgments.
Takeaways & Limitations
Repository evolution can provide an online memory for adapting coding agents to project-specific development patterns beyond the latest code snapshot.
Takeaways & Limitations
Validation is limited to one expert-maintained industrial repository, and LLM-based judges may introduce bias or inaccuracies.
Abstract
from arXiv · showhide
Large language model (LLM)-based coding agents achieve impressive results on controlled benchmarks yet routinely produce pull requests that real maintainers reject. The root cause is not functional incorrectness but a lack of organicity: generated code ignores project-specific conventions, duplicates functionality already provided by internal APIs, and violates implicit architectural constraints accumulated over years of development. Simply exposing an agent to the latest repository snapshot is not enough: the snapshot reveals the final state of the codebase, but not the repository-specific change patterns by which that state was reached. We introduce Learning to Commit, a framework that closes this gap through Online Repository Memory. Given a repository with a strict chronological split, the agent performs supervised contrastive reflection on earlier commits: it blindly attempts to resolve each historical issue, compares its prediction against the oracle diff, and distils the gap into a continuously growing set of skills-reusable patterns capturing coding style, internal API usage, and architectural invariants. When a new PR description arrives, the agent conditions its generation on these accumulated skills, producing changes grounded in the project's own evolution rather than generic pretraining priors. Evaluation is conducted on genuinely future, merged pull requests that could not have been seen during the skill-building phase, and spans multiple dimensions including functional correctness, code-style consistency, internal API reuse rate, and modified-region plausibility. Experiments on an expert-maintained repository with rich commit history show that Online Repository Memory effectively improves organicity scores on held-out future tasks.
1 Introduction
Learning to Commit addresses the gap between functional benchmark success and maintainers’ rejection of code that is stylistically foreign, architecturally dissonant, or redundant. It uses chronological repository history to extract reusable conventions and evaluates future-task organicity with multidimensional criteria.
- Benchmark coding agents often treat issues as isolated tasks and ignore the chronological evolution of repository conventions.
- Maintainers reject many functionally correct patches because they violate project style, architecture, internal API usage, or preferred patch scope.
- Learning to Commit performs contrastive reflection on historical commits, comparing blind attempts with oracle diffs to accumulate repository-specific development skills.
- The framework conditions future pull-request generation on accumulated skills covering naming conventions, preferred abstractions, and maintainer preferences.
- Evaluation uses a strict chronological split and measures functional correctness alongside code style, internal API reuse, and modified-region plausibility.
2 Related Work
Related work has expanded from isolated coding benchmarks to repository-level tasks, historical-context systems, and continual-learning agents. However, existing approaches generally lack the active, oracle-grounded trial-and-error process used here.
- Repository-level benchmarks extend evaluation from isolated code generation to multi-file implementation and more realistic software-engineering tasks.
- Prior approaches use automated data harvesting, workflow specialization, static weight updates, passive retrieval, or unsupervised reflection to incorporate repository knowledge.
- Learning to Commit differs by combining chronological continual learning with active trial-and-error reflection against oracle changes.
3 Methodology
The methodology splits repository history chronologically, learns a reusable skill document through on-policy contrastive reflection, and conditions future task resolution on that memory. It evaluates generated patches using deterministic code metrics and pairwise multidimensional LLM judgments.
- 3.1 Problem Formulation: A strict cutoff partitions chronological commits into a history prefix for learning and a held-out future set for evaluation.
- 3.2 Repository Onboarding via Contrastive Reflection: Repository onboarding iteratively compares an agent’s blind historical-task attempt with the oracle diff and updates a skill document through contrastive reflection.
- 3.2 Repository Onboarding via Contrastive Reflection: The accumulated skill document records coding conventions, internal API usage, architectural constraints, module boundaries, and maintainer preferences.
- 3.2 Repository Onboarding via Contrastive Reflection: On-policy updates calibrate extracted patterns to the agent’s own capability gaps, targeting mistakes it would otherwise make.
- 3.3 Skill-Conditioned Resolution: Future-task resolution receives the repository snapshot, task description, and accumulated skills, while leaving tool use and skill selection to the agent.
- 3.5 Evaluation Metrics: Evaluation compares generated and oracle patches using file IoU, trajectory steps, line deviation, and pairwise LLM judgments of scope, logic, redundancy, and style.
4 Experiments
Experiments on an expert-maintained repository show that repository-specific skills improve localisation, efficiency, patch-size alignment, and several organicity dimensions, with gains varying by capability and learning configuration.
- Dataset and setup: 386 high-quality substantive commits were selected from 2,738 non-merge commits, spanning seven core development categories.The filtering and LLM-assessment pipeline yielded a 77.2% suitability rate.
- Deterministic metrics: The skill-conditioned agent achieves 80% File IoU versus 61% for the baseline in seq-all, a gain of 19 percentage points.In the same setting, it uses 56.8 versus 71.9 trajectory steps, or 21% fewer tool calls, and has lower line deviation ratio.
- LLM-judge results: The skill-conditioned agent wins against the baseline in three of four settings under both judges, reaching 54%/57% in par-bycat and 55%/58% in seq-all.In par-bycat, the advantage is concentrated in Q2 logic similarity (50% vs. 25%) and Q3 redundancy reduction (54% vs. 41%).
- LLM-judge results: Q3 redundancy and hallucination shows the most consistent skill advantage across settings, with win rates of 39–69% and high inter-judge agreement.The multi-dimensional evaluation compares scope alignment, logic similarity, redundancy and hallucination, and code style.
- Learning-mode analysis: Sequential full-corpus learning produces the highest-quality skills, while par-bycat matches its 80% File IoU through targeted category-specific skills.The results support both progressive fusion and focused, domain-aligned learning as effective curriculum assignments.
- Capability analysis: Skills provide asymmetric benefits: File IoU improves by 10–18%, Q2 reaches 50% versus 25%, while Q1 and Q4 effects remain marginal or neutral.The pattern suggests stronger help with locating changes and reusing internal patterns than with surface-level stylistic conformity.
- Case studies: A concurrency-bug case reaches File IoU = 100% with skills versus 0% for the baseline, whereas a guard-logic case shows occasional redundant caution without functional harm.The examples illustrate both improved target selection and a limited tendency toward more verbose patches.
5 Conclusion and Future Work
Learning to Commit uses historical commit adaptation to improve the organicity of coding-agent contributions. The authors identify broader repository and evaluator validation as necessary next steps.
- Conclusion: Learning to Commit distils implicit architectural constraints and coding conventions from historical commits into reusable repository memory.The framework is presented as an agentic approach for improving file localisation, reducing patch bloat, and aligning contributions with expert maintainers’ expectations.
- Future work: Generalisability remains unverified across a diverse set of high-quality, popular open-source GitHub repositories.The current validation uses a highly coupled, industrial codebase, so broader repository coverage is identified as a critical next step.
- Future work: LLM-based evaluation may introduce biases and inaccuracies, motivating more robust and verifiable metrics for alignment with human coding preferences.The limitation concerns the evaluator rather than the repository-memory learning procedure itself.
- Future work: The transfer of repository-specific skills to broader coding tasks and established evaluation settings remains an open research direction.The authors frame transfer as necessary for clarifying the scope and limits of historical adaptation.