Source-linked AI summary
PatchWrite: One Line, Not One Section -- Compile-Gated, Validity-Preserving Editing for AI-Drafted Manuscripts
Weiwei Yang
TL;DR
Automated manuscript revision can repair a local defect while silently changing unrelated facts in a still-compiling PDF. PatchWrite applies bounded edits with compile and evidence gates, and its oracle and live-model evaluations show improved preservation under guarded acceptance, while exposing limits in citation correctness and evaluation scope.
Problem
Whole-section regeneration can change unrelated metrics and citations even when the resulting manuscript still compiles.
Method
PatchWrite reuses bounded EDIT N M editing and rollback, requiring fatal-log compilation checks and evidence locks for cited keys and experimental numeric tokens before committing a candidate.
Results
PatchWrite preserved the unrelated layer line in 192/192 oracle cases, while whole-slot rewriting preserved it in 0/192; writer-model candidates were accepted in 75% of cases and 93.75% fixed the injected fault.
Takeaways & Limitations
Compile-plus-evidence acceptance makes bounded manuscript mutation preserve lab-grounded facts in the evaluated oracle stress test and remain effective under live generation.
Takeaways & Limitations
The evidence gate verifies that a citation key is attested, not that it is semantically correct for the specific sentence.
Abstract
from arXiv · showhide
Automated manuscript pipelines often regenerate an entire section to repair a local defect, allowing unrelated metrics and citations to change even when the resulting PDF still builds. PatchWrite instead constrains how candidate edits become committed manuscript states: it reuses bounded EDIT N M editing and rollback, but tightens compilation acceptance with fatal-log checks and adds evidence locks that require every cited key and experimental numeric token to be attested by a reference registry or experimental log. Candidates that fail either check are rejected and the previous HEAD is retained. On a 24-manuscript x 8-fault oracle stress test (768 jobs, evenly split between compile-breaking and content-only faults), whole-slot rewriting mutated an unrelated "12-layer" line in every case (0/192 preserved; numeric Jaccard 0.6667), whereas PatchWrite preserved it in 192/192 cases. Removing the compile gate reduced acceptance to 0, while removing the evidence gate allowed a hallucinated citation to pass. The same pattern held across all eight faults. To test the protocol with generation rather than oracle edits, we reran the 192 jobs with the writer model proposing the edits. The model's candidates were accepted in 75% of cases; nearly all rejections came from one reproducible failure mode in which the model attempted to delete a line using an empty replacement unsupported by the current grammar. Every accepted candidate passed both gates, and 93.75% fixed the injected fault; the remaining cases involved a technically valid but sentence-inappropriate citation and one markup-changing near-miss. In a blind evaluation of sixteen PDF pairs, both raters preferred PatchWrite for preserving lab-grounded facts (C1 Likert 5.0 vs. 2.0), while rating prose quality nearly identically. Logs from 193 in-product drafting tasks show the same classes of failures occurring in practice.
1 Introduction
PatchWrite addresses non-monotonic manuscript revision by replacing whole-section regeneration with bounded edits accepted only when compilation and evidence checks pass. The paper evaluates this mechanism with oracle repairs, writer-model proposals, human PDF comparisons, and product logs.
- Whole-section rewriting can preserve a compiling PDF while moving an unrelated lab-grounded number, because the mutation unit is the section rather than the line.
- PatchWrite combines bounded interval editing, fatal-log compilation checks, citation and numeric evidence locks, rollback, and venue-specific manuscript shells.Its stated invariant is that the on-disk file remains the last compiling, evidence-clean manuscript.
- The writer-model experiment reruns the same 24×8 corpus to measure legal EDIT proposals, gate passage, and actual fault correction.The paper also evaluates sixteen PDF pairs with blind raters and compares product logs from 193 drafting tasks.
- 192 oracle jobs across 24 mini-articles and eight faults isolate whether surgical edits preserve unrelated tokens, whether compilation rejects bad patches, and whether evidence locks catch invented citation keys.The oracle design separates mechanism evaluation from generation quality.
- The contribution is a stricter, empirically validated acceptance predicate applied to an existing bounded-edit mechanism rather than a new edit grammar.
2 Related Work
PatchWrite extends bounded manuscript editing with compile and evidence constraints, distinguishing its correctness signal from those used by related paper-writing, code-editing, and review systems.
- Paper agents: PaperOrchestra supplies venue shells, citation policy, and claim–evidence checks, while PatchWrite replaces its whole-slot mutation operator inside ContentRefinementAgent.
- PaperSolver vs. PatchWrite: PatchWrite reuses PaperSolver’s EDIT N M grammar, immediate compilation, and rollback, but adds fatal-log compilation checks and external citation and numeric grounding.Its acceptance criterion checks cite-key membership and empirical-number support against a registry and experimental log.
- Code editing: SWE-bench and SWE-agent provide analogous patching comparisons, but PatchWrite checks pdflatex plus citation and number registries instead of a held-out test suite.The paper does not build matched infrastructure for a head-to-head comparison across the two settings.
- Concurrent work: PaperJury shares the principle that deterministic orchestration should control safety and completion logic, but targets pre-submission hardening of human-authored computer-science papers.
- Long-form synthesis: STORM and AutoSurvey generate long-form synthesis but do not provide compile-gated LaTeX editing or bind numeric tokens to laboratory logs.Their citation-coverage and outline-coherence metrics address a different question from whether revision moves an untouched number.
- Judges: Because automated reviewers can over-score generated papers, the study reports human PDF-pair preferences separately and does not use an LLM overall score as a halt condition.
3 Method
PatchWrite treats manuscript revision as a guarded state transition: a bounded EDIT creates a candidate, and only a candidate passing compilation and evidence checks can replace HEAD.
- HEAD invariant: PatchWrite replaces the current compiling document T only when every enabled gate returns true; failed candidates leave the bytes on disk unchanged.
- Edit interface: The proposer sees numbered source and emits one EDIT N M whose closed interval is replaced by the supplied body.Ranges must satisfy 1 ≤ N ≤ M and contain at most 40 lines; malformed, empty, over-long, or past-EOF edits are rejected before compilation.
- Runtime: The runtime parses one EDIT block, applies it to produce T ′, compiles with a unique job name, and accepts only a sufficiently large PDF without fatal log markers.A PDF produced by nonstopmode is insufficient by itself because syntax errors may still leave a PDF.
- Evidence gate: The evidence gate permits cite keys attested by the bibliography, citation map, or HEAD and checks every empirical number against the experimental log; either violation rolls back the candidate.
- Fallback: If parsing fails or a gate fails, fallback to slot-fence rewriting lets the job finish but reintroduces whole-slot mutation.Line numbers are regenerated after committed insertions to prevent interval drift.
- Evaluation: In the oracle design, the 0.00-versus-1.00 layer split tests whether gates preserve HEAD rather than whether an LLM finds the correct patch.The stress test contains 24 × 8 faults per condition and uses Jaccard against pre-fault HEAD.
4 Experiments
PatchWrite evaluates bounded one-line edits using compile and evidence gates, preserving unrelated manuscript content across oracle repairs while exposing live-model failure modes and rater outcomes.
- Oracle revision stress: 768 jobs across 24 manuscripts and eight faults compared whole-slot rewriting with bounded EDIT N M patches under four conditions.The corpus was evenly split between content-class and compile-class faults.
- Oracle revision stress: Removing the compile gate reduced PatchWrite acceptance to 0, while removing the evidence gate admitted an unattested citation and reduced cite Jaccard to 0.6667.The compile gate prevents replacement after forced compilation failure; the evidence gate checks citation and numeric-token grounding.
- Oracle revision stress: Both PDFs built in the bad_number case, but slot rewriting changed encoder depth from 12 to 16 while PatchWrite’s EDIT 11 11 did not.This illustrates why compilation alone cannot detect unrelated content mutation.
- Live-model EDIT proposals: With qwen3.7-plus proposing edits, 75% of 192 candidates were accepted, every accepted patch passed both gates, and fault_fixed of accepted was 0.9375.The 48 fallbacks all came from todo and undefined_cmd, where the model attempted an empty replacement unsupported by the grammar.
5 Analysis
PatchWrite’s results show that gate architecture preserves unrelated manuscript facts, while live-model performance exposes grammar and semantic limits. The analysis also identifies practical risks in fallback behavior, evidence conservatism, citation correctness, and evaluation scope.
- Validity preservation: All four added faults left layers_preserved at 0.00/1.00 and reproduced the same per-fault accept-rate split across all eight faults.The expanded test covered three compile-class faults and one additional content-class fault, with n = 24 per fault.
- Scope and risks: The compile gate must scan fatal logs because PDF existence under nonstopmode can promote broken math into HEAD.Unique jobnames and stale-auxiliary cleanup are also treated as part of the compile gate.
- Live-model behavior: 25% of 192 live-model jobs fell back to slot rewriting because todo and undefined_cmd triggered unsupported empty replacements.Those two fault types accounted for all 48 fallback jobs; the other six faults contributed zero.
- Live-model behavior: 87.5% of Kimi K2.6 proposals were accepted, and accepted patches fixed faults 91.1% of the time while preserving layers 91.7% of the time.Fallback concentrated on unclosed_env, showing that the prompt fix addressed model-specific fault types rather than the general grammar gap.
- Scope and risks: The evidence gate rejects compiling patches when numeric evidence is absent from the log, but it accepts locally attested citations that are semantically wrong for the claim.In-product snapshots recorded 45 failing files and 118 table_number_not_in_log issues; citation correctness beyond key membership remains out of scope.
6 Limitations •
The evaluation separates oracle gate behavior from writer-model generation, but the reported writer-model results are based on single-shot calls and model-specific fallback patterns.
- Evaluation scope: Table 2 isolates gate behavior with oracle repairs, while Tables 3 and 3b measure single-shot proposals from qwen3.7-plus and Kimi K2.6.The oracle design prevents generation quality from confounding Questions 1–3.
- Evaluation scope: Fallbacks concentrate on different fault types across models, so the prompt-level fix is reactive rather than general.Qwen’s failures centered on todo/undefined_cmd, whereas Kimi’s centered on unclosed_env.
7 Ethics
PatchWrite’s ethical framing keeps responsibility for manuscript claims with authors and discloses the system’s development, evaluation, and platform affiliations.
- Author responsibility: PatchWrite does not create experimental facts, and authors remain responsible for every claim in the manuscript.A compile gate can still pass a fluent falsehood when the evidence gate is disabled or the experimental log is fabricated.
- Disclosure: The manuscript discloses author-directed LLM assistance in implementation and drafting, along with the provenance of table data and generated artifacts.Table 3 records live model calls, while the human evaluation uses generated artifacts rather than a hired-annotator study.
- Disclosure: The author’s Solus affiliation is a conflict of interest because PatchWrite is deployed there and the in-product logs come from that platform.The affiliation and deployment are explicitly disclosed.
8 Conclusion
PatchWrite limits manuscript mutation to bounded edits accepted only when compilation and evidence checks pass, preserving the last valid state otherwise. Its evaluations show stronger preservation of facts, practical reader preference, and public reproducibility.
- 1.00 vs. 0.00 layers_preserved: PatchWrite preserved unrelated content across all 24 × 8 oracle jobs, unlike whole-slot rewriting.The result was reproduced at four times the scale of an earlier four-fault pass.
- 84.9% of accepted writer-model patches met the strict fault-fixed check after the deletion prompt was corrected.The initial accepted-patch rate was 75%, and the remaining cases included a sentence-inappropriate citation and a markup-changing near-miss.
- 16/16 reader comparisons preferred the surgical draft when facts had to remain unchanged, while both drafts repaired the local defect.The evaluation used sixteen PDF pairs.
- The bounded-edit, validity-predicate, and rollback pattern extends beyond LaTeX manuscripts to domains such as code review.The broader pattern replaces pdflatex and citation checks with domain-specific validity tests.
- The mechanism, evaluation harnesses, and 24-manuscript corpus are publicly available under the MIT license.The repository includes tex_patch.py, revision_stress.py, llm_edit_stress.py, and corpus.py.
A Corpus titles
The corpus contains English and Chinese-venue surrogate manuscript titles spanning machine learning, language, speech, graphs, retrieval, interfaces, and scientific prediction.
- 16 English corpus titles cover topics including vision encoders, instruction tuning, streaming ASR, retrieval, message passing, and noisy-gradient optimization.
- 8 Chinese-venue surrogates extend the corpus with ASCII-titled variants of the manuscript topics.
B Fault recipes
The fault recipe suite contains eight oracle edits divided between content-class faults that still compile and compile-class faults that break pdflatex.
- Content-class faults: Four content-class faults modify citations, numbers, TODO text, or swapped accuracy and baseline values while leaving the document compilable.The recipes are fake_cite, bad_number, todo, and swapped_acc.
- Compile-class faults: Four compile-class faults introduce broken math, unmatched braces, an unterminated quote environment, or an undefined citation.These faults cause the broken HEAD to fail pdflatex.
C Latency
Latency was similar across the compared conditions, and the authors treat these measurements as ordinary-load wall-clock observations rather than a controlled benchmark.
- 0.2209 s: PatchWrite’s mean wall time including latexmk, compared with 0.2236 s for PO-slot rewriting.
- 0.2177 s and 0.2185 s: mean wall times for the no-compile-gate and no-evidence-gate conditions, respectively.
- ±0.01–0.02 s: all three runs agreed within this range, with the ordering and qualitative claim unchanged.The authors state that compile dominates and the gates add negligible overhead.
D Per-fault uniformity
Across all eight faults, PatchWrite preserved the targeted manuscript state while whole-slot rewriting changed unrelated content. The same uniform pattern held for acceptance, preservation, and citation-grounding metrics.
- D Per-fault uniformity: n = 24 per fault × condition cell, covering eight faults across content and compile classes.The expanded pass increased each cell from n = 12 to n = 24.
- D Per-fault uniformity: 1.00 acceptance and 0.00 preservation metrics were reported for po_slot across all eight faults, with numeric Jaccard = 0.6667.The unrelated numeric content therefore was not preserved under whole-slot rewriting.
- D Per-fault uniformity: All PatchWrite preservation metrics were 1.00 across every fault.This uniform result held for both content-class and compile-class faults.
- D Per-fault uniformity: PatchWrite_nocompile had accept = 0.00, while PatchWrite_nogate had cite Jaccard = 0.6667 for every fault.Removing either acceptance safeguard produced the corresponding uniform failure pattern.
- E Per-pair rater annotation: Sixteen PDF pairs were evaluated by two raters, and PatchWrite was preferred in every pair.The evaluation doubled the original comparison from eight pairs to sixteen.