PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Revision vs. Prior Injection: Probing Retrieval-Augmented Patent Claim Amendment

Josepha Michiko Leo, Hyun-seok Min, Yehoon Jang, Irvan Zidny, Jin-Woo Chung, Sungchul Choi

arXiv:2609.36550v1cs.CL

TL;DR

Whether retrieval grounds revisions or injects templates is rarely tested where correctness is definable. The paper uses examiner-aligned patent amendments, controlled retrieval probes, and deterministic scoring; across tested models, it finds no detectable classical prior injection, while revision locality reveals a model-specific difference.

  • Problem

    Whether retrieval substantively grounds revision or instead injects generic phrasing remains insufficiently tested where correctness has a definable ground truth.

    Source: §1 · pp. 1–2

  • Method

    The paper releases 7,385 XML-aligned prosecution cases and evaluates seven probes with a deterministic five-channel metric, contrasting random and structural retrieval under a fixed prompt scaffold.

    Source: §1 · p. 2, §4.2 · p. 5, §7 · p. 9

  • Results

    No tested model exceeds the H1 prior-injection threshold (max ∆C5 = +0.20), while revision locality reveals a model-specific difference clearest for GPT-5.4.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The framework exposes model-specific revision-locality differences that the template channel misses and can be applied to commercial stacks and other retrieval configurations.

    Source: §7 · p. 9

    The study covers one rejection-amendment round, one domain, and four models; its retrieval pool shares the test-set corpus distribution, making retrieval effects an upper bound on best-case benefit.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-augmented generation is widely used in professional writing, yet whether retrieval grounds revision or merely injects templates is rarely tested where "correct" has a definable meaning. Patent claim amendment supplies that signal: the examiner names the attacked limitation and cites prior art, providing per-case ground truth. We release three artifacts: (i) a corpus of 7,385 USPTO prosecution cases with XML-aligned pre/post claims, rejection, and cited prior art; (ii) a seven-probe battery comparing random and structural-match retrieval as two policies under a fixed prompt scaffold; (iii) a deterministic five-channel metric (C1-C3 and C5 in main, C4 supplementary) requiring no LLM evaluation. Across 9,600 pre-registered calls on four frontier LLMs (Claude Sonnet 4, Claude Haiku 4.5, GPT-5.4, GPT-4o-mini), no tested model exhibits detectable classical prior-injection behavior; retrieval effects are small and direction-inconsistent between random and structural retrieval, and the null is unchanged under a dense (semantic) retriever, across retrieval depths k in {1,3,5,10}, and under a paraphrase-sensitive grounding metric. Revision locality reveals a model-specific difference that the template channel misses. The four-cell taxonomy, which we treat as exploratory, leaves the prior-injector cell unoccupied.

1 Introduction

Patent amendment offers examiner-identified ground truth for testing whether retrieval grounds revisions or injects generic phrasing. The paper introduces an aligned corpus, controlled probes, and a deterministic metric; its initial findings show no tested model in the prior-injector cell, while revision locality reveals a model-specific difference.

1 Introduction

  • Patent prosecution provides legally defined correctness and aligns each amendment with the examiner-attacked limitation and cited-reference passage.That alignment supplies ground truth often absent from revision corpora.Source: §1 · p. 1
  • Existing patent-generation and revision work does not test retrieval’s role in grounded revision through a controlled policy contrast.The paper distinguishes grounded edits from plausible prose by targeting the attacked limitation, differentiating from cited mechanisms, and avoiding over-narrowing or boilerplate recycling.Source: §1 · pp. 1–2
  • The paper releases 7,385 USPTO prosecution cases as XML-aligned pre- and post-amendment claims, rejection, and cited prior art.The corpus supports a stratified 100-case test cohort and sampling-bias checks.Source: §1 · p. 2
  • Seven probes compare random and structural retrieval policies under a fixed prompt scaffold, with observational probes making that comparison interpretable.The contrast is evaluated alongside a deterministic five-channel metric that requires no LLM judge.Source: §1 · p. 2
  • The four tested models leave the prior-injector cell empty: random and structural retrieval produce small, often opposite-sign shifts, while GPT-5.4 shows a revision-locality difference missed by the template channel.The four-cell taxonomy is exploratory.Source: §1 · p. 2

2 Related Work

Prior patent NLP resources largely address other tasks or score text fidelity, while RAG evaluation is predominantly correlational. This paper combines examiner-aligned inputs with a controlled retrieval-policy contrast to test whether retrieved examples engage with the rejection.

2.1 Patent NLP

  • Patent-CR evaluates rejected-to-granted claim pairs for fidelity but lacks rejection and cited-prior-art inputs; other patent NLP resources target summarization, generation, classification, or evaluation rather than this controlled retrieval contrast.The paper’s corpus aligns pre/post claims, rejection, cited prior art, and amendment differences at the XML level.Source: §2.1 · p. 3

2.2 RAG evaluation and behavioral probing

  • RAG benchmarks remain predominantly correlational, whereas the F-versus-G intervention contrasts random and structurally matched retrieval within a fixed prompt scaffold.Each test case already includes the attacked limitation and cited reference, allowing retrieval-policy shifts to be examined against that specified input.Source: §2.2 · p. 3
  • The contrast distinguishes similarity-driven grounding from generic context injection, rather than identifying the unconditional causal effect of adding retrieval.The design places models in a four-cell behavioral taxonomy instead of ranking them on a single retrieval-quality score.Source: §2.2 · p. 3

3 Patent Amendment 4-Tuple Corpus

The paper releases an XML-aligned corpus linking patent claims, rejection context, cited prior art, and amendment differences. Its scale supports evaluation and retrieval, while corpus statistics show predominantly surgical amendments and substantial overlap between rejected and modified claims.

3 Patent Amendment 4-Tuple Corpus

  • The corpus contains 7,385 prosecution cases aligned at the XML level with pre/post claims, rejection, and cited prior art.It supports a 100-case test cohort, a 4,221-case retrieval pool, and a sampling-bias replication reserve.Source: §3 · p. 3

3.1 Source and alignment

  • The source cases are enriched through USPTO ODP records, and claim-number alignment with added and removed spans establishes the amendment difference used as scoring ground truth.For modified claims, the difference includes a similarity ratio and lists of added and removed spans.Source: §3.1 · pp. 3–4

3.2 Summary statistics

  • 64.5% of per-claim actions are modifications at median similarity 0.933, while 94.5% of scorable cases show overlap between rejected and post-amendment-modified claim numbers.Among 7,385 parseable triples, 5,755 admit a case-level C2 denominator.Source: §3.2 · p. 4

3.3 Release

  • The release is the only one among Patent-CR, PANORAMA, PEDANTIC, and PILOT-Bench that XML-aligns rejection context, cited prior art, pre/post claims, and amendment differences.It includes parsed JSONL and code under a permissive open license, plus an application-number index for reconstruction from USPTO ODP.Source: §3.3 · p. 4

4 Method

The method pairs seven controlled probes with a deterministic five-channel metric. Observational perturbations characterize model behavior, while a fixed-scaffold retrieval contrast compares random and structural policies without treating it as an unconditional retrieval effect.

4.1 Probe Battery

  • The seven-probe battery combines five observational probes with random-versus-structural retrieval under a fixed prompt scaffold, interpreted against the observational baseline.The F-versus-G contrast isolates switching retrieval policy, not the unconditional effect of adding retrieval.Source: §4 · p. 4, §4.1 · pp. 4–5
  • The observational probes test sensitivity to decisive-claim edits, rationale disruption, decoy citations, boilerplate, and drafting hints.They assess expected changes in grounding alignment, scope preservation, template dependence, and other channel scores.Source: §4.1 · p. 4
  • Probe G retrieves amendments matched by statute section, subsection, and a coarse attacked-limitation pattern, contrasting with Probe F’s random retrieval.Both use the same default k of 3 and prompt-insertion template; the feature-based matcher keeps the policy contrast interpretable.Source: §4.1 · pp. 4–5

4.2 Five-Channel Evaluation Metric

  • The deterministic five-channel metric uses text, parser output, and XML gold claims rather than an LLM judge, with channels designed to be jointly informative.C1–C3 and C5 are per-amendment main-paper channels; C4 is a probe-pair robustness check reported in supplementary material.Source: §4.2 · p. 5
  • C1 measures overlap between modified and rejected limitations, C2 compares generated and gold edit scale, and C3 measures preservation of the original invention core.C2 distinguishes under-editing from over-rewriting; C3 normalization isolates scope drift from mechanical narrowing.Source: §4.2 · p. 5
  • C5 counts canonical amendment phrases per 1,000 generated characters, but prior injection is assessed through retrieval-induced ∆C5 against the same case-model baseline.Because canonical phrasing is normal in drafting, C5 is a proxy and cannot alone distinguish appropriate phrasing from recycling.Source: §4.2 · p. 5
  • Construct checks and near-independence support metric validity, with all off-diagonal correlations at |ρ| ≤0.29 across 3,062 triples.The formal attorney-rating κ validation is deferred to the post-review study.Source: §4.2 · p. 6

5 Experiments

The protocol was fixed before model calls, using a stratified 100-case cohort and a separate retrieval pool. A two-stage model matrix tested preregistered hypotheses about retrieval’s effects on grounding and template dependence.

5 Experiments

  • The full experimental protocol was fixed before any model call, following recommended preregistration discipline for probe-based evaluations.The authors state this was done before any model call.Source: §5 · p. 6

5.1 Cohort and Retrieval Pool

  • The 100-case test cohort was stratified across six axes, while a non-overlapping 4,221-case retrieval pool was indexed by statute and limitation pattern.The cohort matched marginal distributions of the 4,270-case eligibility pool; the retrieval pool came from the remaining 7,285 cases after attacked-claim alignment filtering.Source: §5.1 · p. 6

5.2 Model matrix

  • Stage 1 tested Claude Sonnet 4 and GPT-5.4, with Stage 2 extending to Claude Haiku 4.5 and GPT-4o-mini under specified effect-size criteria.Each case had 8 conditions and 3 replicates per model, with retrieval k = 3; depth sensitivity tested k ∈ {1, 3, 5, 10}.Source: §5.2 · p. 6

5.3 Pre-registered hypotheses

  • H1 predicts random retrieval raises C5 by at least 0.2 without improving C1 by more than 0.1; H2 compares random and structural retrieval, and H3 predicts neither lowers C5 below baseline.H2 tests whether F and G produce statistically indistinguishable shifts in C1, C2, and C5.Source: §5.3 · p. 6

6 Results and Analysis

Across 9,600 calls on four frontier LLMs, retrieval showed no detectable prior injection, while revision locality revealed model-specific differences and the results remained stable across retrieval variants.

6.1 Pre-registered hypothesis verdicts

  • No model showed detectable prior injection: ∆C5 stayed within the preregistered ±0.2 margin, with no significant shift across eight baseline-versus-retrieval comparisons.GPT-4o-mini reached +0.20, the threshold boundary; equivalence bounds the effect but does not establish its absence.Source: §6.1 · pp. 6–7, Table 1 · p. 7

6.2 Model divergence on revision locality

  • Retrieval reduced edit magnitude in both Claude models and GPT-4o-mini, but slightly increased it for GPT-5.4; none of three nominally significant comparisons survived Holm correction.All three nominally significant C2 comparisons were decreases, and none survived Holm correction.Source: §6.2 · p. 7, Table 2 · p. 7

6.3 Where the evidence places retrieval

  • Against Probe D’s boilerplate baseline, random retrieval’s dosage was similar for three models, while Haiku 4.5 was the exception.For Sonnet 4, GPT-5.4, and GPT-4o-mini, the descriptive gap between the absolute F and D shifts was at most 0.13.Source: §6.3 · p. 8
  • The exploratory taxonomy places Sonnet 4 as an anchor-grounder, GPT-5.4 and GPT-4o-mini as structural-matchers, and Haiku 4.5 as inert or mildly prior-injecting; no prior-injector cell was occupied.Taxonomy membership uses a post-hoc interpretive overlay rather than a separately preregistered threshold.Source: §6.3 · pp. 7–8, Figure 2 · p. 8

6.4 Benchmark methodology and metric validity

  • At N=10, GPT-5.4/F ∆C5 was +0.84, reversing to −0.12 at N=300; channels were near-independent, and C4 remained a supplementary check.The strongest channel relationship was C2–C3 Spearman at −0.29; C4 carried none of the headline verdicts.Source: §6.4 · pp. 8–9, Table 3 · p. 9

6.5 Three commercial properties not confirmed

  • The three advertised properties—grounding, structural retrieval outperforming random retrieval, and more expert-like writing—were not confirmed in this instrument, which does not establish general failure.Grounding alignment shifts were at most |∆C1| = 0.014; the maximum observed |∆C5| was 0.20.Source: §6.5 · p. 9

6.6 Robustness of the null

  • The prior-injection null persisted with dense retrieval, across k ∈ {1, 3, 5, 10}, and under a paraphrase-sensitive semantic grounding metric.Across 1,200 dense-retrieval calls, no C5 increase was significant after Holm correction.Source: §6.6 · p. 9

7 Conclusion

The paper releases a probe-based framework for evaluating rejection-grounded patent claim amendment. In four models, the pre-registered tests find no detectable prior-injection behavior, while revision locality reveals model-specific differences.

7 Conclusion

  • Across four models, no model exceeds the H1 prior-injection threshold (max ∆C5 = +0.20); H2 holds only for Sonnet 4 and H3 only for Haiku 4.5.The null persists across random, structural, and dense retrieval and retrieval depths k ∈ {1, 3, 5, 10}; revision locality (C2) reveals a model-specific difference clearest for GPT-5.4.Source: §7 · p. 9
  • The released framework combines a 7,385-case USPTO corpus, seven probes, and a deterministic five-channel metric, with C4 reported as supplementary.C1–C3 and C5 carry the main-paper verdicts.Source: §7 · p. 9
  • The released framework, corpus, and scoring code support scrutiny of commercial claims and application to other retrieval configurations, where taxonomy-cell occupancy may differ.Source: §7 · p. 9

Limitations

The findings are bounded by a single-round, U.S. patent-amendment setting and four evaluated models. The same-corpus retrieval pool and statistical properties further constrain how the null and retrieval effects should be read.

Limitations

  • The study covers one rejection-amendment round, one domain, and four models; generalization to legal briefs or scientific revision remains conjectural.Source: Limitations · p. 9
  • The retrieval pool shares the test set’s corpus distribution, so observed retrieval effects are interpreted as an upper bound on best-case benefit.A different-corpus commercial retriever could yield different effects.Source: Limitations · pp. 9–10
  • The prior-injector null is a controlled-policy finding: it persists with dense retrieval, but a different retriever indexing another corpus could relocate a model.Source: Limitations · p. 10
  • Although median ∆C5 is equivalent to zero within the pre-registered margin for all four models, wide per-case variation leaves the heavy-tailed mean weakly bounded.Per-case C5 varies by ±2 to 8; the null is framed around central tendency and rank distribution rather than the mean.Source: Limitations · p. 10
  • Results are conditional on the parseable subset, and prompts truncate the examiner body in 80% of cases while retaining structured rejection information.A common 65-case subset leaves the prior-injection null unchanged; the channels are not a substitute for attorney judgment.Source: Limitations · p. 10

Ethical and legal considerations

The paper evaluates whether LLMs could perform grounded patent claim amendment, but takes no position on unassisted deployment. It releases the corpus and probes for scrutiny rather than as a recommendation to replace human counsel.

Ethical and legal considerations

  • The work evaluates whether LLMs could in principle perform grounded amendment, not whether they should draft without assistance.Source: Ethical and legal considerations · p. 10
  • The corpus and probes are released to enable scrutiny of commercial claims, not to recommend replacing human counsel.Source: Ethical and legal considerations · p. 10
  • The data come from public USPTO Open Data Portal artifacts and include inventor and attorney names already present in the public record; the authors do not re-identify or link filings beyond that source.Source: Ethical and legal considerations · p. 10
  • The evaluated commercial models provide the experimental outputs; AI assistants were used in manuscript preparation for coding, language, and typo editing, not research ideation, design, or analysis.Source: Ethical and legal considerations · p. 10

Reproducibility Statement

The appendix documents released reproduction artifacts, corpus construction and parsing, corpus characteristics, run costs, and worked examples that clarify what the evaluation channels capture.

Reproducibility Statement

  • The release includes the 7,385-case parsed corpus, seven probe templates, metric implementation, model-call driver, scoring pipeline, and analysis scripts.The enrichment chain resolves ODP records and selects pre-amendment claims, the first non-final rejection, and the subsequent claims filing.Source: Reproducibility Statement · pp. 10–11, §A.1 · p. 13

A.2 XML schema handling

  • Three XML claim-document schema families require format-specific parsing, while rejection documents provide structured fields for claims, statutes, rejection types, and cited references.Legacy bracket markup and newer insertion/deletion tags are handled alongside rejection-document parsing.Source: §A.2 · p. 13

A.3 Parser hardening

  • A parser-hardening pass raised cohort coverage from 76/100 to 100/100 through sequential schema and claim-matching fixes.The same hardening recovered several thousand additional cases in the full corpus.Source: §A.3 · p. 13

A.5 Relation to other corpora

  • The corpus aligns rejection context, cited prior art, pre/post claims, and XML-level amendment diffs, unlike cited corpora that provide only claim pairs or target other tasks.Source: §A.5 · p. 14

B Per-model cost estimates

  • The 9,600 model calls cost approximately $220 across two stages, with public USPTO API access and local compute adding no infrastructure cost.Source: B Per-model cost estimates · p. 14

C. Worked examples per channel

  • Worked examples show C1 tracking alignment with attacked limitations, C2 distinguishing under-editing from near-complete rewriting, and C3 detecting invention-core scope collapse.C5 distinguishes retrieval-induced canonical-phrase reuse from domain-natural repetition; BLEU, ROUGE, and BERTScore fail to register the same within-case directions in the examples.Source: §C.1 · p. 14, §C.2 · p. 14, §C.3 · p. 14

C.6 Case β: GPT-5.4 F-vs-G sign flip

  • For GPT-5.4 on the same case, random retrieval raises C5 by +2.25 while structural retrieval lowers it by −1.50; the other models show different patterns.The example demonstrates that retrieval-policy choice can change both magnitude and sign of the template-dependence effect.Source: §C.6 · pp. 14–15

D Sample-size reversal

A smoke-run estimate for GPT-5.4 under random retrieval reversed sign when the verdict used the larger run. The authors therefore caution against relying on smoke-run effect sizes.

D Sample-size reversal

  • +0.84 at N=10 reversed to −0.12 at N=300 for GPT-5.4 under random retrieval, changing the cell from apparent H1 support to the opposite sign.The authors attribute the reversal to sampling variability in the 100-case cohort, not calibration.Source: D Sample-size reversal · p. 15
  • The N=10 smoke run did not systematically replicate every model-condition cell, so the documented reversal is not a cell-by-cell stability sweep.The authors recommend at least N=100 per condition with replicates rather than sizing runs from smoke-run effects.Source: D Sample-size reversal · p. 15

E Probe-D anchor: per-model breakdown

The section compares random retrieval with a boilerplate-injection reference, then reports robustness checks, evaluation details, and corpus and cohort design. Its results do not show a consistent template-inflation effect across models or tested retrieval settings.

E Probe-D anchor: per-model breakdown

  • For three models, random-retrieval and boilerplate-shift magnitudes differ by at most 0.13; Haiku 4.5 is the exception, with +0.19 versus |∆D|=0.09, still below H1’s +0.2 threshold.The comparison is descriptive; the per-model magnitudes are reported individually rather than averaged.Source: E Probe-D anchor: per-model breakdown · p. 15, Table 4 · p. 15

F Probe x channel full table

J Attorney validation study

N.1 Why 100 cases

  • The released corpus contains 7,385 aligned cases; the 100-case cohort was sized and stratified to support within-model comparisons and minimum cell counts, using iterative marginal matching on eligible cases.The cohort excludes cases with dismissed or remanded outcomes and obvious data defects; the achieved batch-0 marginals are reported against targets.Source: M Corpus release index · p. 17, §N.1 · p. 17, §N.2 · p. 17

O Significance tests and equivalence

  • 0 of 8 C5 baseline-versus-retrieval comparisons were significant, and median-based TOST found equivalence to zero within ±0.2 for every model and probe.For C2, 3 of 8 comparisons were nominally significant decreases, but none survived Holm correction.Source: Table 6 · p. 18, O Significance tests and equivalence · p. 18

P Paraphrase-sensitive (semantic) C1

  • Under paraphrase-sensitive semantic C1, 0 of 8 retrieval comparisons survived Holm correction; the two uncorrected near-misses were decreases rather than grounding gains.Lexical and semantic C1 were only loosely concordant (Spearman ρ = 0.37).Source: P Paraphrase-sensitive (semantic) C1 · p. 18

Q Dense-retriever realization of Probe G

  • Dense-G retrieved more semantically similar exemplars than F or G (cosine 0.87 vs ∼0.60), yet 0 of 16 comparisons survived correction and no model showed a significant C5 increase.Dense-G swapped only the retrieval mechanism while keeping the pool, k, exemplar fields, insertion format, and base prompt fixed.Source: Q Dense-retriever realization of Probe G · pp. 18–19

R Retrieval-depth sensitivity

  • Across k ∈ {1, 3, 5, 10}, C5 showed no monotonic increase and the pre-registered H1–H3 verdicts stayed qualitatively unchanged.For seven of eight model-condition series, maximum C5 deviation from k=3 was ≤0.18; the larger GPT-4o-mini random-F deviation was non-monotonic.Source: R Retrieval-depth sensitivity · p. 19, Table 8 · p. 19

S Truncation and parse robustness

The robustness checks find that truncation clips the tail of examiner prose but preserves structured rejection information used for scoring. The prior-injection verdict also holds on the common parsed subset, while a smaller complete-rejection subset is not used to adjudicate it.

S Truncation and parse robustness

  • 80% of rejection bodies exceed the 8,000-character cap, but structured rejection instances remain unclipped and retain the attacked claims, statutory basis, and cited references.The attacked-claim identification used by C1 comes from those structured instances, not the clipped tail of the verbatim body.Source: S Truncation and parse robustness · p. 19
  • On the 65 cases parsed by all four models under every condition, no model reached the ∆C5 ≥0.2 threshold; the maximum was +0.10 for GPT-4o-mini.Per-model parse success was 97.4% overall, and the common-subset check leaves the prior-injection verdict unchanged.Source: S Truncation and parse robustness · p. 19
  • The 20 untruncated-rejection cases were not used to adjudicate H1 because their C5 estimates were dominated by small-sample variance.The authors point to the intact structured rejection signal and the common-subset check as the more appropriate robustness evidence.Source: S Truncation and parse robustness · p. 19

Related papers

Loading 2609.36550v1…