PaperlayerView exact revision on arXiv

Source-linked AI summary

PatentEval: Understanding Errors in Patent Generation

You Zuo, Kim Gerdes, Eric Villemonte de La Clergerie, Benoît Sagot

arXiv:2406.06589v2cs.CLcs.AI

TL;DR

Patent generation and evaluation remain challenging because legal text demands precision and expert assessment. PatentEval combines a human-annotated benchmark with an error typology for abstract and next-claim generation. ChatGPT minimized error quantity and diversity across both tasks, while results also show model-specific errors and differences.

  • Problem

    Patent-text generation and quality evaluation remain underexplored because legal precision requires domain expertise to judge generated text.

    Source: §1 · p. 1

  • Method

    PatentEval combines a human-annotated benchmark, pairwise model comparisons, and an error typology for claims-to-abstract and next-claim generation.

    Source: §1 · p. 1, §4.1 · pp. 3–4

  • Results

    ChatGPT excelled at minimizing error quantity and diversity across both task outputs; actual claims-to-abstract fine-tuning maintained relevance, while models showed other errors.

    Source: Q1: What are the error distributions… · pp. 6–7

  • Takeaways & Limitations

    The study's findings identify model strengths and common errors, informing future model enhancement and consideration of AI in patent drafting.

    Source: §8 · p. 9

    Comparisons may be biased because recent LLMs may have encountered USPTO patent documents during pretraining, potentially recalling rather than generating content from learned patterns.

    Source: §9 · p. 10

Abstract

from arXiv · show

In this work, we introduce a comprehensive error typology specifically designed for evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones. We have also developed a benchmark, PatentEval, for systematically assessing language models in this context. Our study includes a comparative analysis, annotated by humans, of various models. These range from those specifically adapted during training for tasks within the patent domain to the latest general-purpose large language models (LLMs). Furthermore, we explored and evaluated some metrics to approximate human judgments in patent text evaluation, analyzing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.

1 Introduction

Patent drafting is costly and demanding, while evaluating generated patent text remains underexplored and requires domain expertise. PatentEval provides a human-annotated benchmark for comparing models and analyzing their errors.

1 Introduction

  • Patent drafting is complex and time-consuming, requires knowledge of the invention and relevant law, and accounts for the biggest part of patent cost.Obtaining a patent can also be costly.Source: §1 · p. 1
  • Deep learning has advanced patent prior-art search and classification, while patent-text generation and its evaluation remain less explored.Prior-art search: Risch et al. (2020); Buckley (2021); Vowinckel and Hähnke (2023). Classification: Lee and Hsiang (2019b); Huang et al. (2019).Source: §1 · p. 1
  • Patent text generation and quality assessment remain underexplored because legal precision demands domain expertise to evaluate generated text.These demands create hurdles for researchers from allied disciplines.Source: §1 · p. 1
  • PatentEval is a human-annotated benchmark for assessing models through pairwise comparisons and detailed error-type analysis.The study aims to clarify potential uses of large language models in patent drafting.Source: §1 · p. 1

2 Preliminaries and Background

Patents contain distinct sections, and this study focuses on abstracts and claims. Claims define an invention's features and rights, while their dependencies shape how claims relate to one another.

2 Preliminaries and Background

  • A patent contains sections including an abstract, description, claims, drawings, and classification information; this study focuses on abstracts and claims.The abstract gives a concise overview for information and search purposes.Source: §2 · p. 2
  • Claims define an invention's features and associated rights, but drafting them is a strategic choice that is not wholly automatable.Systems proposing subsequent claims may help practitioners assess claim-set quality and completeness.Source: §2 · p. 2
  • Independent claims state core features without referring to other claims, whereas dependent claims refer to and build on prior claims.Dependent claims include the referenced claims' limitations and add details, variants, or features.Source: §2 · p. 2

3 Related Work

Prior work adapted language models for patent generation and explored several evaluation approaches, but variation in datasets and methods complicates consistent comparisons. This study benchmarks contemporary models and examines their error characteristics and distributions.

3 Related Work

  • PatentTransformer adapted GPT-2 for claim generation and later expanded to generate patent sections from other sections, such as titles or claims from abstracts.The project framed claim generation as an “augmented inventing” tool for patent writers (Lee and Hsiang, 2020b).Source: §3 · p. 2
  • Prior evaluations used claim-segment relevance, quality-control models, and prior-art searches to assess generated content and investigate its origins (Lee and Hsiang, 2019a; Lee and Hsiang, 2020c).These approaches include a two-Transformer framework and an autocomplete feature for idea exploration.Source: §3 · p. 2
  • The Patent Generative Transformer addressed multiple patent tasks, while its evaluation covered only 44 chemistry patents and remained unpublished.PatentGPT-J work also proposed a metric based on keystroke reduction for claim autocomplete.Source: §3 · pp. 2–3
  • Differences in evaluation methods and datasets across patent-generation studies make their results difficult to compare consistently.Existing resources include Big Patent and HUPD, which includes task-specific benchmarks and metrics such as ROUGE for summarization (Sharma et al., 2019).Source: §3 · p. 3
  • Prior work highlighted factual-consistency challenges and proposed QAGS, FactCC, and FactGraph as alternative evaluation methods.Source: §3 · p. 3
  • This study comparatively assesses contemporary models on claims and abstracts and analyzes the characteristics and distribution of their errors.The motivation includes recent LLM capabilities for zero-shot and instruction-following tasks (Touvron et al., 2023; Penedo et al., 2023).Source: §3 · p. 3

4 Tasks and Criteria

The study evaluates patent generation through two tasks and a criteria framework grounded in observed errors and patent-drafting guidance. It assesses abstract and claim errors while excluding novelty and non-obviousness from claim evaluation.

4.1 Tasks

  • Claims-to-abstract generation uses the full claim set to produce an abstract, while next-claim generation produces the entire subsequent claim rather than incremental text.For next-claim generation, inputs include claim 1, claims 1–2, or claims 1–3; the target claim should match the original's independent or dependent type.Source: §4.1 · pp. 3–4

4.2 Typology of Errors

  • The error typology is based on issues observed in model outputs and refers to the WIPO Patent Drafting Manual.The study also bases its work primarily on USPTO drafting standards.Source: §4.2 · p. 4

4.2.1 Abstract Generation

  • Abstract-generation errors include grammar, irrelevant content, incomplete coverage, excessive length, contradictions, unclarity, and ineffective summarization.Incomplete coverage includes omitting essential concepts, especially from the main claim; ineffective summarization includes reproducing claims rather than providing a concise overview (WIPO, 2022, pp. 106–107).Source: §4.2.1 · p. 4

4.2.2 Claim Generation

  • Claim evaluation focuses on inherent structure without external databases, leaving novelty and non-obviousness assessment for future work.The claim typology also covers preamble and transitional-phrase errors, disconnected claim bodies, and dependency problems such as unclear or overly broad dependent claims.Source: §4.2.2 · pp. 4–5
  • Claim errors also include vague language, antecedent-reference and terminology problems, wishful claiming, and verbose or sub-optimal claim structure (WIPO, 2022).Source: §4.2.2 · p. 5
  • The claim typology includes irrelevant matter, contradictions, repeated claims, grammatical errors, punctuation discrepancies, and formatting errors such as incorrect numbering.Source: §4.2.2 · p. 5

5 Dataset Creation

The evaluation uses granted patents sampled across IPC sections, assesses specialized and general-purpose models, and collects paired human judgments of model outputs.

5.1 Data Selection

  • 400 granted patents from 2017–2018 were sampled, with 50 from each of eight IPC 6 sections; canceled claims were excluded.Claims from these patents supplied inputs for both Claims2Abstract and next-claim generation.Source: §5.1 · pp. 5–6

5.2 Models under Evaluation

  • The evaluation includes patent-specialized models and general-purpose LLMs, including GPT-3.5-turbo-0613, with inference temperature set to 0.Most selected models use decoder-only architectures; the patent-summarization model from Suzgun et al. (2022) uses an encoder-decoder T5 architecture.Source: §5.2 · p. 6

5.3 Annotation Data

  • Human evaluation covered 50 patents each from IPC domains A and G, plus eight examples from other domains for next-claim dependency evaluation.Outputs for the other six domains were reserved for future studies.Source: §5.3 · p. 6
  • The annotation objectives included assessing model capabilities and examining whether evaluators preferred human- or machine-generated content.Source: §5.3 · p. 6
  • Annotators compared outputs from two different models and separately compared a model output with the original abstract or next claim.Model pairs were selected randomly, and the model chosen for comparison with the original was selected independently.Source: §5.3 · p. 6

6 Results and Analysis

The analysis finds that machine outputs can match or outperform human drafts on both patent-generation tasks, while revealing distinct error patterns across models and tasks. Human-drafted abstracts that lose to machine outputs often show incomplete coverage or reflect the timing of drafting relative to later patent amendments.

6 Results and Analysis

  • Two annotators—a patent lawyer with over 15 years of experience and a PhD student—annotated outputs, with a third expert resolving disagreements by consensus.Source: §6 · p. 6

Q1: What are the error distributions of each model?

  • ChatGPT made no errors on claims-to-abstract generation and minimized errors across both tasks, while larger LLMs tended toward fewer error varieties but still missed invention scope or produced verbose abstracts.Falcon-40b often repeated or over-expanded abstracts, while Llama2-70b struggled to capture invention scope; HUPD T5-small’s limited context led it to overlook claim components and include non-factual information, though fine-tuning on actual claims kept its abstracts relevant.Source: Q1: What are the error distributions… · pp. 6–7
  • Next-claim outputs commonly had punctuation and grammar problems; models also repeated prior claims without added specificity, missed dependencies, or misnumbered and misreferenced claims.PatentTransformer and PatentGPT-J also struggled with claim formatting, antecedent references, and preamble consistency; these issues often co-occurred.Source: Q1: What are the error distributions… · p. 7

Q2: Do machines perform better in patent drafting?

  • In claims-to-abstract generation, ChatGPT, Falcon-40b, pgt, and HUPD T5-small had winning rates above 50% against human abstracts; ChatGPT also exceeded or matched human preference in next-claim generation more than 50% of the time.Llama2-7b matched human preference at 50% in next-claim generation.Source: Q2: Do machines perform better in… · pp. 7–8, Figure 2 · p. 8

Q3: When do human drafters perform worse than certain models?

  • When human abstracts lost to machine outputs, they consistently showed incomplete coverage, which the authors attribute to drafters’ strategic choices about search visibility.Including keywords from elsewhere in the claim tree can increase discoverability, whereas copying only claim 1 can decrease it.Source: Q3: When do human drafters perform… · p. 8
  • Human abstracts’ irrelevant content and contradictions may reflect timing: abstracts are usually drafted at filing, while the study used granted patents whose claims may have been amended later.An abstract generally remains as originally filed unless formally amended with examiner and applicant approval.Source: Q3: When do human drafters perform… · p. 8

7 Evaluation of Metrics

The authors compare automated metrics with human pairwise judgments for abstracts and next-claim generation. Metric alignment varies by task, with technical-term coverage and SemSim correlating with abstract judgments, and a rule-based checker performing best for claim generation.

7 Evaluation of Metrics

  • Metrics were applied to the same input-output pairs reviewed by human annotators, and their pairwise rankings were compared with the human rankings.Ties received equal ranks.Source: §7 · p. 9
  • For abstracts, technical-term coverage correlated with human judgments at 0.2865, SemSim at 0.2662, QAFactEval correlated highly, and FactGraph performed subpar.The authors suggest FactGraph may struggle to extract AMR information from patent texts’ complex structures and entity relationships.Source: §7 · p. 9
  • For claim generation, a heuristic rule-based checker had higher correlation with human judgments than the other tested methods.The authors also multiplied other metric scores by the checker score, reasoning that claims with many basic rule errors likely deviate significantly in quality.Source: §7 · p. 9
  • The IPC-fine-tuned model performed worse than the original model for claim generation, which the authors suggest may favor broader category exploration.They say further analysis is needed to explore the implications.Source: §7 · p. 9

8 Conclusions

The study evaluates language models generating patent abstracts and claims, identifying strengths in some models and common errors and their frequencies. The authors present these findings as groundwork for improving models and considering AI in patent drafting.

8 Conclusions

  • The study examines patent abstracts and claims generated by diverse language models.The authors describe the work as an advancement in generating and evaluating these patent texts.Source: §8 · p. 9
  • The investigation identifies strengths of certain LLMs in generating quality patent texts.Source: §8 · p. 9
  • The study identifies common errors in generated patent texts and their frequencies.The authors say these insights lay groundwork for model enhancement and incorporating AI into patent-drafting practices.Source: §8 · p. 9

9 Ethical Considerations and Limitations

The study is limited to claims-to-abstract and next-claim generation, leaving patent descriptions outside its evaluated tasks. Its findings also have language, dataset, and possible pretraining-overlap boundaries; the appendix describes the models and prompting used.

9 Ethical Considerations and Limitations

  • The study covers claims-to-abstract and next-claim generation, while lengthy, mostly unstructured patent descriptions pose greater extraction challenges.Source: §9 · p. 10
  • Segmenting descriptions into smaller sections is proposed, but its efficacy and transfer from standardized sections to less regimented text remain unproven.Source: §9 · p. 10
  • The findings are limited to English-language patent applications in the USPTO database; replication in other languages is uncertain given smaller corpora and reduced model effectiveness outside English.Source: §9 · p. 10
  • Possible exposure of recent LLMs to USPTO patents during pretraining may bias model comparisons toward recall; the authors suggest examining training data to assess potential pollution.Source: §9 · p. 10

A Model Details

  • The evaluated models included PatentTransformer (approximately 390 million patents), Patent Generative Transformer (11.6 million), PatentGPT-J (147B US patent data), HUPD T5 models, Llama 2, Falcon, and GPT-3.5.Source: A Model Details · p. 12
  • The LLMs used uniform prompts; next-claim prompts specified either a dependent claim or an independent claim.Source: A Model Details · pp. 12–13

B.1 Data Statistics

Table 4 summarizes 400 sampled patents by their main IPC section, reporting claim and abstract statistics. The authors plan to extend annotation to examples across more domains.

B.1 Data Statistics

  • Table 4 reports statistics for 400 patents selected from the refined HUPD dataset, grouped by each patent’s main IPC section.The reported measures include average claim count and average word counts for claims and abstracts.Source: §B.1 · p. 13, Table 4 · p. 13
  • Future annotation will cover a broader range of model-generated examples across domains.The authors say this expansion aims to deepen understanding and provide a more comprehensive evaluation of model performance in diverse patent contexts.Source: §B.1 · p. 13

C Metrics

The metrics section evaluates patent generation using semantic similarity and technical-term coverage for abstracts, plus drafting-rule checks and similarity for generated claims. It also describes a patent-trained encoder and its IPC-classification evaluation.

C Metrics

  • Abstract evaluation combines semantic similarity to input claims with coverage of key technical terms, while claim generation uses drafting-rule checks and semantic similarity.The same semantic-similarity methodology is applied to claims-to-abstract and claim-generation evaluation.Source: C Metrics · p. 13

C.1 Semantic Similarity Between Input Claims and Generated Abstracts

  • A patent-trained BERT encoder is fine-tuned on IPC classification to measure similarity between input claims and generated abstracts.The approach is inspired by Christofidellis et al. and uses sentence-transformer semantic similarity; the specialized encoder is fine-tuned for patent IPC classification.Source: §C.1 · p. 13

C.1.1 Main IPC Classification

  • The main-IPC classifier is trained on 1,338,054 HUPD patents from 2016–2017 and tested on 63,862 patents filed in 2018.Abstract-only and claims-only test sets show similar performance to the combined set; performance is measured with weighted precision, recall, and F1-score.Source: §C.1.1 · pp. 13–14, Table 5 · p. 14
  • Figure 4 shows the number of comparison-model pairs selected for each task during human annotation.The annotation included model-versus-model pairs and model-versus-original-abstract-or-next-claim pairs.Source: Figure 4 · p. 14

C.1.2 Method

The methods define semantic relevance and technical-term coverage for generated abstracts, then assess generated claims with drafting-rule checks and semantic similarity. Additional studied metrics include n-gram coverage and established factuality and coherence measures.

C.1.2 Method

  • The relevance score compares embeddings of input claims and generated abstracts using cosine similarity.A fine-tuned BERT-for-patents encoder and mean pooling produce the embeddings.Source: §C.1.2 · p. 14

C.2 Technical Features Coverage in Generated Abstracts

  • Technical-feature coverage measures how many unique terms from the input claims appear in the generated abstract.PyATE’s combo_basic algorithm identifies key terms in claims and abstracts.Source: §C.2 · p. 14

C.3 N-grams Coverage in Generated Abstracts

  • The analysis also measures coverage of generated-abstract n-grams for n = 1 to 4.The n-grams are extracted using NLTK.Source: §C.3 · p. 14

C.4 Verification of Basic Rules for Generated Claims

  • The claim checker gives zero for a non-distinctive repetition of an input claim and scores other checks for repetition, punctuation, numbering, and dependency.It adds one for each satisfied check and normalizes by the four checks, excluding distinctiveness.Source: §C.4 · p. 15

C.5 Semantic Similarity Between Input Claims and Generated Claim

  • Claim-generation similarity compares embeddings of the input claims with embeddings of those claims followed by the generated next claim.A second variant normalizes this similarity score by the rule-based checker score, lowering scores for claims that fail drafting standards or required dependency.Source: §C.5 · p. 15

C.6 Other Studied Metrics

  • The paper also studies FactGraph, QAFactEval, and EntityGrid for factuality or coherence evaluation.FactGraph and QAFactEval are described as factual-consistency metrics; EntityGrid assesses coherence from entity-distribution regularities.Source: §C.6 · p. 15

D Examples of Typology of Errors

This section illustrates the error typology with examples from claims-to-abstract and next-claim generation, including a generated claim that repeats earlier content without adding scope or detail. It also describes the annotation interface and process used to compare outputs and identify errors.

D Examples of Typology of Errors

  • For claims-to-abstract generation, ineffective summarization means failing to provide a concise, comprehensive overview, including by reproducing claim text.The section pairs this error label with a generated abstract example.Source: D Examples of Typology of Errors · p. 18
  • Next-claim examples identify structural errors involving transitional phrases, disconnected claim-body elements, and non-compliant or unclear dependencies.The examples include a dependent claim with an improper transitional phrase and one with disconnected elements in its body.Source: D Examples of Typology of Errors · p. 19
  • The typology also flags dependent claims that insufficiently narrow scope and independent claims that repeat or closely resemble earlier claim scope.Examples show a dependent claim repeating a limitation and a generated independent claim that restates features of earlier claims.Source: D Examples of Typology of Errors · pp. 20–21
  • Clarity errors include vague wording, missing antecedent basis, and inconsistent terminology for the same element.The section illustrates these categories with generated dependent claims.Source: D Examples of Typology of Errors · p. 21
  • Other examples cover wishful claims, verbose redundancy, complex claim structure, and matter unrelated to disclosed embodiments.These categories respectively concern speculative objectives, wordiness without substance, language that could be simplified, and potentially overbroad unrelated matter.Source: D Examples of Typology of Errors · pp. 21–22
  • Effectiveness errors include claims that conflict with earlier claims or repeat claims without distinction.The section labels these errors contradictory claims and non-distinctive claim repetition.Source: D Examples of Typology of Errors · p. 23
  • A generated claim exemplifies lack of effectiveness by repeating an earlier claim without adding scope or detail.The example repeats the imaging tray’s selectively lockable closed position from claim 2.Source: D Examples of Typology of Errors · p. 23

E Annotation Interface

  • Annotators selected a preferred output for each input-claim set and identified errors, with novel words highlighted, patent context provided, and comments enabled.The Label Studio interface is shown for both tasks in Figures 5 and 6; preprocessed annotation data and platform settings were to be made available.Source: E Annotation Interface · p. 23, Figure 5 · p. 24, Figure 6 · p. 24

Related papers

Loading 2406.06589v2…