PaperlayerView exact revision on arXiv

Source-linked AI summary

PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

Jieh-Sheng Lee, Jieh Hsiang

arXiv:1906.02124v2cs.CLcs.LGstat.ML

TL;DR

Patent classification is challenging across many labels, and the paper evaluates fine-tuned BERT on a large CPC patent dataset using claims alone. PatentBERT outperforms DeepPatent, and the authors conclude that claims alone are sufficient for classification within this study’s scope.

  • Problem

    Patent classification can involve more than 630 subclass labels, while differences in dataset size have made comparisons across prior patent studies difficult.

    Source: §1 · p. 1

  • Method

    The authors fine-tune a pre-trained BERT model for patent classification using patent claims, focusing on the first claim of each patent.

    Source: §1 · p. 2, §6 · p. 4

  • Results

    PatentBERT outperforms DeepPatent, with precision of 84.26% reported for PatentBERT.

    Source: §6 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    The authors conclude that patent claims alone are sufficient for classification and that pre-training followed by fine-tuning is promising for patent research.

    Source: §6 · pp. 4–5

    The authors assume methods benchmarked with DeepPatent are unlikely to perform better on datasets larger than USPTO-2M.

    Source: §2 · p. 2

Abstract

from arXiv · show

In this work we focus on fine-tuning a pre-trained BERT model and applying it to patent classification. When applied to large datasets of over two millions patents, our approach outperforms the state of the art by an approach using CNN with word embeddings. In addition, we focus on patent claims without other parts in patent documents. Our contributions include: (1) a new state-of-the-art method based on pre-trained BERT model and fine-tuning for patent classification, (2) a large dataset USPTO-3M at the CPC subclass level with SQL statements that can be used by future researchers, (3) showing that patent claims alone are sufficient for classification task, in contrast to conventional wisdom.

1 Introduction

Patent classification is a multi-label task with more than 630 possible subclass labels, and prior datasets and metrics complicate comparisons. The paper evaluates fine-tuned BERT on a large CPC dataset using patent claims, focusing on the first claim of each patent.

  • Fine-tuning a pre-trained BERT model is evaluated for patent classification, where the subclass task can involve more than 630 labels.Source: §1 · p. 1
  • The authors prepared a CPC dataset with more than three million US patents to provide a large-dataset baseline and improve comparability across patent studies.They note that past dataset sizes varied widely, making comparisons difficult, and some datasets were outdated.Source: §1 · p. 1
  • Because P@1 can conceal low recall, the authors use F1@1 as their preferred patent-classification performance measure.They benchmark against the best F1 values reported in other works, whether F1@1 or F1@5.Source: §1 · pp. 1–2
  • The dataset uses patent claims alone; the authors focus on the first claim per patent and leave other independent and dependent claims for future research.They motivate this choice by noting that claims define the invention’s legal scope and that claims are commonly drafted first.Source: §1 · p. 2

2 Related Work

Prior patent-classification studies used varied datasets, classification schemes, and metrics, limiting direct comparison. The authors position DeepPatent as their baseline while noting differences in scope and benchmark feasibility across related work.

  • DeepPatent, a CNN with word embeddings, reported 73.88% precision @ Top 1 on USPTO-2M, but disclosed no F1@1; its best F1 was about 43% @ Top 5.The authors use DeepPatent as their baseline and assume the other methods it benchmarked are unlikely to perform better on a dataset larger than USPTO-2M.Source: §2 · p. 2
  • Hepburn used ULMFiT on 75,250 Australian patents at IPC section level, reporting F1 of 78.4%, while this paper reports 80.98% F1 at CPC section level on over three million patents.The authors identify the pre-trained model and dataset size as differences between the studies.Source: §2 · p. 2
  • Tran and Kavuluru reported 69.89% micro-F1 on 436,993 U.S. patents at CPC subclass level using logistic regression with additional data or methods.The authors skip benchmarking against this result because their target dataset is larger and they avoid ad-hoc feature engineering, which they say is difficult to scale.Source: §2 · p. 2
  • The authors skip direct CLEF-IP comparisons because this work targets CPC and a larger dataset, while noting Yadrintsev et al. reported 71.02% micro-F1 at IPC subclass level.Source: §2 · pp. 2–3
  • Lim and Kwon reported 87.2% precision for Korean patent classification using multiple patent fields, but omitted recall and F1, preventing a fair comparison.Source: §2 · p. 3
  • Hu et al.’s hierarchical feature model outperformed standalone CNN, LSTM, or BiLSTM models on mechanical patents, with best F1 of 63.97% @ Top 1.The cited CLEF-IP subset had 96 labels and 72,532 training patents.Source: §2 · p. 3

3 Data

The paper builds USPTO-3M from Google Patents Public Datasets using SQL statements, which the authors present as a reusable and flexible alternative to distributing preprocessed datasets. They combine it with USPTO-2M when needed for benchmarking.

  • They argue SQL statements lower data-preparation barriers and are easier to reuse or revise than conventional datasets with embedded preprocessing or postprocessing.The training-dataset SQL statement appears in Appendix A.Source: §3 · p. 3
  • The authors use Google Patents Public Datasets on BigQuery to create USPTO-3M, a dataset of 3,050,615 patents.Source: §3 · p. 3
  • For DeepPatent benchmarking, the authors use USPTO-2M when feasible and combine it with USPTO-3M when needed, since USPTO-2M lacks claims.The combined data let them examine how claims affect performance.Source: §3 · p. 3

4 Method & Experimental Setup

The method fine-tunes BERT-Base for multi-label patent classification with minimal changes to the released BERT example. The authors retain default hyperparameters and leave level-specific tuning for future work.

  • The experiments use the released uncased BERT-Base model, with 12 layers, 768 hidden units, 12 heads, and 110M parameters.BERT-Large is left for future work because the authors report BERT-Base is already sufficient to outperform DeepPatent.Source: §4 · p. 3
  • For multi-label classification, the implementation replaces softmax with sigmoid cross entropy with logits while otherwise keeping code changes minimal.Source: §4 · p. 3
  • Hyperparameters remain at their defaults, including max_seq_length 128; testing different values for different CPC levels is left for future work.The authors observed that shorter sequences might suffice with fewer labels, such as nine labels at CPC section level.Source: §4 · p. 3

5 Results: PatentBERT vs DeepPatent

PatentBERT outperforms DeepPatent on reported patent-classification benchmarks. Its F1 remains stable as test size triples and with USPTO-3M, although metrics drop slightly when test years move farther from training.

5 Results: PatentBERT vs DeepPatent

  • 46.85% F1 @ Top 5 exceeds DeepPatent’s USPTO-2M result, reported below 45% @ Top 5.Table 1 presents the performance comparison.Source: §5 · p. 4, Table 1 · p. 4
  • 66.83% F1 is reported for PatentBERT at the CPC subclass level, and its F1 reaches 64.91% @ Top 1.Source: §5 · p. 4
  • 84.26% precision from PatentBERT outperforms DeepPatent’s precision in both reported comparison rows.Source: §5 · pp. 3–4
  • F1 remains stable when the test dataset triples in size.Source: §5 · p. 4
  • F1 remains stable with USPTO-3M, while F1, precision, and recall drop slightly as test years move farther from training.Source: §5 · p. 4

6 Conclusion

The paper presents fine-tuned pre-trained BERT as a new state-of-the-art approach for patent classification, with results showing claims alone are sufficient. It describes this two-stage framework as promising for further exploration in patent research.

6 Conclusion

  • Fine-tuning a pre-trained BERT model is presented as a new state-of-the-art approach that outperforms DeepPatent.Source: §6 · p. 4
  • Patent claims alone are sufficient for the patent-classification task.Source: §6 · p. 4
  • The pre-training-and-fine-tuning framework is described as promising for future exploration, with patent classification offered as one example.Source: §6 · pp. 4–5

Appendix A

The appendix gives SQL for selecting first claims of US utility patents from 2013 and aggregating CPC codes at subclass level.

Appendix A

  • The SQL selects the first claims of US utility patents in 2013 and aggregates CPC codes at subclass level.Source: Appendix A · p. 5
  • The query draws from patent, current CPC, and claim tables in Google Patents Public Datasets on BigQuery.Source: Appendix A · p. 5
  • The query groups records using the listed fields t1.id, t1.date, and t3.text, which are the grouping keys specified in the cited passage.Source: Appendix A · p. 6
Loading 1906.02124v2…