PaperlayerView exact revision on arXiv

Source-linked AI summary

PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT

Hamid Bekamiri, Daniel S. Hain, Roman Jurowetzki

arXiv:2103.11933v3cs.LGecon.EM

TL;DR

Patent similarity methods based on transformers face computational costs, patent-specific language, and multi-label classification challenges. The paper combines augmented SBERT claim embeddings with KNN classification, reporting 66.48% subclass-level F1 and 58% accuracy while describing the results as better than previous models.

  • Problem

    Transformer-based patent analysis faces computational expense, domain-specific jargon and legal language, and the multi-label, multi-class nature of patent classification.

    Source: §1 · pp. 2–3

  • Method

    The paper uses augmented SBERT to create patent-claim embeddings and KNN to predict CPC labels from the most similar claims.

    Source: §1 · p. 3, §4 · p. 10

  • Results

    At subclass level, F1 score was 66.48% and accuracy was 58%; the authors report better accuracy than previous models.

    Source: CPC Prediction · p. 15

  • Takeaways & Limitations

    The authors present KNN classification as an interpretable application of patent-to-patent similarity, with predictions that users can inspect and explain.

    Source: §1 · p. 3, §6 · p. 17

    The authors identify sample size as a significant limitation and plan to test the model on a larger sample.

    Source: §7 · p. 17

Abstract

from arXiv · show

This study provides an efficient approach for using text data to calculate patent-to-patent (p2p) technological similarity, and presents a hybrid framework for leveraging the resulting p2p similarity for applications such as semantic search and automated patent classification. We create embeddings using Sentence-BERT (SBERT) based on patent claims. We leverage SBERTs efficiency in creating embedding distance measures to map p2p similarity in large sets of patent data. We deploy our framework for classification with a simple Nearest Neighbors (KNN) model that predicts Cooperative Patent Classification (CPC) of a patent based on the class assignment of the K patents with the highest p2p similarity. We thereby validate that the p2p similarity captures their technological features in terms of CPC overlap, and at the same demonstrate the usefulness of this approach for automatic patent classification based on text data. Furthermore, the presented classification framework is simple and the results easy to interpret and evaluate by end-users. In the out-of-sample model validation, we are able to perform a multi-label prediction of all assigned CPC classes on the subclass (663) level on 1,492,294 patents with an accuracy of 54% and F1 score > 66%, which suggests that our model outperforms the current state-of-the-art in text-based multi-label and multi-class patent classification. We furthermore discuss the applicability of the presented framework for semantic IP search, patent landscaping, and technology intelligence. We finally point towards a future research agenda for leveraging multi-source patent embeddings, their appropriateness across applications, as well as to improve and validate patent embeddings by creating domain-expert curated Semantic Textual Similarity (STS) benchmark datasets.

1. Introduction

Patent similarity measures are increasingly derived from text embeddings, but transformer-based patent analysis faces computational, domain-language, and multi-label classification challenges. The paper proposes an efficient SBERT-based similarity framework, augmented fine-tuning, and an interpretable KNN classifier.

1. Introduction

  • The paper positions its framework for semantic search, patent landscaping, technology mapping, and patent quality evaluation.The introduction describes these as potential applications of technological-distance measures and the proposed framework.Source: §1 · pp. 2–3
  • Text embeddings represent text as numeric vectors while preserving meaning, supporting semantic patent-to-patent similarity beyond metadata-based measures.The paper situates text-based similarity alongside earlier measures based on keywords, classifications, or citations.Source: §1 · p. 2
  • Transformer models can be costly for large-scale patent similarity, while generic language models face patent jargon and legal language and patent classification is multi-label and multi-class.The paper also notes that expert-labeled in-domain examples can be sparse or unavailable and costly to create.Source: §1 · pp. 2–3
  • The paper combines efficient SBERT embeddings, augmented fine-tuning for patent language, and a KNN workflow for patent similarity and classification.The authors describe the workflow as usable even without expert labels and its predictions as easy to inspect and explain.Source: §1 · p. 3

2. Related work

Related work spans keyword, semantic-structure, ontology, and machine-learning approaches to patent similarity and classification. The paper notes that comparisons across classification tasks and datasets are difficult to make fairly.

2. Related work

  • Patent-similarity methods are grouped into keyword-based, Subject-Action-Object, ontology-based, and machine-learning approaches.Keyword methods use frequency or co-occurrence, SAO extracts subject-object-action triples, and ontology methods annotate text against domain concepts.Source: §2 · p. 4
  • Machine-learning methods map complex text relationships and have drawn growing interest for patent similarity, while patent-classification literature is difficult to compare fairly.The passage attributes the comparison difficulty to differences in datasets, evaluation structures, classification systems, and input information.Source: §2 · p. 4
  • Earlier examples include a three-phase SVM, K-means, and KNN hybrid with 53.25% sub-group accuracy [Chen and Chang, 2012] and an SVM/ULMFiT approach with 78.4% F1 for eight IPC section labels.These results concern different methods and evaluation scopes.Source: §2 · pp. 4–5
  • DeepPatent uses CNNs and title-and-abstract embeddings, reporting about 43% F1 and 73.88% classification precision on its described USPTO-2M evaluation.The study describes the evaluation as 2,000,147 records in 637 IPC subclass categories.Source: §2 · p. 5
  • PatentBERT fine-tunes BERT on patent claims and reports 44.75% F1 Top 5 for IPC+Title+Abstract, but does not provide patent similarity measures [Lee & Hsiang, 2020; Hain et al., 2021].The passage also says claims alone were sufficient for classification and produced approximately equal F1 to title-and-abstract inputs.Source: §2 · p. 5

3. Data

The study uses US patent claims and CPC labels from PatentsView-related data, encoding the first claim for similarity and classification. Its dataset spans 663 subclass labels, with substantial class imbalance.

Dataset Description

  • The study uses PatentsView patent data and patents from 2013–2017 with at least one claim, totaling 1,492,294 records, with 8% reserved for testing.Duplicate patent IDs and claim text were removed.Source: Dataset Description · pp. 5–6

Textual Data: Patent Claims

  • Patent claims are used as the textual input, with only the first claim selected as the backbone of the claim hierarchy.The mean number of claims per patent was 17; the study set maximum input length to 510 tokens and used padding or truncation.Source: Textual Data: Patent Claims · pp. 6–7

Patent Classification: CPC Classes

  • CPC is hierarchical, with Section, Class, Subclass, and Group levels; the dataset includes 663 subclass labels.The cited passage gives 667 as the number of CPC subclass labels overall.Source: Patent Classification: CPC Classes · p. 8, Figure 3 · p. 9
  • After extracting classes at the subclass level, the model was evaluated on 10,000 patents, and F1 score increased by 67.23%.The dataset had 663 labels, 159 with fewer than 350 samples; the passage describes the distribution as imbalanced and KNN as weak for handling it.Source: Patent Classification: CPC Classes · pp. 8–9

4. Method and experimental setup

The framework combines claim embeddings from SBERT with K-nearest-neighbor classification, using Augmented SBERT to adapt the embeddings to patent language. The training process uses cross-encoder-labeled claim pairs because annotated in-domain pairs were unavailable.

4. Method and experimental setup

  • Pretrained language models are fine-tuned for downstream tasks using in-domain data, and SBERT requires target-task fine-tuning to achieve competitive semantic-similarity performance.The paper frames this as a practical alternative to BERT for large-scale similarity calculations.Source: §4 · p. 9
  • SBERT provides a practical way to calculate patent-to-patent similarity, while the hybrid model uses the most similar claims to predict patent subclasses with KNN.The study contrasts SBERT’s practical efficiency with BERT’s higher performance but slower operation.Source: §4 · pp. 9–10, Figure 4 · p. 10
  • With no annotated claim pairs, the study labels sampled in-domain pairs using a cross-encoder and trains the bi-encoder on those labels plus STS data.Sampling avoids using all n × (n − 1)/2 possible pairs, which the authors say would add computational overhead without improving performance.Source: §4 · pp. 11–12, Figure 5 · p. 11, Figure 6 · p. 11
  • The training pipeline used 1,143 claim sentences, selected 3,432 claim pairs from 652,653 possible combinations, then fine-tuned SBERT on labeled patent pairs and STS data.RoBERTa first labels the selected pairs as a cross-encoder.Source: §4 · p. 12

5. Results

The results cover claim-based semantic search and CPC prediction. The reported search example retrieved similar patents from a corpus of 1,492,294, while classification performance varied with the number of neighbors.

P2P similarity and semantic search

CPC Prediction

  • PatentBERT and DeepPatent had subclass-level F1 Top 5 scores below 45% and 43%, respectively.These are the prior-model results reported for comparison in Table 4.Source: CPC Prediction · p. 15, Table 4 · p. 15
  • At the sub-class level, the model achieved an F1 score of 66.48% and accuracy of 58%; the authors conclude its accuracy was significantly better than previous models.Source: CPC Prediction · p. 15
  • F1 and accuracy were best at K=8, with precision and recall of 74% and 60%; at K=20, precision rose to 86% while F1 and accuracy fell to 58% and 46%.Figure 7 presents the performance metrics across K values.Source: CPC Prediction · p. 15, Figure 7 · p. 16
  • The classifier makes subclass predictions interpretable by showing users the top semantically similar patents for evaluating each result.The paper says users can compare results with existing biological knowledge.Source: CPC Prediction · p. 16

6. Conclusion

The paper presents an efficient, scalable augmented-SBERT approach for embedding patent claims and calculating p2p similarity. It uses those similarities for CPC prediction and argues that the approach preserves technological properties while making predictions understandable to end-users.

6. Conclusion

  • Efficient SBERT embeddings enable p2p similarity calculation across large patent datasets.The paper fine-tunes general pre-trained SBERT on patent claims using an augmented approach.Source: §6 · pp. 16–17
  • The framework uses augmented SBERT fine-tuned to patent claims to create transformer-based patent embeddings.Source: §6 · pp. 16–17
  • A simple KNN model predicts CPC classes from the most similar patents, supporting automated classification and interpretable decisions.The authors say the predictions also validate that the embeddings preserve patents’ technological properties.Source: §6 · p. 17

7. Limitations & Future Research

The authors identify sample size and reliance on a single claim as limitations, and outline plans to test larger inputs and multiple patent information sources. They also propose exploring an expert-panel patent STS dataset.

7. Limitations & Future Research

  • Sample size is identified as a significant limitation; testing a larger sample with Annoy is planned.The authors describe Annoy as finding approximate rather than exact similarities in a space.Source: §7 · p. 17
  • The current approach uses one claim; future work will test multiple information sources and longer inputs without truncation.Source: §7 · p. 17
  • The authors plan to explore an expert-panel patent STS dataset for fine-tuning and validating transformer models.They state that such a dataset can increase model accuracy through fine-tuning.Source: §7 · p. 17
Loading 2103.11933v3…