Connecting the Dots: Inferring Patent Phrase Similarity with Retrieved Phrase Graphs

1 Introduction

• Patent phrase similarity can inform categorization, retrieval, and litigation analysis, and may help the USPTO streamline examination and identify prior art.The introduction also identifies assessing the distinctiveness of newly filed applications as a possible use.Source: §1 · p. 1
• Patent phrases use technical, context-dependent language that general semantic-similarity methods represent poorly, while expert annotations are costly and require domain knowledge.The resulting challenge is inferring phrase similarity without training labels.Source: §1 · p. 1
• The method retrieves a subgraph from the patent universe to capture phrase representations, following a strategy inspired by patent experts.The subgraph supplies broader patent context for a focal phrase.Source: §1 · pp. 1–2
• RA-Sim combines a phrase's textual contextual embedding with a graph embedding derived from its focal patent, citation-related patents, and associated phrases.A graph attention network processes the phrase graph; a topology-based self-supervised objective trains both textual embeddings and graph parameters end to end.Source: §1 · p. 2
• RA-Sim substantially improves self-supervised patent phrase similarity inference and consistently outperforms state-of-the-art methods in supervised evaluation.The authors report these outcomes on a large patent phrase similarity dataset.Source: §1 · p. 2

2 Related Work

• General-text similarity methods and Patent-BERT perform poorly on patent phrase similarity, while technical language, phrase brevity, and scarce expert labels motivate domain-specific modeling.The passage describes word- and sentence-level embedding approaches, including Word2Vec, FastText, BERT, RoBERTa (Liu et al., 2019), SBERT, and SimCSE.Source: §2 · p. 2
• Retrieval has been used across NLP generation, question answering, dialogue, and other traditional tasks, but the authors report no prior retrieval-module use for patent-domain tasks.Examples include question answering (Kumar et al., 2016; de Masson D’Autume et al., 2019; Chen et al., 2023), dialogue, and other NLP tasks.Source: §2 · p. 2
• Graph neural networks have been applied to patent analysis and technical text, but graph use in patent phrase similarity remains underexplored.Patent-analysis examples include Tang et al. (2020), Fang et al. (2021), Siddharth et al. (2022), and Zuo et al. (2022); technical-text work includes Shbita et al. (2023).Source: §2 · pp. 2–3
• The paper presents the first self-supervised patent phrase similarity framework that retrieves a domain graph to amplify phrases' global contextual information.Source: §2 · p. 3

3.1 Constructing Patent-Phrase Universe

• RAKE supplies phrase nodes, BM25 retrieves top-k patents per phrase, and phrase–patent links plus patent citations form the patent-phrase universe.Source: §3.1 · p. 3

3.2 Phrase Representation Augmentation with Phrase Ego Graph

• Recursive sampling builds each phrase’s ego graph from containing and referenced patents; a GNN readout augments its text embedding, and phrase similarity uses cosine similarity.Source: §3.2 · p. 4

3.3 Phrase Ego Graph Representation

• The text encoder initializes phrase and patent nodes, then relation-specific GAT transformations aggregate retrieval and citation neighbors into graph representations.Source: §3.3 · pp. 4–5

3.4 Learning Objective

• With no labeled similarity scores, RA-Sim jointly trains text and graph encoders end-to-end using retrieval-based phrase–patent and citation-based patent–patent contrastive objectives.Source: §3.4 · p. 5

4.1 Data

• Self-supervised evaluation uses only the 9,232-pair test split; the graph corpus includes 7,619,250 utility patents and 102,674,056 citation records.RAKE extracts three key phrases per patent abstract, with filtering yielding a phrase set of 26,555.Source: §4.1 · pp. 5–6

4.2 Metrics / 4.3 Training Setup

• Performance is measured by Pearson and Spearman correlations against expert-labeled similarities, averaged over three runs; training uses Sentence-BERT all-mpnet-base-v25 on four RTX 3090 GPUs.Source: §4.2 · p. 6, §4.3 · p. 6

4.4 Baselines

• Baselines include pretrained language models, Patent-BERT, contrastively fine-tuned models, word embeddings, BM25-based RetrieveAvg, and a graph-only control.Source: §4.4 · p. 6

4.5 Main Results

• RA-Sim exceeds second-best SBERT by 5.8% in Pearson R and 9% in Spearman R.It also outperforms Contriever and word-embedding methods by a large margin; retrieval-based methods improve inference, while graph topology alone contributes and patent text remains important.Source: Table 1 · p. 6, §4.5 · p. 7

4.6 Ablations

• Using 1% of phrases lowers Pearson R from 0.633 to 0.616, while 10% causes only minor degradation; changing the phrase source or dropping lemmatization leaves performance unchanged.Source: §4.6 · p. 7
• BM25 retrieval performs best at k=5; three graph-expansion iterations perform worse than two, and 3-hop neighbors may be too relevant to use as hard negatives.Source: §4.6 · pp. 7–8

4.7 Supervised Learning with Training Set

• In supervised experiments, RA-Sim also outperforms the other benchmarks, though the paper’s main focus remains the more challenging label-absent setting.Source: Table 3 · p. 8, §4.7 · p. 8

4.6 Ablations

• The tested attention-based graph layer performs best, and the losses complement one another: retrieval loss is needed for aggregation, while citation loss alone lowers Pearson R from 0.633 to 0.551.Source: §4.6 · p. 8

5.1 Alignment and Uniformity

• At threshold 0.756, lower alignment loss indicates more similar positive pairs, while lower uniformity loss indicates better information retention; similar patterns appear at other thresholds.Source: §5.1 · p. 8
• RA-Sim preserves uniformity while improving alignment versus SBERT; SimCSE’s improved uniformity accompanies poor alignment, while RetrieveAvg and graph retrieval improve alignment, with graph retrieval outperforming patent-only retrieval.Figure 2 colors methods by similarity-inference performance and omits Word2Vec and Graph-Only because of poor performance.Source: Figure 2 · p. 8, §5.1 · p. 8

5.2 Qualitative Results

• In Table 4’s top-five retrieval examples, RA-Sim returns directly relevant “planting tree” results and more precise eye-anatomy phrases than SBERT, whose outputs include topology or hand-quadrant terms.For “eye quadrants,” both methods retrieve “eye region anatomy.”Source: Table 4 · p. 8, §5.2 · pp. 8–9

• Patent phrase similarity differs from classic semantic textual similarity because patent language is highly technical.Source: §6 · p. 9
• RA-Sim augments contextualized phrase information with a graph of the focal patent, citation-linked related patents, and associated phrases, and trains with self-supervised objectives centered on that graph.Source: §6 · p. 9
• Experiments show superior performance for RA-Sim, highlighting retrieval augmentation’s benefits for contextualized embeddings in domain problems.Source: §6 · p. 9

• The method has only been tested on patent data, so its potential in other domains, such as scientific articles, remains unevaluated.Source: §7 · p. 9
• Generating a phrase set for the vast patent universe is costly; the authors use efficient RAKE and plan to explore more computationally expensive phrase-generation methods.Source: §7 · p. 9
• Learning from the retrieved graph module incurs additional computational costs compared with non-graph-based methods.Source: §7 · p. 9

• Table 5 summarizes statistics of token number.Source: Table 5 · p. 12, A Token Statistics · p. 12

B Full Training Details

• Among graph depths of 1–3 layers, two performed best; checkpoints were selected by validation performance, then evaluated on the testing phrase set, with reported results averaged over three runs.BM25 retrieval sizes were k from {3, 5, 7, 50}; neighbor sampling used 1, 2, or 3 iterations and 1, 3, or 5 neighbors per iteration. Margin parameters δr and δc were chosen from {0.01, 0.02, 0.05, 0.1, 0.2, 0.5}.Source: B Full Training Details · p. 12

C Supervised Learning Setup

• RA-Sim was trained with supervised and self-supervised losses, while benchmark models used supervised loss alone.RA-Sim first trained with supervised loss alone for two epochs; the overall budget was five epochs.Source: C Supervised Learning Setup · p. 12

D Baseline Models

• The evaluation includes pretrained language-model baselines and SimCSE-finetuned BERT, RoBERTa, and SBERT variants.The pretrained comparisons include BERT, SBERT, Patent-BERT, Contriever, GTE, and instructor-xl.Source: D Baseline Models · pp. 12–13
• The baseline list also includes off-the-shelf GloVe, Word2Vec, and FastText word embeddings.Source: D Baseline Models · p. 13
• RetrieveAvg combines a phrase embedding with the BM25-retrieved patent embedding through validation-tuned weighted averaging.SBERT provides the embeddings for phrases and retrieved patents.Source: D Baseline Models · p. 13
• Graph-Only tests graph structure by replacing RA-Sim's initial phrase and patent text embeddings with random embeddings.Source: D Baseline Models · p. 13

E.1 Phrase Examples

• Table 6 presents patent abstract and phrase examples generated using RAKE.Source: §E.1 · p. 13

E.2 BM25 Retrieval Example

• Table 7 shows sample patents retrieved from the USPTO database for “Wireless Communications Network.”The table presents the three most relevant patents.Source: §E.2 · p. 13, Table 7 · p. 14