PaperlayerView exact revision on arXiv

Source-linked AI summary

TechNet: Technology Semantic Network Based on Patent Data

Serhad Sarica, Jianxi Luo, Kristin L. Wood

arXiv:1906.00411v4cs.IRcs.CL

TL;DR

Engineering knowledge retrieval lacks a comprehensive semantic network of technology concepts and relations, and existing NLP benchmarks are not suited to evaluating engineering retrieval and inference. The paper constructs TechNet from patent texts using NLP term extraction and word embeddings, and reports that its word2vec networks outperform existing public semantic networks on engineering-relevant tasks. The authors also identify scope and construction choices that leave room for further development.

  • Problem

    Existing semantic networks lack sufficiently representative technology terminology and engineering-design relations, while available NLP benchmarks are unsuitable for evaluating engineering knowledge retrieval and inference.

    Source: §1 · p. 3, §6 · p. 29

  • Method

    The paper combines patent data, NLP-based technical-term extraction, word-embedding models, and an engineering-context Technical Term Relevance benchmark to construct and evaluate TechNet.

    Source: §6 · pp. 28–29

  • Results

    TechNets, especially those based on word2vec, outperformed existing public semantic networks on engineering-relevant retrieval and inference tasks, while ConceptNet had lower technical-term coverage than TechNets.

    Source: §6 · pp. 29–30

  • Takeaways & Limitations

    TechNet offers technology-term and relevance retrieval through a public interface and APIs, and the authors identify potential for integration with ConceptNet.

    Source: §5 · p. 27, §6 · p. 30

    The study mined patent titles and abstracts, used two embedding algorithms and a small set of hyperparameters, and curated a small TTR dataset; term extraction also involved considerable manual effort.

    Source: §6 · p. 29

Abstract

from arXiv · show

The growing developments in general semantic networks, knowledge graphs and ontology databases have motivated us to build a large-scale comprehensive semantic network of technology-related data for engineering knowledge discovery, technology search and retrieval, and artificial intelligence for engineering design and innovation. Specially, we constructed a technology semantic network (TechNet) that covers the elemental concepts in all domains of technology and their semantic associations by mining the complete U.S. patent database from 1976. To derive the TechNet, natural language processing techniques were utilized to extract terms from massive patent texts and recent word embedding algorithms were employed to vectorize such terms and establish their semantic relationships. We report and evaluate the TechNet for retrieving terms and their pairwise relevance that is meaningful from a technology and engineering design perspective. The TechNet may serve as an infrastructure to support a wide range of applications, e.g., technical text summaries, search query predictions, relational knowledge discovery, and design ideation support, in the context of engineering and technology, and complement or enrich existing semantic databases. To enable such applications, the TechNet is made public via an online interface and APIs for public users to retrieve technology-related terms and their relevancies.

1. Introduction

TechNet is proposed as a technology-focused semantic network of engineering concepts and their relations, addressing the need for technology-oriented knowledge discovery and retrieval. The paper uses patent data to build this network, motivated by the breadth and technical descriptions of patents.

1. Introduction

  • TechNet represents technology-related concepts and semantic relations to support engineering knowledge discovery, information retrieval, design aids, and innovation management.The authors describe it as infrastructure for semantic-level integration and analysis of technology-related data.Source: §1 · p. 2
  • The complete USPTO patent database is presented as a broad source of engineering knowledge, with patent descriptions externally validated through examinations for usefulness and novelty.The authors argue that its organic growth and coverage make it a natural source for TechNet construction.Source: §1 · p. 3
  • Earlier engineering semantic-network work used mixed academic and design-blog data, while many patent text analyses were limited to small samples or document retrieval.Patent-class networks covered technology domains but relied on classification systems and document-level analysis.Source: §1 · pp. 2–3
  • The construction approach mines patent data, extracts generic engineering terms with NLP, and uses neural word embeddings to vectorize terms and establish their relationships.The authors expect the full patent corpus to support broad domain coverage and unsupervised, data-driven associations.Source: §1 · pp. 3–4

2. Related Work

Prior semantic resources and engineering information-retrieval methods provide useful foundations, but the paper identifies a lack of comprehensive, technology-oriented semantic relations. It reviews word embeddings as a way to represent term relationships from context and motivates applying them to patent data.

2.1 Semantic representation of information

  • Public semantic databases such as WordNet, ConceptNet, and BabelNet do not derive entity relations from an engineering-design viewpoint.The paper contrasts these resources with a need for technology-oriented terminology and relations.Source: §2.1 · p. 5, §2.2 · p. 10
  • Engineering information-retrieval research spans document search and concept retrieval, including ontologies, semantic networks, and methods that learn relations from text.The reviewed approaches range from predefined structures and expert input to unsupervised or semi-supervised extraction.Source: §2.1 · pp. 6–7

2.2 Word embeddings models

  • Word embeddings represent words and phrases as dense vectors in a lower-dimensional space, using relationships between words and their contexts.The review names word2vec and GloVe as algorithms that consider context during training.Source: §2.2 · pp. 7, 10
  • In continuous skip-gram, a target word is used to predict surrounding context words, with a softmax classifier estimating their probabilities.The described architecture uses a word-vector lookup projection and vocabulary-sized input and output layers.Source: §2.2 · p. 9
  • The paper identifies a gap in comprehensive technology knowledge graphs built from patent text and proposes extracting technical terms and learning their contextual relations with word embeddings.The resulting TechNet is intended to represent functions, components, structures, properties, operating principles, and other technical concepts.Source: §2.2 · p. 10

3. Construction of the TechNet

TechNet is constructed by extracting technical terms from patent text, embedding them as vectors, and linking concepts through semantic relevance. The pipeline uses USPTO utility-patent titles and abstracts, then processes and vectorizes the resulting corpora.

3. Construction of the TechNet

  • TechNet extracts technically meaningful words and phrases from patent texts, embeds them as vectors, and associates the vectors to form a technology-concept network.The vectors represent terms in a shared space, where cosine similarity indicates semantic relevance.Source: §3 · p. 10, §3.3 · p. 18

3.1 Source Data

  • The source was 5,771,030 USPTO utility patents granted from 1976 to October 2017; design patents were excluded and titles and abstracts retained.The authors selected titles and abstracts for computational efficiency and accuracy, describing them as concise invention descriptions.Source: §3.1 · pp. 11–12

3.2 Term Extraction

  • Four phrase extractors—Mikolov’s statistical method, noun-phrase rules, TextRank, and RAKE—were compared, with phrases capped at four words.Mikolov phrasing used two passes with threshold tuples (200, 100), (50, 25), and (5, 2.5).Source: §3.2 · p. 13
  • A human reader filtered patent-specific noisy phrases, while lemmatization, stop-word and digit removal, and singleton-term filtering further cleaned the corpora.The reader examined 1,000 randomly selected sentences and identified patent-specific stop words such as “disclosure,” “plurality,” and “thereof.”Source: §3.2 · p. 14
  • The six corpus vocabularies are patent-based and specialized in technology terms, differentiating them from WordNet and ConceptNet.Source: §3.2 · p. 15

3.3 Term Vectorization

  • Word2vec and GloVe were trained with vector sizes of 150, 300, or 600, context windows of 10 or 20, and a frequency-based down-sampling threshold.The threshold was set using corpus term frequencies; the selected frequency for “connect” was 0.0039.Source: §3.3 · pp. 15–17
  • Cosine similarity between term vectors supplies semantic relevance; the network stores vectors and retrieves pairwise relevance on demand because of its scale.The corpora contain millions of unique terms, implying more than 10^12 potential bidirectional links.Source: §3.3 · p. 18

4. Evaluation

The evaluation measures engineering-term coverage and pairwise semantic relevance, contextualizing both retrieval and inference in engineering and technology. It then characterizes the selected TechNet’s scale and structure.

4. Evaluation

  • The study evaluates term retrieval against an engineering dictionary and pairwise relevance against three general-knowledge benchmarks plus a custom technical benchmark.The custom TTR benchmark comprises 276 term pairs rated by 10 experienced engineers and engineering researchers, with inter-rater reliability of 0.88.Source: §4 · p. 18, §4.2 · p. 20

4.1 Evaluation of Term Coverage

  • Corpus #3 generally retrieves more engineering dictionary terms than the other evaluated vocabularies, and all TechNet vocabularies show superior engineering-term coverage.The Multidisciplinary Design Project Engineering Dictionary contains 2,704 terms across six categories.Source: §4.1 · pp. 18–19

4.2 Evaluation of Term-To-Term Semantic Relevance

  • Word2vec-trained TechNets generally outperform existing semantic networks, while GloVe-trained candidates do not; TechNet #9 performs best on the TTR benchmark.TechNets generally show stronger correspondence with TTR than with the three general-knowledge benchmarks.Source: §4.2 · p. 21

4.3 Structural Characteristics of the TechNet

  • TechNet #9 contains 4,038,924 technology-related terms and roughly 8.15x10^12 bidirectional pairwise relevance values.Across 125 patent classes, term distribution correlates with patent distribution at 0.976.Source: §4.3 · pp. 22–23, Table 6 · p. 22
  • More than 99.997% of term pairs have relevance values above zero, and filtering to the strongest 0.15% still leaves about 12x10^9 links.Those strong links average around 6,000 per term, motivating efficient storage and retrieval methods.Source: Figure 7 · p. 23, §4.3 · p. 24, Figure 8 · p. 24

5. Applications

The TechNet interface and APIs expose technology terms and their semantic relations for engineering knowledge discovery, retrieval, design, and innovation support. They provide term-relevance lookup, text-derived adjacency matrices, and user-guided concept-tree exploration.

5. Applications

  • REST APIs use on-demand storage lookups rather than keeping the graph database in server memory, with computation performed server-side.The authors report that the interface remains responsive even on mobile devices.Source: §5 · p. 25
  • The interface retrieves the 20 most related terms to “wireless charger,” which represent technical functions, components, configurations, or mechanisms absent from several general semantic networks.The passage contrasts these results with Google recommendations that are more related to consumer brands and products.Source: §5 · p. 26, Table 7 · p. 26
  • Users can retrieve a text’s technical terms together with their pairwise relevance as a TechNet subgraph, visualized as an adjacency matrix and available as CSV data.The example uses a Wikipedia passage about radio technology.Source: §5 · pp. 26–27, Figure 11 · p. 27
  • A tree-expansion search lets users choose breadth and depth from a root term to explore relevant surrounding concepts for engineering design ideation.The example expands “flying car” to depth 3 and breadth 3, with options to choose branches and layers manually.Source: §5 · p. 27, Figure 12 · p. 28

6. Discussion and Concluding Remarks

The discussion presents TechNet as a patent-trained engineering semantic network evaluated with a new engineering relevance benchmark, reporting stronger performance on engineering-related tasks while noting limits in its data and construction. It points to broader data and methods, regular updating, and possible integration with general semantic networks.

6. Discussion and Concluding Remarks

  • TechNet combines the complete USPTO patent database, word embeddings, and a new engineering-context benchmark, addressing the lack of suitable evaluations for engineering semantic-network retrieval and inference.The authors describe the approach as the first to train recently emerged word embedding models on the complete USPTO patent text database and curate an engineering-context TTR benchmark.Source: §6 · pp. 28–29
  • TechNets, especially word2vec-based ones, outperform public semantic networks on engineering-relevant knowledge retrieval and inference despite the study’s limited data, algorithms, hyperparameters, and TTR dataset.The study mined only patent titles and abstracts, used two word embedding algorithms and a small set of hyperparameter values, and curated a small TTR dataset.Source: §6 · p. 29
  • The current study mines patent titles and abstracts, uses two embedding algorithms and few hyperparameter values, and relies on a small TTR dataset; detecting noisy terms also requires considerable manual effort.The authors identify manual noisy-term detection as part of the current term-extraction procedure.Source: §6 · p. 29
  • Future work could mine patent descriptions and other databases, automate noisy-term detection, test more embedding algorithms and similarity metrics, and broaden TTR term pairs and expert evaluation.The authors also call for wider exploration of term-extraction techniques and training hyperparameters.Source: §6 · p. 29
  • TechNet fills a gap in large-scale technology semantic networks and could be integrated with general-knowledge networks for technology-related text analysis and artificial intelligence.The authors suggest integration as an alternative to standalone applications.Source: §6 · p. 29
  • On the TTR benchmark, ConceptNet scores best among general-knowledge networks and exceeds GloVe-trained TechNets, but has significantly lower technical-term coverage than TechNets.The authors suggest that TechNet’s technical-context performance and ConceptNet may be complementary.Source: §6 · p. 30
  • As technologies emerge and patent data grows, TechNet will need regular updates and scaling through further training on new patent data.The authors also point to advances in data science and NLP as opportunities to improve the corpus and embeddings.Source: §6 · p. 30
Loading 1906.00411v4…