Source-linked AI summary
Stopwords in Technical Language Processing
Serhad Sarica, Jianxi Luo
TL;DR
Engineering NLP commonly uses general stopword lists, but technical texts contain additional frequent, uninformative terms and lack a standard technical list. The paper analyzes a large patent corpus with multiple statistical metrics, then validates candidates through expert evaluation. It produces a statistically identified and human-evaluated complementary stopword list for technical language processing.
Problem
General stopword lists may miss frequent, uninformative terms specific to engineering and technical texts, for which no standard list exists.
Method
The paper preprocesses patent texts, ranks candidate stopwords using TF, TFIDF, and entropy, and validates them through human evaluation.
Results
The study derives a final list of 87 technical stopwords by integrating 62 newly identified stopwords with 25 qualified stopwords from a previous study.
Takeaways & Limitations
The final list can complement NLTK and USPTO stopword lists in NLP and text analysis involving technology, engineering design, and innovation.
Abstract
from arXiv · showhide
There are increasingly applications of natural language processing techniques for information retrieval, indexing and topic modelling in the engineering contexts. A standard component of such tasks is the removal of stopwords, which are uninformative components of the data. While researchers use readily available stopword lists which are derived for general English language, the technical jargon of engineering fields contains their own highly frequent and uninformative words and there exists no standard stopword list for technical language processing applications. Here we address this gap by rigorously identifying generic, insignificant, uninformative stopwords in engineering texts beyond the stopwords in general texts, based on the synthesis of alternative data-driven approaches, and curating a stopword list ready for technical language processing applications.
1 Introduction
Engineering texts contain frequent, uninformative terms that generic English stopword lists may miss. The paper proposes identifying additional technical stopwords through data-driven analysis and human evaluation for direct filtering use.
- Generic stopword lists may miss terms that are less prevalent in layperson language but frequent in engineering and technical texts.
- The paper identifies generic, insignificant, and uninformative stopwords in engineering texts beyond general-language stopwords.
- The resulting list is statistically identified and human-evaluated for technical language processing applications.
- Researchers can apply the list to denoise and filter technical text without manually discovering and removing uninformative words.
2 Our approach
The approach mines a large patent corpus, ranks candidate stopwords with multiple statistical metrics, and uses expert evaluation to confirm which terms are uninformative for technical texts. Human review is necessary because technically significant phrases can be statistically indistinguishable from stopwords.
- Corpus: The corpus contains 781,156,082 tokens from 30,265,976 sentences in 6,559,305 USPTO utility patents from 1976 through 31 December 2019.
- Candidate identification: TF, TFIDF, and information entropy automatically identify candidate stopwords in technical-language texts.
- Human evaluation: Human experts evaluate candidate terms for technical insignificance because significant phrases can resemble stopwords across statistical metrics.
- Preprocessing: Preprocessing removes punctuation, lowercases text, detects phrases, and lemmatizes patent natural texts before candidate ranking.
3 Implementation
The implementation preprocesses a large patent corpus, detects phrases, removes existing stopwords and rare terms, then ranks candidate stopwords using four statistics and human evaluation. This pipeline produces a final technical stopword list by combining data-driven identification with prior validated terms.
- Corpus preparation: Preprocessing lowercases and tokenizes sentences, preserves hyphens and slashes, lemmatizes terms with POS tagging, removes NLTK and USPTO stopwords, and discards singleton terms.After lemmatization, the vocabulary contains 8,144,852 terms, including 6,418,992 phrases; after later filtering, it contains 6,645,391 terms, including 5,834,072 phrases.
- Corpus preparation: Phrases are detected through repeated cooccurrence-based n-gram extraction, using thresholds of 5 and 2.5 to identify terms up to four grams.The first run detects bigrams with a higher threshold, while the second enables combinations of bigrams with a lower threshold.
- Term statistics: Four metrics—TF, IDF, TFIDF and Shannon entropy—are combined to identify frequently occurring terms with little information about engineering and technology.Higher entropy indicates more even distribution across patents and therefore less information about the patents where a term appears; the four distributions exhibit complementary occurrence patterns.
- Candidate evaluation: The procedure unions the top 2,000 terms from four ranked lists, yielding 2,305 overlapping candidates that two researchers evaluate before final selection.The lists are sorted by decreasing TF, increasing IDF, increasing TFIDF and decreasing entropy.
- Final list: The final list contains 87 technical stopwords, integrating 62 newly identified terms with 25 qualified terms from the previous study.The authors suggest applying the technical, NLTK and USPTO stopword lists together in technical language processing applications.
4 Concluding Remarks
The study combines multiple statistical metrics with human evaluation to produce a validated stopword list for engineering and technology texts.
- Four statistical metrics produced highly overlapping candidate stopword results.The metrics ranged from term frequency to entropy.
- Human evaluations validated, screened, and finalized stopwords from the automatically generated candidates.The evaluations achieved high inter-rater reliability, suggesting evaluator independence.
- The final list complements NLTK and USPTO stopword lists in technology, engineering design, and innovation text analysis.
Appendices
The appendices provide ranked term lists and distributions for four statistics, alongside a comparison with stopwords identified in an earlier study.
- Table A1 lists the top 30 terms ranked by term-frequency, IDF, TFIDF, and entropy.
- Table A2 marks terms shared with the previous study, newly defined as stopwords, and no longer considered stopwords.Asterisks indicate terms also identified in the current study, while plus signs indicate current-study stopwords.
- Figure A1 shows term distributions by term-frequency, IDF, TFIDF, and entropy, with filtered term-frequency and TFIDF histograms retaining longer right tails.