Source-linked AI summary
Patent Claim Generation by Fine-Tuning OpenAI GPT-2
Jieh-Sheng Lee, Jieh Hsiang
TL;DR
The paper addresses whether GPT-2 can generate coherent patent claims, a rarely explored language-generation task relevant to augmented inventing. It fine-tunes GPT-2 on patent-claim data, probes early adaptation, and evaluates sampling approaches, finding that the 345M model produces patent claims of reasonable and sometimes highly coherent quality while also revealing substantial quality problems.
Problem
Patent claim language has rarely been explored, motivating automatic generation of coherent claims for potential augmented inventing.
Method
The authors fine-tune GPT-2 on 555,890 first and independent claims, probe its first 100 training steps, and evaluate conditional and unconditional sampling including dynamic_kp.
Results
The fine-tuned 345M GPT-2 model generated patent claims of reasonable quality, with the first patent-like text appearing by step 36 and some outputs described as coherent and complicated.
Takeaways & Limitations
GPT-2 appears viable for augmented inventing and provides generated patent claims, source code, and an e-mail bot for further research.
Takeaways & Limitations
Generated claims sometimes become too long, difficult to understand, divergent, repetitive, or obviously incorrect, and fixing these problems is left for future research.
Abstract
from arXiv · showhide
In this work, we focus on fine-tuning an OpenAI GPT-2 pre-trained model for generating patent claims. GPT-2 has demonstrated impressive efficacy of pre-trained language models on various tasks, particularly coherent text generation. Patent claim language itself has rarely been explored in the past and poses a unique challenge. We are motivated to generate coherent patent claims automatically so that augmented inventing might be viable someday. In our implementation, we identified a unique language structure in patent claims and leveraged its implicit human annotations. We investigated the fine-tuning process by probing the first 100 steps and observing the generated text at each step. Based on both conditional and unconditional random sampling, we analyze the overall quality of generated patent claims. Our contributions include: (1) being the first to generate patent claims by machines and being the first to apply GPT-2 to patent claim generation, (2) providing various experiment results for qualitative analysis and future research, (3) proposing a new sampling approach for text generation, and (4) building an e-mail bot for future researchers to explore the fine-tuned GPT-2 model further.
1. Introduction
The paper applies GPT-2 fine-tuning to patent-claim generation, motivated by the strong performance of pretrained language models and the limited exploration of patent-claim language. Using the released 345M model, the authors report coherent generated claims and rapid emergence of patent-like text.
- Pretrained models such as GPT-2 and BERT have achieved strong results across language tasks, motivating their application to patent claims.
- The authors generated patent claims by fine-tuning GPT-2’s released 345M medium version.
- The generated claims were often coherent and complicated, although quality varied across outputs.
- Patent-like text first appeared after surprisingly few fine-tuning steps, motivating further experimentation with larger models.
2. Related Work
Prior work shows broad use of deep learning for intellectual-property analysis but limited attention to patent generation. The paper positions pretrained Transformer-based language generation as an opportunity for patent applications beyond established tasks such as machine translation.
- Reviews identify knowledge management, technology management, economic value, and information extraction as major AI applications in intellectual-property data.
- Modern neural networks have been used primarily for machine translation in patent-related applications, leaving other tasks relatively open.
- Transformer transfer learning with pretrained language models has produced strong results across NLP tasks and is extended here to patent-focused natural-language generation.
3. Data
The authors construct a GPT-2 training corpus from granted U.S. utility-patent claims, splitting claims into human-aligned spans and encoding them with special boundary markers. This pipeline supports both model training and later identification of generated claim boundaries.
- The dataset contains 555,890 first and independent claims from granted U.S. utility patents in 2013.
- Data pipeline: The data pipeline collects raw claims, splits them into spans, and encodes the spans for GPT-2.
- Span-based data: Claim spans are segments of lengthy claims aligned with human-curated line breaks and intended as a comprehensible unit of inventive thought.
- SQL-based data: The authors use SQL-based preparation to separate data retrieval from processing and make the dataset easier to customize.
- Encoding: A heuristic uses punctuation and omitted line breaks to identify span boundaries, then inserts @@@ between spans and start/end-of-text tags around each claim.
- Encoding: Encoded training data is stored in compressed NumPy format, while both NumPy and plain-text versions are shared for reuse.
4. Experimental Setup
The experiments fine-tune GPT-2 using cloud computing and a memory-efficient training workaround. Although four model sizes exist, the study uses the 345M model within its computational limits and identifies larger models and patent-specific pretraining as future directions.
- Model selection: OpenAI’s GPT-2 models range from 117M to 1.5B parameters, and this study experiments with the 345M model.
- Model selection: The authors found the 117M model sufficient for impressive results but state that larger models generally produce better results.
- Computing environment: Google Colab supplied cloud GPU and CPU resources, with checkpoints restored across sessions when training exceeded the 12-hour session limit.
- Computing environment: Because Colab’s roughly 15GB GPU memory was insufficient for all-layer fine-tuning of the 345M model, the authors used a memory-efficient gradient technique.
- Future directions: The authors identify building a patent-specific pretrained model from a patent corpus as a follow-up direction, including possible use of classification information.
5. Experiments
The experiments probe GPT-2’s early adaptation to patent claims, compare training behavior and sampling strategies, and assess generated-claim quality. Results show coherent and diverse outputs alongside substantial quality failures and unresolved validation questions.
- 5.1. How fast GPT-2 adapts: The authors note that the effectiveness of early fine-tuning is difficult to interpret because the number of patents in GPT-2’s original WebText training data is unknown.They leave clarification of this uncertainty to future work.
- 5.1. How fast GPT-2 adapts: The first 100 steps were archived and special-tag occurrences were used as an intuitive probe of the fine-tuning process.The authors caution that not every occurrence necessarily makes sense.
- 5.2. Training loss: Using a learning rate of 1e-5 produced slower convergence, and no patent-like claim appeared within the first 100 steps.The baseline used learning rate 1e-4, temperature 1.0, top_k 40, and batch size 1.
- 5.3. Unconditional random sampling: Dynamic_kp sets a cutoff relative to the top token’s probability, making the effective k or p threshold adaptive during sampling.The experiment used a cutoff probability of 0.1, but the authors state that its optimal value remains unresolved.
- 5.3. Unconditional random sampling: Across qualitative samples, generated claims showed coherence, practical sense, diversity, and plausible item ordering, but many others were too long, difficult to understand, repetitive, or incorrect.The comparison generated 30 claims each with dynamic_kp (0.1), top_k (40), and top_p (0.9).
- 5.4. Conditional random sampling: Short inputs could produce comparatively long outputs with different, acceptable-quality details, suggesting a possible exploratory use for inventors without a complete idea.The authors describe this as a contemplated use case rather than an established application.
- 5.4. Conditional random sampling: The e-mail-bot test produced a significant amount of nonsensical text, while dynamic_kp appeared slightly better than top_k and top_p without a conclusive overall winner.The authors present the comparison as preliminary rather than definitive.
6. Looking forward
The paper identifies supervised patent knowledge and multilingual modeling as directions for improving patent claim generation. It also highlights cross-lingual generalization as a basis for multilingual approaches.
- Supervised knowledge: Patent classification is proposed as supervised knowledge that could improve patent claim generation when incorporated into a Transformer model.The paper describes a three-stage approach that transfers general language knowledge and knowledge from supervised tasks.
- Multilingual generation: Multilingual patent claim generation could use either a pre-trained model fine-tuned in another language or a multilingual pre-trained model.The paper presents multilingual modeling as a separate direction for generating claims beyond English.
- Multilingual generation: Cross-lingual generalization makes multilingual models compelling because annotations in one language may support fine-tuning for another.The paper attributes this possibility to prior findings about multilingual BERT.
- Multilingual generation: The authors conjecture that supervised knowledge, including patent classification in other languages, could make multilingual patent claim generation more effective.This is presented as a conjecture for future work rather than an established result.
7. Conclusion
The paper concludes that fine-tuned GPT-2 can generate patent claims of reasonable quality and may support augmented inventing. It provides research data and an e-mail bot while identifying larger models and additional annotations as future directions.
- Conclusion: Fine-tuning GPT-2’s 345M model generated patent claims of reasonable quality and was presented as a possible basis for augmented inventing.The authors describe this as a step toward human-machine co-inventing.
- Conclusion: The work contributed hundreds of generated patent claims as research data and provided an e-mail bot for further experimentation.These resources are intended for future researchers to explore the fine-tuned model.
- Future directions: The authors identify larger pre-trained models, patent-only pre-training, and additional human annotations such as patent classification as potential ways to advance claim generation.These directions are framed as promising possibilities for future research.
- Conclusion: The paper characterizes Transformer models as an opportunity for patent research and its work as a step toward human-machine co-inventing.The conclusion uses cautious language about GPT-2’s viability and future potential.
Appendix A
The appendix uses SQL to select first claims of U.S. utility patents in 2013 and aggregate CPC codes at the subclass level. The query joins patent, CPC, and claim data before grouping the results.
- Query purpose: The appendix query selects first claims of all U.S. utility patents in 2013 and aggregates CPC codes at subclass level.The data source is identified as Google Patents Public Datasets on BigQuery.
- Query structure: CPC group identifiers are aggregated into distinct ordered strings for each patent record.The aggregation is expressed through STRING_AGG over distinct t2.group_id values.
- Query structure: The results are grouped by patent identifier, patent date, and claim text.The grouping fields are t1.id, t1.date, and t3.text.