Large Language Model for Patent Concept Generation

• Patent concept generation remains difficult because patent writing requires specialized expertise and clear, technically precise descriptions that meet strict legal standards.The proposed research question asks how AI can generate innovative concepts to facilitate invention.Source: §1 · pp. 1–2
• PatentGPT combines patent-text knowledge-graph construction, hybrid-input pre-training, invention-focused SFT, and RLHF based on human knowledge recognition.The framework is intended to help the LLM mine, understand, and apply domain-specific knowledge for invention generation.Source: §1 · p. 2
• Continuous Knowledge Injection Pre-training improves subsequent fine-tuning learning capabilities compared with traditional continuous pre-training.Source: §1 · p. 2
• PatentGPT is presented as the first system tailored to generate initial patent concepts suitable for U.S. patent applications.Source: §1 · p. 2
• PatentGPT achieves state-of-the-art performance on patent-related benchmarks with only a minimal sacrifice in general abilities.Source: §1 · p. 2

2.1. Data-driven innovation for concept generation

• C-K Theory describes invention as creating concepts by combining knowledge or conceptions, while patents formalize innovative concepts as new knowledge.Concepts are potential or existing terms; knowledge consists of validated axioms that can support them.Source: §2.1 · p. 2
• Earlier encoding-based methods map concepts and knowledge into vector spaces to analyze existing data, rather than directly generating new concepts and inventions.Examples include patent technology-element dependency networks and dynamic network analysis with LDA.Source: §2.1 · p. 2
• LLMs can learn patent language and technical detail through pre-training and SFT, and generate innovation points from scratch rather than only analyze or predict.Source: §2.1 · p. 2

2.2. Fine-tuning method for patenting

• Patent-focused SFT can teach writing style and requirements, but typically needs substantial annotated data and may generalize poorly beyond its training examples.Source: §2.2 · p. 2
• Prior work explored patent summarization and fine-tuned GPT models for design-concept generation, while another study evaluated patent abstracts without proposing an LLM training method.The cited work spans case studies, fine-tuning, and few-shot learning.Source: §2.2 · pp. 2, 4
• The review identifies a gap in jointly learning structured domain knowledge and text, and proposes KFT to bridge knowledge-based and textual learning.KFT combines knowledge injection from patent triples, patent-specific SFT, and RLHF to refine generated content against human knowledge recognition.Source: §2.2 · p. 4

3. Method

• PatentGPT’s four-stage workflow extracts knowledge graphs, converts triples into training text, applies dialogue SFT, and refines outputs with feedback.The stages are knowledge extracting, injecting, learning, and feedbacking.Source: §3 · p. 4

3.1. Knowledge extracting

• An LLM extracts concept entities and relationships from patent text to construct a knowledge graph of triples.The extraction prompt is based on Luan et al..Source: §3.1 · p. 4

3.2. Knowledge injecting

• Structured triples are converted into natural-language sentences and used with the general corpus for continued pre-training.This converts structured knowledge into text for LLM learning.Source: §3.2 · p. 5

3.3. Knowledge learning

• SFT on dialogue data teaches the model to apply injected knowledge and concepts to task-specific scenarios.The stated aim is more accurate, contextually relevant replies.Source: §3.3 · p. 5

3.4. Knowledge feedbacking

• RLHF uses preference-labeled responses to train a reward model, then PPO fine-tunes the LLM using its reward signal.PPO’s clipping mechanism is used to stabilize reinforcement learning.Source: §3.4 · p. 5

4.1. Experiment setup

• Experiments use Qwen2-1.5B with LoRA fine-tuning for three epochs, batch size 1, and learning rate 5e-5.Fine-tuning and inference use NVIDIA 4090 GPUs; inference temperature is 0.7.Source: §4.1 · pp. 5–6

4.2.1. USPTO patent text dataset

• The USPTO dataset covers 2021–2022 patents across eight IPC sections, with approximately 20,000 titles, abstracts, and claims for pre-training.Each IPC section contains between 2,000 and 4,000 patents.Source: §4.2.1 · p. 6

4.2.2. Patent knowledge graph dataset

• A patent knowledge graph dataset uses 30% of the USPTO patent corpus to produce about 100,000 entities and 300,000 relationships.Extracted triples are converted into natural-language descriptions for knowledge injection.Source: §4.2.2 · p. 6

4.2.3. SFT dialogue dataset

• The SFT dialogue dataset contains 60,000 patent-drafting question–answer pairs covering titles, abstracts, and claims.Source: §4.2.3 · p. 6

4.2.4. Human feedback dataset

• The human-feedback dataset contains 2,000 interactions with preferred and non-preferred answers, with preferred answers annotated by patent practitioners.Source: §4.2.4 · p. 6

4.3. Baselines

• Baselines include models from Qwen, Llama, phi, ChatGPT, Gemini, and Claude, plus the IP-domain model MoZI-7b.The compared models vary in scale and architecture.Source: §4.3 · p. 6

4.4. Metrics

• Evaluation combines text-generation metrics, IP understanding benchmarks, and concept assessments of reasonability and innovation.Text metrics include ROUGE-1, ROUGE-2, ROUGE-L, BLEU-4, and BERTScore; concept assessment uses ChatGPT4 judgments and TechNet scoring against patent databases.Source: §4.4 · pp. 6–7

5.2. Comparison results of different methods

• Without task-specific fine-tuning, KPT and PT performed similarly on BERT Score and BLUE-4; the paper describes KPT as a foundation for later gains.Fig. 4 compares traditional and KFT-based methods across five metrics and shows larger subsequent improvement after domain knowledge is injected.Source: §5.2 · p. 7, Figure 4 · p. 8
• KPT + SFT achieved BERT Score 88.305 and BLUE-4 44.661, versus PT + SFT’s BERT Score 86.621 and BLUE-4 30.848.The authors report larger performance boosts from SFT when it follows knowledge-injected pre-training.Source: §5.2 · p. 7, Table 5 · p. 8

5.1. Patent writing

• 45.36 BLEU-4: PatentGPT outperformed state-of-the-art LLMs across all metrics on patentable-text generation.The reported BERTScore comparison also favors PatentGPT over ChatGPT-4o (86.899) and Gemini (86.541).Source: §5.1 · p. 7, Table 4 · p. 8

5.3. Ablation study

• The ablation reports BLEU-4 dropping by 51.4% and ROUGE-1 by 54.9%, emphasizing SFT’s critical role; removing KPT decreases BLEU-4 by 78.6% and ROUGE-2 by 77.6%.Source: §5.3 · p. 8

5.4. Patent bench

• PatentGPT performed better on most benchmarks, especially IP-specific tasks, while scoring lower than larger models on MMLU and sacrificing some general capability versus same-size Qwen2.The paper characterizes its general capabilities as competitive overall.Source: §5.4 · pp. 8–9, Figure 6 · p. 10

5.5. Reasonability and innovation

• 95% win rate: PatentGPT’s generated concepts met USPTO writing standards more often than all baseline models.Reasonability was assessed by conformity to USPTO standards and quantified by win and loss.Source: §5.5 · p. 9, Figure 7 · p. 10
• 0.578 Average Rareness Score: PatentGPT exceeded MoZI (0.402) and Qwen2 (0.431) on concept novelty measured by distance from existing patent concept combinations.The paper describes the score as a measure of novelty; a higher score indicates greater innovation.Source: §5.5 · p. 9, Figure 8 · p. 11

5.6. Case study

• In the hybrid-vehicle case, PatentGPT’s title was more specific, its abstract more operational, and its claims more comprehensive than ChatGPT-4o’s.The same query asked both models to generate a patent title, abstract, and claims.Source: §5.6 · pp. 9–11

6.1. Implications for Practice and Governance

• KFT offers a way to integrate domain knowledge into LLM training for specialized tasks such as initial patent drafting, while AI-generated patents raise legal, economic, and ethical questions.The authors also illustrate PatentGPT generating a title, abstract, and claims for a hybrid-vehicle concept.Source: §6.1 · pp. 11–12
• Patent-inventorship rulings remain globally unsettled: U.S., European, and UK decisions require a natural-person inventor, while Australia's AI-inventor ruling was overturned.South Africa approved an AI-inventor application, but the passage describes that decision as an outlier.Source: §6.1 · p. 11
• PatentGPT could lower drafting costs and broaden access to patenting, but the authors also note potential displacement of patent-drafting and review roles.They identify originality and authenticity of AI contributions as ethical questions.Source: §6.1 · p. 11

6.2. Limitation and future work

• Because PatentGPT is trained primarily on USPTO patents, it may produce claims that do not meet other jurisdictions’ requirements.The authors propose retrieval-augmented generation, jurisdiction-specific prompts, and cross-jurisdictional datasets as future work.Source: §6.2 · p. 12
• Training data from 2021–2022 may leave PatentGPT short of comprehensive knowledge in emerging fields, risking hallucinated or technically inaccurate claims without external retrieval or sufficient instructions.The authors propose continuous updates with emerging patent data.Source: §6.2 · p. 12
• Novelty scores do not replace legal assessment, and the current evaluation lacks attorney validation and systematic cross-jurisdiction compliance checks; unintentional plagiarism and excessive low-quality filings are also concerns.The authors propose legal-expert evaluation and auditing frameworks to address these risks.Source: §6.2 · p. 13

7. Summary

• PatentGPT combines KPT, domain-specific SFT, and RLHF through the KFT framework to support generation of concepts and patents together.The paper says the approach equips the model with specialized knowledge for generating concepts and inventions.Source: §7 · p. 13
• The paper reports that PatentGPT surpasses existing models on IP-related benchmarks.It also describes the results as setting a new benchmark for AI-driven IP creation.Source: §7 · p. 13
• The authors conclude that PatentGPT can generate concepts that are patentable and that patent texts represent novel and useful concepts.The passage characterizes the work as generating concepts and patents together.Source: §7 · p. 13

CRediT authorship contribution statement

• Runtao Ren is credited with drafting and editing, visualization, validation, software, methodology, formal analysis, data curation, and conceptualization.The passage also credits Jian Ma with supervision, funding acquisition, and conceptualization, and Jianxi Luo with editing, validation, and supervision.Source: CRediT authorship contribution statement · p. 13
• Jian Ma is credited with supervision, funding acquisition, and conceptualization.These roles are listed alongside the contributions of Ren and Luo.Source: CRediT authorship contribution statement · p. 13
• Jianxi Luo is credited with writing review and editing, validation, and supervision.These roles are listed alongside the contributions of Ren and Ma.Source: CRediT authorship contribution statement · p. 13

Appendix B. Prompt for generating knowledge corpus

• The passage refers to interests or personal relationships that could have influenced the reported work.Source: Appendix B · p. 14