PaperlayerView exact revision on arXiv

Source-linked AI summary

Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

Toshiaki Koike-Akino, Vladislav Blaykhman, Ye Wang, Jing Liu, Gene V. Vinokur

arXiv:2609.13422v2cs.AIcs.LGcs.MA

TL;DR

The paper asks whether LLM judges can reliably evaluate and guide revision in complex professional work, where assessment ordinarily requires specialized expertise. It uses Vibe Patenting to generate coordinated patent drafts and evaluate them with a separate QA judge. Judge-guided revision improves judge-assessed quality, while agreement with attorney judgment remains metric dependent.

  • Problem

    Whether LLM judges can reliably evaluate and guide revision of complex professional work remains unclear, and patent-draft assessment ordinarily requires specialized expertise.

    Source: §1 · p. 1

  • Method

    Vibe Patenting transforms technical materials into coordinated patent packages, which a separate QA judge evaluates across five professional dimensions and can guide through structured revision feedback.

    Source: §2 · p. 2

  • Results

    6.71 overall judge score for Instant+QA nearly matched 6.73 for Extra-High without QA by round 4, while judge-guided revision improved as generic revision saturated.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    LLM judges can provide useful evaluation and optimization signals, but gains against the judge should not be assumed to translate directly into gains under expert evaluation.

    Source: §4 · p. 4

    Increasing the judge’s score does not establish corresponding improvement under professional judgment, so metric-specific calibration and independent expert validation remain important.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.

1 Introduction

The paper asks whether LLM judges can reliably evaluate and guide revision of complex professional work, using patent drafting as a demanding testbed. Judge-guided revision improves judge-assessed quality, but agreement with attorney judgment varies by metric.

1 Introduction

  • Patent drafting is a demanding test because complete drafts require strong claims, sufficient disclosure, coherent figures, prosecution resilience, and filing readiness.Meaningful assessment ordinarily requires specialized professional expertise.Source: §1 · p. 1
  • Judge-guided revision continues to improve judge-assessed quality as generic revision tends to saturate, and low-reasoning feedback approaches a substantially more expensive high-reasoning agent.The results are presented as useful optimization signals, not proof of equivalent gains under professional judgment.Source: §1 · p. 1
  • The Vibe Patenting framework analyzes technical evidence into a shared representation, generates coordinated patent materials, and lets a QA judge initiate revision.The package includes claims, specification, figures, and supporting analyses.Source: Figure 1 · p. 2
  • The judge’s agreement with a professional patent attorney is strongly metric dependent, with some dimensions showing meaningful ranking agreement despite score bias and others showing weaker correlation.Improvement under the automated judge does not by itself establish improvement under expert judgment.Source: §1 · pp. 1–2

2 Vibe Patenting: Professional Evaluation Testbed

Vibe Patenting turns heterogeneous technical materials into coordinated patent packages through staged analysis and domain-oriented drafting skills. A separate QA judge scores drafts across professional dimensions and can guide iterative revision.

2 Vibe Patenting: Professional Evaluation Testbed

  • The system produces a coordinated package of claims, specification, figures, and supporting analyses, whose components must remain mutually constrained and technically consistent.Claim scope should be supported by the specification, and figures should reflect disclosed embodiments.Source: §2 · p. 2
  • The drafting pipeline structures inventions with a Problem–Insight–Solution–Effect representation, then uses patent-oriented skills to generate package components.The paper evaluates general-purpose chat AI, meta-agent, and domain-specialized custom-agent configurations.Source: §2 · p. 2
  • A separate patent-QA judge scores drafts on five dimensions using a 1-to-10 rubric and provides critique and revision recommendations.Each draft is evaluated independently five times; the overall score averages the five dimensions.Source: §2 · p. 2
  • Judge-guided revision gives the drafting agent the current draft and structured QA report, unlike generic revision, which receives no QA report.The judge evaluates draft Dt to produce Qt, and the agent uses that report to produce the next revision.Source: §2 · p. 2
  • The skilled human drafter provides a generation baseline, while a separate patent attorney independently evaluates drafts to validate the LLM judge.The attorney’s scores are expert reference judgments, not the human drafting baseline.Source: §2 · p. 3

3 Experiments and Results

Across drafting configurations, judge-guided revision continued to improve judge-assessed quality as generic revision saturated. Independent attorney evaluation showed meaningful but metric-dependent agreement, with calibration differences that limit what judge-score gains establish.

3 Experiments and Results

  • The study evaluated more than one hundred drafts from ten scientific reports, using five repeated LLM-QA evaluations per draft and averaging five dimension scores.Configurations included model and reasoning variants, agent modes, a domain-specialized agent, and a skilled human drafter baseline.Source: §3 · p. 3

3.1 Scaling Professional Patent-Drafting Agents

  • Under GPT-5.6 Sol, agentic scaffolding improved on zero-shot generation, and the domain-specialized custom agent achieved the highest overall judge score.Across model, reasoning, and agent configurations, quality generally increased with computation (Pearson r = 0.63), but similar thinking times could yield substantially different scores.Source: §3.1 · p. 3

3.2 Judge-Guided Iterative Revision

  • Instant with QA rose from 5.43 to 6.71 by round 4, versus 6.33 without QA; Extra-High with QA rose from 6.26 to 7.18, versus 6.73 without QA.Generic revision improved initially but tended to saturate, while QA-guided revision continued improving through round 4.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3 Agreement with Professional Evaluation

  • LLM-QA and attorney scores had an overall association of Pearson r = 0.717, but agreement varied by dimension; Figure Quality aligned most strongly and Disclosure Strength also tracked attorney judgments.Claim Strength showed weak correlation (r = 0.103) despite small absolute error and little systematic bias (0.06); Figure and Disclosure scores tracked relative judgments while scoring higher than attorney ratings.Source: §3.3 · p. 4, Table 1 · p. 4
  • Higher LLM-judge scores do not establish corresponding improvement under professional judgment; therefore, metric-specific calibration and independent expert validation remain important for professional agentic workflows.Source: §3.3 · p. 4

4 Conclusion

The paper uses patent drafting as a testbed for LLM judges and presents Vibe Patenting, an agentic workflow for turning technical materials into coordinated patent packages. It concludes that judge-guided revision can improve judge-assessed quality, while attorney comparisons show metric-dependent agreement and calibration.

4 Conclusion

  • Vibe Patenting shows that judge-guided revision improves judge-assessed quality and can bring a low-reasoning agent close to substantially more expensive high-reasoning generation, but attorney agreement and calibration vary by metric.The authors caution that gains against the judge should not be assumed to translate directly into gains under expert evaluation.Source: §4 · p. 4

A.1 LLM-as-a-Judge

  • Prior LLM-judge research motivates scalable evaluation of open-ended outputs, while studies of judge reliability identify variation, bias, and weaknesses on difficult comparisons.This work examines complete professional artifacts and feeds structured judge critiques back into multi-round revision.Source: §A.1 · pp. 7–8

B Vibe Patenting: Agentic Framework

  • Vibe Patenting decomposes patent drafting into invention understanding, protection-oriented reasoning, document generation, and quality assurance, producing claims, a specification, figures, and supporting analyses.Its inputs can include research papers, technical reports, slides, experimental results, and source code.Source: B Vibe Patenting: Agentic Framework · p. 9

B.1 From Technical Evidence to Patent Package

  • The package components are mutually constrained: claims need specification support, figures should reflect disclosed embodiments, and the package must remain consistent with the technical evidence.The workflow proceeds through evidence ingestion, invention structuring, specialized package generation, and QA with possible revision.Source: §B.1 · p. 9

B.2 PISE: A Structured Invention Representation

  • PISE organizes an invention as Problem, Insight, Solution, and Effect, with the solution addressing the problem through the insight and effects attributable to the solution.It provides an intermediate abstraction between scientific or technical documents and patent organization, and can represent multiple related inventive concepts.Source: §B.2 · p. 10

B.3 Professional Patent-Drafting Skills

  • Patent-oriented skills share the invention representation and prior-stage artifacts to coordinate strategy, claims, embodiments, specification, figures, and supporting analysis.For example, specification drafting expands concepts into enabling embodiments, while figure generation aligns graphical structures with the specification.Source: §B.3 · p. 10

B.4 Professional Agent Implementations

The implementations differ in how they encode professional patent-drafting knowledge: a zero-shot generalist, a general orchestrator using reusable skills, or a specialized patent agent. The workflow produces a coordinated patent package from technical material.

B.4 Professional Agent Implementations

  • The three architectures range from zero-shot generalist drafting to modular patent skills orchestrated by a general agent and a specialized patent agent.The custom agent places patent-specific workflow logic and drafting skills directly in its configuration.Source: §B.4 · pp. 10–11, Figure 4 · p. 11

B.5 End-to-End Workflow

  • Both agentic implementations transform technical source material into a structured invention representation, protection decisions, claims, specification, figures, analyses, and QA.The workflow notation identifies these outputs as ZPISE, R, C, S, F, A, and Q, respectively.Source: §B.5 · p. 11

B.6 Custom Patent Agent

  • The custom agent bundles patent-specific instructions, reference materials, templates, workflow, and specialized skills in a dedicated domain agent.It is instructed to analyze the invention, construct PISE, develop a protection strategy, and coordinate a complete patent package.Source: §B.6 · p. 11, §B.4 · p. 11

B.7 Meta-Agentic Implementation

The meta-agentic system composes reusable patent skills through a general orchestrator, using PISE to structure and extend invention concepts and guide downstream drafting and QA.

B.7 Meta-Agentic Implementation

  • The meta-agentic implementation separates a general orchestrator from reusable patent-specific skills, while the custom agent embeds more professional behavior in its dedicated configuration.Both use professional structure; they differ in where that structure is represented.Source: §B.7 · p. 12

C.1 PISE Structure

  • PISE organizes invention-oriented reasoning as Problem, Insight, Solution, and Effect, linking a technical limitation to an inventive insight, its implementing mechanism, and resulting consequences.The representation is reused for claim scope, fallback limitations, embodiments, specification, figures, and QA criteria.Source: §C.1 · pp. 13–14

C.2 Multiple PISE Units

  • When technical work supports multiple inventive concepts, the system maintains distinct PISE units and can selectively combine them for independent claims and fallback positions.The hierarchy also extends a unified core and explores additional units through reasoning about root causes, solution impacts, and possible extensions.Source: §C.2 · p. 14

D.1 Patent QA Skill

  • The patent QA skill reviews drafts and figures for claim, disclosure, and filing defects, cross-checking claim limitations against application support and supplemental technical sources.It distinguishes intrinsic draft findings from supplemental-source and web-assisted findings, and prioritizes findings by severity before scoring.Source: §D.1 · pp. 14–16

D.2 Scoring Rubric

  • The scoring rubric reserves the highest scores for filing-ready drafts and assigns lower scores to increasingly material, systemic, or severe deficiencies.Scores of 10 and 9 correspond to essentially filing-ready and filing-ready work, while 1 denotes fundamental deficiency requiring substantial reconstruction.Source: §D.2 · p. 17

D.3 Patent Quality Dimensions for QA Scoring

  • QA scores five dimensions: prosecution resilience, claim strength, disclosure strength, figure quality, and filing readiness.The dimensions use a ten-point scale, with mean scores from five QA agents; the framework also supports comparison with a professional patent attorney's evaluations.Source: §D.3 · p. 17

E QA Judge-Guided Revision

A separate QA agent evaluates each patent draft and returns structured scores and revision recommendations; the drafting agent can use that report in subsequent rounds. The section also describes the prompts and representative artifacts used in the workflow.

E QA Judge-Guided Revision

  • A separate QA agent evaluates each draft and returns a structured report that the drafting agent uses with the draft and original technical context to revise iteratively.The report provides a stable domain-specific interface across drafting models and enables measurement of quality across revision rounds.Source: E QA Judge-Guided Revision · pp. 17–18, Figure 7 · p. 18

E.1.1 Instruction Prompting

  • Agents receive the technical material and a concise patent-drafting request, while the prompt withholds the proposed systems' full professional workflow and modular skills.The same task prompt is used across reasoning-effort settings whenever possible.Source: §E.1.1 · p. 18, §E.1 · p. 18

E.1.2 Patent QA Prompt

  • The QA agent evaluates a completed draft and available figures on five ten-point dimensions and provides structured justification and recommendations.Its prompt invokes the patent-QA skill.Source: §E.1.2 · p. 19

E.1.3 QA Judge-Guided Revision Prompt

  • For guided revision, the drafting agent receives the previous draft and QA report; generic revision uses the draft without that report.The guided prompt explicitly asks the agent to revise according to the quality analysis report.Source: §E.1.3 · p. 19, §E.1.4 · p. 19

F Examples of Generated Patent Artifacts

  • The examples illustrate coordinated artifacts generated from one technical input rather than evaluating the legal merit of an individual application.The framework generates patent-oriented figures rather than directly copying source-publication figures.Source: F Examples of Generated Patent Artifacts · p. 19, §F.1 · p. 19

F.1 Representative Patent Figures

  • Generated patent figures can depict system architectures, processes, component arrangements, embodiments, and interactions in patent-oriented views.Figure 8 presents representative outputs.Source: §F.1 · pp. 19–20, Figure 8 · p. 20

F.2 Representative Claim Generation

  • Claim generation starts from broad protection concepts in PISE and adds narrower dependent limitations as fallback positions.Claim drafting is coordinated with specification drafting so important claim elements have corresponding embodiments and terminology.Source: §F.2 · pp. 20–21

F.3 Representative Specification Expansion

Specification generation expands the invention beyond the technical source's narrow implementation by adding alternative embodiments and design-around variations.

F.3 Representative Specification Expansion

  • Specification expansion can add alternative architectures and parameterizations, software or hardware embodiments, centralized or distributed implementations, optional stages, operation orderings, and variations addressing foreseeable design-arounds.Source: §F.3 · p. 21

G Patent Analysis and QA Report Examples

The patent-drafting system produces analysis artifacts for an internal review loop, while a separate LLM judge generates a structured QA report. The examples show the report’s contents and its five-dimension scoring format.

G.1 Patent Analysis

  • The drafting agent’s analysis report includes invention and prior-art analyses, strengthening strategies, embodiment expansion, claim support, figure alignment, and a recommended plan.These artifacts are intended to expose reasoning behind the protection strategy and improve quality within an internal review loop.Source: §G.1 · p. 21, G Patent Analysis and QA Report Examples · p. 21

G.2 QA Report

  • A separate LLM judge’s QA report covers filing priorities, claims, disclosure and §112, figures, web-assisted research, revisions, and scoring.The QA agent evaluates the complete draft along the five experimental quality dimensions.Source: §G.2 · pp. 21–22
  • Table 3 provides an example of the QA report’s scoring.Figure 9 shows a sample patent-analysis report.Source: Table 3 · p. 22, Figure 9 · p. 22

H.1 Model, Reasoning, and Agent Scaling

Across the evaluated configurations, stronger models, greater reasoning effort, and domain-specific agentic scaffolding generally accompany higher judge-assessed patent quality. The appendix also reports generation time, quality–time associations, revision trajectories, attorney agreement, and QA-score repeatability.

H.1 Model, Reasoning, and Agent Scaling

  • 6.58 overall score for GPT-5.6 Sol Extra High, versus 5.53 for Instant; the custom patent agent scores 7.02 across two patents, an illustrative rather than balanced comparison.GPT-5.6 Sol Medium and High score 5.97 and 6.36; GPT-5.6 agent Extra High scores 6.80. The custom result is based on two patents, compared with seven for several configurations.Source: Figure 10 · p. 23, §H.1 · p. 23

H.2 Per-Dimension Model and Agent Results

  • Figure 11 reports five individual quality dimensions and the overall score for the evaluated model and agent configurations.The supplied passage identifies the comparison’s scope but does not state dimension-level outcomes.Source: §H.2 · p. 23, Figure 11 · p. 24

H.3 Inference-Time Scaling

  • 5.3 minutes for GPT-5.6 Instant chat, versus 41.5 minutes for the GPT-5.6 Extra-High agent.Figure 12 reports mean generation time with one standard deviation over repeated drafting runs.Source: §H.3 · p. 23, Figure 12 · p. 24

H.4 Quality Versus Thinking Time by Dimension

  • Overall thinking-time and quality correlation is approximately r = 0.63, but the configuration-level associations are descriptive given the small, non-randomized set.Figure 13 presents separate quality-versus-time scatter plots by dimension and overall score.Source: §H.4 · p. 25, Figure 13 · p. 25

H.5 Per-Dimension QA-Guided Revision

  • Figure 14 provides five-axis QA-guided revision trajectories that complement the main paper’s overall-score trends.The supplied passage does not state dimension-specific outcomes.Source: §H.5 · p. 25, Figure 14 · p. 26

H.6 Thinking Time Over Revision

  • Instant reasoning is approximately 3.3 times faster than Extra-High reasoning during revision.Figure 15 shows mean and standard deviation of thinking time across revisions with and without QA-judge guidance.Source: §H.6 · p. 25, Figure 15 · p. 26

H.7 AI–Expert Agreement

  • AI–attorney agreement varies by quality dimension: some dimensions show relative association despite score bias, while others show weak discrimination.The comparison reports Pearson and Spearman correlations; correlation and calibration metrics capture different forms of agreement.Source: §H.7 · p. 25, Figure 16 · p. 27, Figure 17 · p. 27

H.8 QA Agent Repeatability

  • Repeated QA scores have within-class standard deviations mostly below 0.5 on the integer scale, suggesting relatively stable scores.Figure 18 summarizes mean within-class standard deviation across QA-agent runs.Source: §H.8 · p. 26

H.9 Summary of Findings

The findings combine gains from judge-guided revision with dimension-dependent agreement between the LLM judge and a professional patent attorney. They frame judge feedback as a useful optimization signal, but not as a substitute for independent expert validation.

H.9 Summary of Findings

  • Judge-assessed quality generally improves with stronger model capability, greater inference-time reasoning, and domain-specific agentic scaffolding, though added computation alone does not guarantee proportional gains.The skilled human drafter is a generation baseline, distinct from the attorney used for expert evaluation.Source: §H.9 · p. 27
  • Judge-guided revision continues to improve judge-assessed quality while generic revision tends to saturate, and low-reasoning generation approaches substantially more expensive high-reasoning generation after repeated feedback.The effect is particularly pronounced for the low-reasoning generator.Source: §H.9 · pp. 27–28
  • Attorney agreement is dimension dependent, and rising scores under judge-guided optimization alone do not establish improvement under professional evaluation.Some dimensions show relative agreement despite systematic bias, whereas others have weak ranking agreement; correlation, calibration, and repeatability are distinct properties.Source: §H.9 · p. 28

I.1 Experimental Limitations

The study’s conclusions are bounded by limited and uneven evaluation coverage, a single attorney evaluator, changing model configurations, and evaluation before prosecution. The paper also identifies professional oversight and confidentiality controls as necessary boundaries for deployment, and lists broader evaluations as future work.

I.1 Experimental Limitations

  • The modest number of inventions and incomplete configuration coverage mean results distinguish balanced evaluations from illustrative comparisons.The paper reports contributing case counts where appropriate.Source: §I.1 · p. 28
  • Expert evaluation relies on one patent attorney, so the study cannot characterize inter-expert variability or a human performance range.The authors identify multiple practitioners and human–human agreement as future evaluation needs.Source: §I.1 · p. 28
  • Inference settings are product-level configurations, not stable algorithms, and their behavior, resource allocation, and latency may change over time.The paper names Instant, High, Extra High, and Pro as examples.Source: §I.1 · p. 28
  • The LLM-based QA evaluator has dimension-dependent calibration bias and is not an objective ground-truth metric.The study compares its scores with professional evaluation.Source: §I.1 · p. 28
  • Evaluation covers draft quality before prosecution, not later outcomes such as allowed claim scope, enforceability, litigation, or commercial relevance.Those factors cannot be established from an initial application alone.Source: §I.1 · p. 29

I.2 Professional Oversight

  • Automated drafting is not intended to replace professional judgment on patentability, inventorship, claim strategy, filing approval, or prosecution.The system’s objective is to reduce repetitive drafting and coordination effort, leaving technical truth and strategic decisions to people.Source: §I.2 · p. 29

I.3 Confidentiality and Data Governance

  • Confidential technical information requires controls for retention, model access, logging, external tools, and disclosure, consistent with organizational policies.Patent drafting frequently involves unpublished and commercially sensitive information.Source: §I.3 · p. 29

I.4 Future Work

  • Future work includes larger and more diverse invention sets, multiple practitioners, reproducible model comparisons, component ablations, and longitudinal prosecution outcomes.The authors also propose adaptive reasoning allocation, learned revision stopping, and extending structured agent workflows to other fields.Source: §I.4 · p. 29
Loading 2609.13422v2…