PaperlayerView exact revision on arXiv

Source-linked AI summary

Novel or Not? PatEx-Gym: Agentic Prior-Art Search for Novelty Assessment in Patent Examination

Qiyao Wang, Xinyi Chen, Hongbo Wang, Jiayuan Lu, Jianhang Li, Minghuan Tan, Yuan Lin, Min Yang

arXiv:2610.33503v1cs.IRcs.AIcs.CL

TL;DR

Patent examination requires agents to discover and assess prior art through an iterative search-and-reasoning process, while existing benchmarks often constrain candidate sets or isolate components. PatEx-Gym combines a benchmark and harness, annotation-free trajectory synthesis, and a 4B trained agent; its experiments show that prior-art discovery remains difficult and the compact agent is competitive with larger agents.

  • Problem

    Existing patent-examination benchmarks are often static or narrowly scoped, while realistic novelty assessment requires discovering prior art and using evidence to guide claim-level reasoning and further search.

    Source: §1 · pp. 1–2

  • Method

    PatEx-Gym combines a benchmark and execution harness, PASS trajectory synthesis using document visits as implicit relevance signals, and a 4B PatEx-Agent trained by supervised fine-tuning on curated trajectories.

    Source: §1 · pp. 2–3

  • Results

    Prior-art discovery and evidence inspection remain challenging even for frontier models, while the 4B PatEx-Agent achieves prior-art discovery performance comparable to agents with ∼30B parameters.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    The paper presents patent examination as a demanding testbed for agentic search and deep research, with evidence inspection and selection remaining central challenges.

    Source: §6 · p. 15

    Searchable prior art for each application is restricted to documents published before that application’s filing date.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Patent examination is a challenging real-world task that requires agents to discover sparse prior art, iteratively refine queries, inspect technical evidence, and produce novelty assessments documented in Office Actions (OAs) across heterogeneous scientific and technical domains. However, existing benchmarks largely focus on static and limited candidate sets, leaving the full search--reasoning--assessment process underexplored. We present PatEx-Gym, the first framework for developing and evaluating agents for patent examination, which integrates three components. First, PatEx-Bench provides 1K realistic examination tasks, two large-scale retrieval corpora containing over 1M patents and 52K scientific papers, and a harness runtime supporting iterative search, document visits, reasoning, and OA generation. Second, we introduce PASS, an annotation-free Prior-Art Agentic Search Trajectory Synthesis framework. Specifically, it exploits document visits as implicit relevance signals and evidence-acquisition anchors to model intra-turn query behavior, inter-turn search progression, and trajectory-level evidence and outcome quality, yielding PatEx-SynData. Third, we train PatEx-Agent at the 4B scale via supervised fine-tuning on the curated trajectories, establishing a lightweight baseline. Extensive experiments show that final decision accuracy alone does not fully reflect examination quality, while prior-art discovery remains difficult even for strong frontier models. Our PatEx-Agent achieves competitive performance against substantially larger general-purpose agents. These results show that training on PatEx-SynData enables a compact 4B model to serve as a competitive baseline for prior-art search and patent examination. We view patent examination as a demanding testbed for agentic search and deep research agents.

1. Introduction

Patent novelty assessment requires discovering prior art, comparing it with claim limitations, and using findings to guide further search and produce an Office Action. PatEx-Gym unifies evaluation, trajectory synthesis, and agent training to study that process.

1. Introduction

  • Novelty assessment requires discovering prior art from a large search space, comparing it with claim limitations, and using intermediate findings to guide subsequent search and reporting.This goes beyond matching or reasoning over a pre-identified set of documents.Source: §1 · pp. 1–2
  • Existing benchmarks remain static or narrowly scoped, evaluating components separately, using small predefined candidate pools, or assuming relevant prior art is already available.The authors argue these settings cannot capture adaptive, iterative evidence discovery and reasoning.Source: §1 · p. 2
  • PatEx-Bench provides 1K realistic examination tasks, large-scale patent and scientific-paper corpora, and tools for prior-art search, reasoning, and Office Action generation.The benchmark exposes the search–reasoning–assessment process rather than evaluating only the final Office Action.Source: §1 · p. 2
  • PASS uses document visits as implicit relevance signals and evidence-acquisition anchors to model search behavior and progression, producing trajectories for agent training.The resulting PatEx-SynData contains 3K filtered trajectories.Source: §1 · pp. 2–3
  • The 4B PatEx-Agent performs comparably to models with ∼30B parameters and achieves Searched Recall on par with Qwen3.7-Max.The authors present it as a lightweight baseline for prior-art search and novelty assessment.Source: §1 · p. 3

2. Related Work

Agentic search research studies systems that plan and adapt search through sequential retrieval and reasoning, while patent examination benchmarks increasingly incorporate evidence and reasoning. The paper positions PatEx-Bench as evaluating the complete trajectory over large prior-art corpora.

2.1. Agentic Search and Deep Research

2.2. Patent Retrieval and Examination

  • Patent retrieval benchmarks such as NTCIR Patent Retrieval [Fujii et al., 2004] and CLEF-IP support work on lexical retrieval, query expansion, cross-lingual matching, and patent-specific ranking.The broader examination process also involves query refinement, inspecting disclosures, and legally grounded reasoning.Source: §2.2 · pp. 3–4
  • Recent patent benchmarks add finer-grained evidence and reasoning, but evaluate stages separately, restrict retrieval to small fixed pools, or limit evidence to a given document.PatEx-Bench instead evaluates autonomous search across large-scale prior-art corpora and the full search–reasoning–assessment trajectory.Source: Table 1 · p. 3, §2.2 · p. 4

3. PatEx-Bench

PatEx-Bench evaluates agents that autonomously find temporally eligible prior art, inspect disclosures, reason over claims, and produce traceable Office Actions. Its harness records the process, and its evaluation covers document quality, decisions, statutory grounds, and evidence retrieval through selection.

3. PatEx-Bench

  • PatEx-Bench requires agents to discover eligible patents and non-patent literature, inspect disclosures, and produce an evidence-grounded novelty assessment as a traceable Office Action.Tasks run in an isolated harness over Patent-1M and Paper-52K, allowing evaluation of the full trajectory as well as the final Office Action.Source: §3 · p. 4

3.1. Problem Definition

  • For each application, the searchable corpus includes only prior-art documents published before its filing date.The corpus combines patent documents and scientific papers.Source: §3.1 · p. 4

3.2. Auditable Execution Harness Runtime

  • The runtime provides Boolean, sparse, and dense search plus document visits that reveal richer content, including claims, as agents inspect selected results.The harness records auditable trajectories, search logs, final environment state, and requested artifacts; execution is separated from preparation and evaluation.Source: §3.2 · pp. 4–5

3.3. Evaluation Criteria

  • Evaluation covers Office Action quality, decision accuracy, statutory-ground accuracy, and recall as references move from search to visits and final selection.Decision Accuracy compares predicted and gold outcomes; Statute F1 compares statutory-ground sets, while Search-R, Visit-R, and Select-R measure reference recall at successive stages.Source: §3.3 · pp. 5–6

3.4. Data Collection, Processing and Statistics

  • The benchmark contains 1,000 tasks across all eight IPC sections, with 2,946 examiner-cited patents and 178 cited papers; its corpora contain 1,011,901 patents and 52,105 scientific papers.The underlying records use examiner-authored Office Actions, and every examiner-cited oracle reference is verified as retrievable from the corpora.Source: §3.4 · p. 6

4. PatEx-SynData and PatEx-Agent

PASS uses document visits to segment patent-search trajectories and assess search behavior, evidence grounding, and examination outcomes. It selects high-quality trajectories for PatEx-SynData, which supports supervised fine-tuning of PatEx-Agent.

4.1. Prior-Art Search Trajectory Synthesis

  • PASS synthesizes prior-art search trajectories and evaluates their quality at both turn and trajectory levels, using document visits as relevance signals and evidence-acquisition anchors.The framework is annotation-free and uses a teacher model with structured patent documents as inputs.Source: §4 · p. 6, §4.1 · p. 7

4.1.1. Visit-Guided Relevance-Aware Rollout Segmentation

  • Each search turn groups search actions followed by visits; a new turn begins when the agent searches after consuming evidence visited in the preceding turn.Multiple search or visit tools may be invoked in parallel within a single tool call.Source: §4.1.1 · p. 8

4.1.2. Hierarchical Query Behavior and Trajectory Quality Modeling

  • PASS assesses query behavior within turns, search progression across turns, evidence grounding, and final examination quality rather than selecting trajectories by outcome alone.The intra-turn score measures efficiency in turning retrieved candidates into inspections, while the inter-turn score measures substantive query refinement in response to visited evidence.Source: §4.1.2 · p. 8

4.1.3. Tournament-Based Trajectory Ranking for Rejection Sampling

  • PASS rejects trajectories missing an Office Action or selected references, or exceeding the turn budget, then ranks survivors through pairwise tournaments across four quality dimensions.The aggregation preserves the score vector instead of collapsing distinct failure modes into one scalar.Source: §4.1.3 · p. 9

4.2.1. Data Recipes

  • From 10K teacher-generated trajectories, PASS filters invalid rollouts, retains 6,321 within budget, and selects the top approximately 3K as PatEx-SynData.The 10K seed patents are balanced across IPC sections and disjoint from PatEx-Bench; Figure 4 also shows token-length and tool-call distributions.Source: §4.2.1 · p. 9, Figure 4 · p. 9

4.2.2. Training Recipes

  • PatEx-Agent is trained with supervised fine-tuning on trajectories represented as model outputs and harness-returned tool results, while system, user, and harness-message tokens are masked from the objective.The model outputs comprise reasoning and tool calls.Source: §4.2.2 · pp. 9–10

5.1. Experimental Setup

The experiments compare frontier and open-source models, retrievers, and agents on PatEx-Bench. PatEx-Agent is fine-tuned from Qwen3.5-4B on PASS-generated data, and evaluations use the Codex harness and multiple LLM judges.

Baselines

  • The evaluation spans 13 models across frontier and open-source families and parameter sizes, plus 12 retrievers evaluated on the PatEx-Bench patent corpus.The retrievers include sparse, dense, and domain-specific systems.Source: §5.1 · p. 10

Implementations

  • PatEx-Agent uses Qwen3.5-4B as its backbone and is fine-tuned for three epochs on PASS-generated PatEx-SynData, with GLM-5.3 as the teacher.Training uses eight NVIDIA H20 GPUs and a learning rate of 1 × 10^-5.Source: §5.1 · p. 10
  • Models are evaluated in the Codex harness for up to 50 steps, with performance assessed across multiple dimensions and scores averaged across three LLM judges.Judges use temperature 0; models supporting configurable reasoning effort use their highest available setting.Source: §5.1 · p. 10

5.2. Main Results on PatEx-Bench

PatEx-Agent improves over its Qwen3.5-4B backbone across search and reasoning metrics, while even strong models struggle to discover and select prior art. The results also show that decision accuracy alone does not capture overall examination quality.

5.2. Main Results on PatEx-Bench

  • +42.7% in OA quality Score, from 3.79 to 5.41, and gains of +17.5% to +91.7% across search and reasoning metrics are reported for PatEx-Agent over its Qwen3.5-4B backbone.The authors report these results as evidence that performance is not solely determined by model scale.Source: §5.2 · pp. 10–11
  • 38.7 Search-R for PatEx-Agent surpasses Qwen3.5-397B-A17B, while Claude-Opus-5-max leads overall Score, Statute F1, and all three reference-recall metrics.Claude-Opus-5-max achieves Score 7.34; the reported model Scores range from 3.79 to 7.34.Source: §5.2 · p. 10, Table 2 · p. 11
  • 87.6% Decision Accuracy for Qwen3.6-27B accompanies lower Score and Select-R than Claude-Opus-5-max: 5.96 vs. 7.34 and 19.5 vs. 32.4, respectively.Decision Accuracy has Kendall’s τ of 0.550 with Score; the three LLM judges have Kendall’s W of 0.849.Source: §5.2 · p. 11
  • 44.1 Search-R for DeepSeek-V4-Flash-0731 accompanies 24.7 Select-R; PatEx-Agent improves over its backbone from 23.2 to 38.7 Search-R and 12.0 to 18.2 Select-R.The respective reported gains for PatEx-Agent are +66.8% in Search-R and +51.7% in Select-R.Source: §5.2 · p. 11

5.3. Prior-Art Search Performance Analysis

Dense retrievers outperform sparse retrievers on Patent-1M, and PatEx-Agent improves retrieval over its backbone. On scientific papers, its performance remains competitive with much larger models across search, visit, and selection stages.

Patent retrieval performance

  • R@100 rises from 24.06 to 36.36 (+51.1%) for PatEx-Agent over its backbone, while PatEmbed-Large leads all retrieval cutoffs and dense retrievers consistently beat sparse ones.PatEmbed-Large’s advantage over SPLADE-v3 increases with retrieval depth.Source: §5.3 · pp. 11–12, Table 3 · p. 12

Non-patent literature search

  • 69.4 Paper Search-R nearly matches Qwen3.8-Max at 69.6, while PatEx-Agent reaches 37.6 Visit-R and 27.7 Select-R.It outperforms many substantially larger models at later stages.Source: §5.3 · pp. 12–13, Figure 5 · p. 12

5.4. Ablation Study

Ablations show that curated trajectories and a complementary retrieval toolset support stronger search and evidence selection. Stronger semantic retrieval can improve recall without improving final evidence selection.

5.4.1. Ablation on Trajectory Sampling Strategy

  • Select-R reaches 18.24 (+51.6%) with PASS-filtered trajectories, while random sampling lowers it from 12.03 to 11.96 (-0.6%).With the same 3K trajectory budget and training settings, random sampling also reduces R@10 from 14.9 to 11.6 (-22.1%) and R@20 from 19.2 to 17.1 (-10.9%).Source: §5.4.1 · p. 13

5.4.2. Ablation on Search Toolset

  • The full toolset gives the strongest retrieval; under Boolean-only search, PatEx-Agent raises Search-R from 10.08 to 25.12 (+149.2%) and Visit-R from 6.35 to 16.14 (+154.2%) over its backbone.BGE-M3 performs best among isolated tools, but the full toolset remains stronger in Search-R and Select-R.Source: §5.4.2 · p. 13, Figure 7 · p. 13

5.4.3. Ablation on Semantic Search Tool

  • For Qwen3.5-4B, PatEmbed-Large raises Search-R from 23.20 to 24.60 and R@100 from 24.06 to 24.78, while Select-R falls from 12.00 to 11.60.The authors suggest stronger retrieval may introduce more similar candidates, making evidence selection harder.Source: Figure 7 · p. 13, §5.4.3 · p. 14

5.5. Case Study

The case study contrasts successful retention of relevant prior art through the examination pipeline with a failure to use a retrieved reference in the final evidence set.

Successful prior-art retention

  • In the successful example, the agent decomposes the claim, reformulates queries across tools, verifies prior art, and assigns evidentiary roles before producing the correct decision and statutory ground.The oracle reference is retained through Search–Visit–Select.Source: §5.5 · p. 14

Failure to use retrieved evidence

  • In the contrasting example, the agent retrieves the oracle reference but does not promote it to document inspection or final selection, yielding an incomplete statutory prediction without incorrect grounds.The case study concludes that broad retrieval alone is insufficient without verified, correctly selected evidence.Source: §5.5 · p. 14

6. Conclusion

The conclusion presents PatEx-Gym as a framework for developing and evaluating agentic patent examination models, including a lightweight 4B baseline. Its experiments identify evidence inspection and selection, alongside prior-art discovery, as continuing challenges.

Conclusion

  • Final decision accuracy alone does not fully capture examination quality, and prior-art discovery, evidence inspection, and selection remain challenging even for strong frontier models.The conclusion identifies inspection and selection as central challenges for agentic search and deep research.Source: §6 · p. 15
  • Despite its 4B scale, PatEx-Agent substantially improves over its base model and achieves prior-art discovery performance comparable to agents with approximately 30B parameters.PatEx-Gym comprises PatEx-Bench, PASS for synthesizing PatEx-SynData, and the lightweight PatEx-Agent baseline.Source: §6 · pp. 14–15
  • The authors propose that effective agents coordinate adaptive querying, tool use, evidence integration, and reasoning throughout the search process.They present PatEx-Gym as a testbed for advancing evidence-grounded agentic search and deep research.Source: §6 · p. 15
Loading 2610.33503v1…