PaperlayerView exact revision on arXiv

Source-linked AI summary

From Rules to Neural Graphs: Scalable Structured Prediction for Patent Prior Art Search

Nikolai Zenovkin, Sebastian Björkqvist

arXiv:2610.01553v1cs.IRcs.CL

TL;DR

Patent search must cover very long documents, but neural retrieval often truncates inputs and graph retrieval depends on brittle rule parsers. This paper introduces a neural parser using local biaffine attention and shared weights for long-document deployment. The resulting graphs surpass the rule-based teacher in downstream citation recall at lower inference cost.

  • Problem

    Patent descriptions are extremely long, while dense retrieval models typically process truncated inputs and graph-based retrieval depends on brittle rule parsers.

    Source: §1 · p. 2

  • Method

    The paper distills a neural graph parser from rule-parsed documents, using shared-weight biaffine scoring to train on short sequences and deploy on long ones.

    Source: §1 · p. 3

  • Results

    The neural parser surpasses its rule-based teacher on downstream citation recall while reducing inference cost by 3×.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    Neural structured prediction can replace brittle rule-based patent graph extraction while supporting long documents and improving downstream search.

    Source: §6 · p. 14

    The chemical subset’s gains do not reach significance, and domain-specific evaluation protocols remain future work.

    Source: Search Recall · p. 11

Abstract

from arXiv · show

Patent search requires processing documents routinely exceeding tens of thousands of tokens. Most neural retrieval approaches operate on truncated inputs, limiting their effectiveness. Graph-based retrieval addresses this by representing each patent as a structured invention graph, but constructing these graphs relies on brittle rule-based parsers. We present the neural parser, which adapts biaffine attention from dependency parsing to predict invention graphs directly from patent text. Our local biaffine attention restricts pairwise scoring to a sliding window, reducing complexity from $O(n^2)$ to $O(n \cdot w)$. Since local and global scoring share the same weights, the model trains on short sequences and deploys on documents exceeding 40,000 tokens without retraining. Distilled from 1 million rule-parsed documents, it surpasses its teacher at 3$\times$ lower inference cost: neural graphs improve citation recall by 0.5% on short queries and 1.1% on full documents in a downstream Graph Transformer retrieval system.

1 Introduction

Patent search must handle long documents whose relevant content is missed by truncated neural retrieval, while graph-based systems depend on brittle rule parsers. The paper introduces a neural graph parser designed for long documents and reports improved downstream search with lower inference cost.

1 Introduction

  • Patent documents are long: descriptions in the training corpus average 55,000 characters, and current dense retrieval models typically use only titles and abstracts.The 90th percentile is 104,000 characters; truncation sacrifices coverage of claim-relevant content in the full text.Source: §1 · p. 2
  • Existing graph-based patent search relies on brittle rule parsers that can fail silently on novel constructions and require ongoing manual curation.Scoring all token pairs also makes standard neural structured prediction intractable at patent length, and corpus-scale manual annotation is impractical.Source: §1 · p. 2
  • Shared local and global biaffine weights let the parser train on short sequences and deploy on full-length patent documents without retraining.The approach extends sliding-window biaffine attention to tree/DAG induction.Source: §1 · p. 3
  • The neural parser generalizes beyond hand-crafted patterns, yielding better graph quality and improved downstream prior-art search, with 3× lower inference cost and flexibility for diverse data sources.Supported data sources include human annotation and LLM-generated labels.Source: §1 · p. 3
  • The paper releases an evaluation set, a training-set sample, and the framework used to compute graph-quality metrics.The evaluation set and training sample each contain 5,000 patent documents.Source: §1 · p. 3

2 Related Work

Prior patent retrieval research uses dense representations or invention graphs, but graph-based systems rely on rule parsers. This work applies sliding-window biaffine scoring to long-document tree and DAG induction, with shared weights supporting short-sequence training and long-document deployment.

2.1 Patent Retrieval and Graph Representations

  • Patent dense retrieval models face a length gap: recent benchmarks support at most 4,096 tokens, far below typical patent descriptions.LongEmbed reports embedding degradation beyond 2,048 tokens.Source: §2.1 · p. 3
  • Graph-based patent retrieval improves on bag-of-words baselines, and Graph Transformers achieve further gains, but both depend on rule-based parsers.Invention graphs organize core technical features hierarchically; open relation extraction yields flat triples instead.Source: §2.1 · p. 3

2.2 Neural Structured Prediction for Long Documents

  • The work extends sliding-window biaffine attention from flat span extraction to per-word parent selection for tree and DAG induction at 40,000+ tokens.This targets structured prediction at longer scales than the thousands of tokens addressed by document-level NER.Source: §2.2 · p. 4
  • Local biaffine scoring uses the same weight matrix as global scoring, enabling global-attention training on short sequences and local-attention deployment on long documents without retraining.Unlike DocBAN, the approach does not add a U-Net module for cross-window context.Source: §2.2 · p. 4
  • Knowledge distillation from a rule-based teacher, combined with loss weighting, supplies training signal without corpus-scale manual annotation.Source: §2.2 · p. 4

3 Methodology

The methodology represents patent content as a hierarchical feature graph with functional links, then predicts graph structure using word-level biaffine heads and local scoring. Shared scoring weights support short-sequence training and long-document inference.

3.1 Invention Graphs

  • Invention graphs pair a part-of and example-of feature hierarchy with cross-cutting functional relations; per-word parent predictions and edge labels assemble the structures.Syntactic edges are used in postprocessing to assemble multi-word feature spans.Source: §3.1 · p. 4

3.2 Model Architecture

  • The model combines a contextual encoder with feature and relation heads; two pseudo-nodes classify words as non-features, root features, or words attached to a graph parent.The feature head predicts parent structure and syntactic-versus-semantic edge labels, while the relation head predicts functional relationships.Source: §3.2 · pp. 5–6, Figure 3 · p. 6

3.3 Local Biaffine Attention

  • O(n·w) local biaffine scoring restricts each word to a sliding window plus pseudo-nodes, while shared weights allow short-sequence training and long-document inference without retraining.The model can switch window sizes at inference time.Source: §3.3 · pp. 6–7, Figure 4 · p. 7

Training Objective

  • Weighted cross-entropy trains the feature-parent and relation heads, down-weighting the dominant None class while the edge-label head uses standard cross-entropy.The None-class weights are 0.05 for the feature head and 0.1 for the relation head; other positions receive weight 1.0.Source: Training Objective · p. 8

Dataset Construction

  • Training labels come from production rule parser on 1 million jurisdiction-stratified documents; training truncates to 600 words with at least 70% graph coverage, while 5,000 test documents are processed full-length.Patent families appearing in either evaluation set are excluded from training.Source: Dataset Construction · p. 8

4 Experimental Setup

The evaluation measures graph quality against rule-based teacher graphs and tests whether parser changes affect citation retrieval. Search results use examiner-cited references as relevance labels, with Recall@3 as the primary metric.

Intrinsic Graph Quality Metrics

Extrinsic Search Evaluation

  • The rule-based Graph Transformer baseline achieves Recall@3 = 0.416, exceeding PaECTER at 0.280, Stella at 0.273, and GTE-ModernBERT at 0.200.The dense retrieval systems use up to 4,096 tokens per chunk.Source: Extrinsic Search Evaluation · pp. 9–10
  • The extrinsic evaluation uses examiner-cited references as relevance labels and reports citation-averaged Recall@k, with k=3 as the primary metric.It includes all 90,082 queries, 11,698 chemical queries, and 78,384 non-chemical queries.Source: Extrinsic Search Evaluation · p. 10

5 Results

Neural graphs show partial agreement with the rule-based teacher while covering more words, and replacing rule-based parsing improves downstream citation recall. Reparsing both query and corpus graphs produces the larger reported gain.

Intrinsic Graph Quality

  • The neural parser achieves 0.653 IoU with the rule-based teacher, while word coverage exceeds the teacher at 0.861 vs 0.847.The IoU measures distillation fidelity rather than absolute graph quality.Source: Intrinsic Graph Quality · p. 10

Search Recall

  • +1.1% vs +0.5%: reparsing both query and corpus graphs yields the larger overall search gain, and both improvements are statistically significant.The paired McNemar test gives p < 0.001 for descriptions and p = 0.04 for claims.Source: Table 3 · p. 11, Search Recall · p. 11
  • Chemical-query improvements are not statistically significant (p = 0.42 claims, p = 0.14 descriptions), despite the largest qualitative graph-structure improvements.The authors suggest the chemical validation set may be too small to capture the observed gains.Source: Search Recall · p. 11

Graph Structure Analysis

The neural parser produces graphs with greater word coverage and slightly less depth than the rule-based parser, suggesting a more balanced representation.

Graph Structure Analysis

  • The neural parser’s graphs are slightly shallower (depth 2.057 vs 2.157) but have higher coverage across all metrics.For a chemical patent, it identifies 13 chemical constituents as individual features, compared with 3 top-level features from the rule-based parser.Source: Graph Structure Analysis · p. 11

Training Configuration Ablation

The ablations examine training scale, length filtering, loss weights, and root penalization. More training data gives the largest reported single improvement, while length filtering improves recall without reducing IoU.

Training Configuration Ablation

  • Scaling training examples from 100K to 1M raises IoU from 0.614 to 0.651 and coverage from 0.797 to 0.846.The passage attributes this to generalization across a wider vocabulary of feature expressions.Source: Training Configuration Ablation · p. 12, Table 5 · p. 13
  • Training only on documents with ≤600 words, alongside the 70% coverage filter, improves recall from 0.772 to 0.780 while maintaining IoU.Inference still uses full-length documents.Source: Training Configuration Ablation · p. 12
  • The attention-window evaluation measures Recall@3 on all-claims queries; production selects global attention below 5K words, a 2048-word window for 5K–15K, and 1024 above 15K.The adaptive strategy chooses the largest window that fits NVIDIA L4 memory (24 GB).Source: Table 6 · p. 13
  • Increasing the root loss weight does not improve IoU (0.637 vs 0.653) and decreases precision; the final configuration uses equal root weight.The ablation also reports that down-weighting the None class is essential, with asymmetric feature and relation weights outperforming symmetric alternatives.Source: Training Configuration Ablation · pp. 12–13

Window Size and Document Length

Most parent-child relationships occur within 2048 words, supporting local attention for patent graphs. Larger inference windows improve search recall, while an adaptive strategy selects the largest window that fits GPU memory.

Window Size and Document Length

  • 98% of parent-child relationships occur within 2048 words, supporting the use of local attention.Window-size recall evaluation uses all independent claims concatenated as queries because full-corpus global attention exceeds memory.Source: Window Size and Document Length · p. 13
  • Larger inference windows consistently improve recall, with global attention serving as the upper bound.The adaptive strategy assigns each document the largest window that fits the GPU memory budget.Source: Window Size and Document Length · p. 13
  • The adaptive strategy assigns each document its largest feasible window to maximize recall within the GPU memory budget.Source: Window Size and Document Length · p. 13

5.3 Deployment Experience

The neural parser is deployed for online query parsing and has been evaluated for corpus indexing on a 3-million-document subset. Deployment improves query throughput and latency, while challenging structures and scaling requirements remain practical constraints.

5.3 Deployment Experience

  • The neural parser replaces rule-based query parsing in production, with expert evaluation finding structural equivalence for most documents and additional features in some challenging domains.The query-parsing Claims configuration corresponds to +0.5% recall; corpus indexing was validated on 3 million documents, with +1.1% recall, while scaling to 150 million remains in progress.Source: §5.3 · p. 13
  • Throughput rises from 2 to 10 queries per second per CPU instance, and average latency falls from 2–3 seconds to under 1 second.Queries previously timing out after 60+ seconds complete in under 5 seconds; inputs are capped at 40,000 characters.Source: §5.3 · p. 14
  • Batch parsing of 3 million documents took two days on 10 NVIDIA L4 spot GPU instances, at an estimated 3× lower cost than the rule-based system.GPU utilization exceeded 70%; reaching 90%+ utilization would reduce estimated cost to 5× lower, while larger-scale parsing requires orchestration improvements.Source: §5.3 · p. 14
  • Markush structures remain difficult, and training on short documents led the parser to retain claim-related jargon that rules had filtered out.A regex post-filter addressed this issue instead of retraining on longer sequences, which risked degrading metrics.Source: §5.3 · p. 14

6 Conclusion and Future Work

The paper concludes that local attention with shared weights makes biaffine structured prediction tractable for long documents and suitable to replace rule-based patent graph extraction. It identifies expert-annotated data, joint training, and multilingual extension as future work.

6 Conclusion and Future Work

  • Local attention with shared weights makes biaffine structured prediction tractable for long documents and can replace brittle rule-based patent graph extraction.Distillation from 1 million rule-parsed examples yields a model that surpasses its teacher on downstream citation recall while reducing inference cost.Source: §6 · p. 14
  • Future work includes expert-annotated data to move beyond the teacher ceiling, joint training with the search model, and multilingual extension.Source: §6 · p. 14
  • Generative AI tools assisted with proofreading, readability, and figure-generation code; the authors state that scientific content, experimental design, and conclusions are theirs.Source: §6 · p. 14
Loading 2610.01553v1…