PaperlayerView exact revision on arXiv

Source-linked AI summary

DAPFAM: A Domain-Aware Family-level Dataset to benchmark cross domain patent retrieval

Iliass Ayaou, Denis Cavallucci, Hicham Chibane

arXiv:2506.22141v2cs.CLcs.IR

TL;DR

Patent retrieval benchmarks often lack explicit domain partitions, limiting systematic evaluation of retrieval across technological boundaries. DAPFAM provides a family-level benchmark with IN/OUT partitions and tests multiple retrieval designs. Its results show persistent cross-domain difficulty, while document-only rank fusion offers a practical effectiveness-efficiency trade-off.

  • Problem

    Existing patent benchmarks often lack explicit domain partitions, making systematic evaluation of retrieval under cross-domain shifts difficult.

    Source: §1.2 · p. 2

  • Method

    DAPFAM aggregates patents at the family level, defines IN/OUT partitions by IPC3 overlap, and evaluates lexical, dense, passage, and hybrid retrieval designs.

    Source: §2.2 · p. 5, §5.1 · p. 17

  • Results

    Passage-level retrieval consistently outperforms document-level approaches; dense-passage is optimal on ALL and IN, while BM25-passage is a reliable fallback on OUT.

    Source: §8 · p. 28

  • Takeaways & Limitations

    Document-only reciprocal rank fusion improves the effectiveness-efficiency frontier while avoiding passage indexing requirements.

    Source: §7 · p. 28, §8 · p. 28

    Existing benchmarks are often constrained by single-jurisdiction focus, missing domain partitions, or absent family-level aggregation.

    Source: §1.2 · p. 2

Abstract

from arXiv · show

Patent prior-art retrieval becomes especially challenging when relevant disclosures cross technological boundaries. Existing benchmarks lack explicit domain partitions, making it difficult to assess how retrieval systems cope with such shifts. We introduce DAPFAM, a family-level benchmark with explicit IN-domain and OUT-domain partitions defined by a new IPC3 overlap scheme. The dataset contains 1,247 query families and 45,336 target families aggregated at the family level to reduce international redundancy, with citation based relevance judgments. We conduct 249 controlled experiments spanning lexical (BM25) and dense (transformer) backends, document and passage level retrieval, multiple query and document representations, aggregation strategies, and hybrid fusion via Reciprocal Rank Fusion (RRF). Results reveal a pronounced domain gap: OUT-domain performance remains roughly five times lower than IN-domain across all configurations. Passage-level retrieval consistently outperforms document-level, and dense methods provide modest gains over BM25, but none close the OUT-domain gap. Document-level RRF yields strong effectiveness efficiency trade-offs with minimal overhead. By exposing the persistent challenge of cross-domain retrieval, DAPFAM provides a reproducible, compute-aware testbed for developing more robust patent IR systems. The dataset is publicly available on huggingface at https://huggingface.co/datasets/datalyes/DAPFAM_patent.

1. Introduction

DAPFAM addresses the lack of explicit cross-domain evaluation in patent retrieval benchmarks with a family-level benchmark and systematic IN/OUT partitions. The paper reports that OUT-domain retrieval remains a major challenge and evaluates retrieval design choices to study it.

1. Introduction

  • Patent retrieval is increasingly difficult as patent collections grow and innovations become more interdisciplinary, especially when relevant patents cross technological domains.The paper frames prior-art retrieval as important to innovation assessment, intellectual property strategy, and technological progress.Source: §1 · p. 2

1.1. The Cross-Domain Challenge

  • Cross-domain prior-art search is common, but vocabulary gaps, divergent terminology, and document structures may lead traditional systems to miss relevant connections.Patent examiners routinely find relevant prior art outside an application's primary classification.Source: §1.1 · p. 2

1.2. Dataset Limitations for Cross-Domain Evaluation

  • Existing benchmarks often lack explicit domain partitions, family-level aggregation, or standardized protocols for rigorous comparison under domain shift.The cited benchmark limitations include single-jurisdiction focus and international filing redundancy.Source: §1.2 · p. 2

1.3. Our Contribution

  • DAPFAM provides a family-level benchmark with balanced queries and explicit IN-domain and OUT-domain partitions based on IPC3 overlap.The partitions enable direct measurement of cross-domain retrieval difficulty.Source: §1.3 · p. 2

1.5. Problem Setting and Design Choices

  • The benchmark evaluates document and passage retrieval, query-field combinations, aggregation strategies, and hybrid fusion rather than comparing many models.Evaluation uses NDCG@100 and Recall@100 with BM25, a multilingual dense encoder, and Reciprocal Rank Fusion.Source: §1.4 · p. 3, §1.5 · p. 3

1.4. Novelty & Significance

  • OUT-domain performance drops approximately 5× compared with IN-domain scenarios, and the study evaluates granularity, query representations, aggregation, and hybrid fusion.The authors report that dense methods lose their advantage over lexical approaches in OUT-domain retrieval.Source: §1.4 · p. 3

1.6. Availability and Reproducibility

  • The complete release includes a family-level corpus, balanced query sets with domain labels, citation-based relevance judgments, and metadata for reproducible experiments.The dataset supports varying query and corpus representations, passage window sizes, and family-level aggregation strategies.Source: §1.6 · p. 3

1.7. Structure of This Paper

  • The paper positions DAPFAM against existing patent benchmarks, details dataset construction and statistics, then reports experiments and discusses findings and future directions.The planned results cover granularity, query representations, passage effects, aggregation, efficiency, and cross-domain challenges.Source: §1.7 · p. 3

2. Background and Related Work

Patent retrieval has moved from classical text matching toward neural and hybrid approaches, while patent-specific metadata and domain shifts pose distinct challenges. DAPFAM is positioned as a family-level, domain-aware benchmark addressing gaps in existing datasets.

2.1. Classical IR Approaches and the Emergence of Query Expansion

  • Classical patent retrieval adapted standard text methods, but domain jargon and multilingual filings created additional hurdles as databases grew more diverse.Earlier approaches included TF-IDF and vector-space models; later work used query expansion and patent metadata such as classifications and citations.Source: §2.1 · pp. 4–5

2.2. Neural Methods and Semantic Text Representations

  • Neural representations and transformers aim to capture semantic nuances, but full patent documents can exceed their practical context length.Passage methods aggregate chunk scores using maxP, avgP, sumP, or avg_top3; RRF combines rankings, with document-only fusion trading some headroom for lower indexing cost and latency.Source: §2.2 · p. 5

2.3. Positioning DAPFAM vs Existing Patent Retrieval Datasets

  • DAPFAM's design combines family-level aggregation, IPC3-based domain partitions, and balanced query sampling to reduce redundancy and support cross-domain evaluation.Family-level full text consolidates titles, abstracts, claims, and descriptions; the dataset is intended to remain manageable for passage-level research.Source: §2.3 · p. 6
  • Existing datasets provide citation-based relevance judgments, but none systematically distinguish in-domain from cross-domain retrieval scenarios for rigorous analysis of domain transfer.The cited comparisons note varying limits: CLEF-IP's European focus, TREC Patent Track's chemistry emphasis, and missing query sets or domain labels in MAREC and BigPatent.Source: §2.3 · p. 6

3. Construction of DAPFAM

DAPFAM is constructed from globally sourced patent records through filtering, family consolidation, query and target selection, citation-based relevance labeling, and IPC3-based domain partitioning. The resulting dataset contains balanced query families and a large target pool for retrieval experiments.

3. Construction of DAPFAM

  • The construction process follows a three-step medallion architecture that filters and curates raw patent data into a user-ready dataset.The overview covers data sourcing, query creation, citation-based relevance judgments, and IN/OUT-domain labeling.Source: §3 · p. 8, Figure 1 · p. 8

3.1. Data Source: Lens.org

  • Lens.org was selected for its multi-office citation networks, global jurisdiction coverage, standardized JSONL format, and research accessibility.The source includes patents from USPTO, EPO, WIPO, and other national offices.Source: §3.1 · p. 9

3.2. Three-Step Medallion Architecture

  • The medallion pipeline parses raw records, groups publications into families, filters for complete English text and post-2000 claim dates, then consolidates family text and metadata.Potential queries also had to meet a minimum of 100 combined forward and backward citations.Source: §3.2 · pp. 9–10

3.3. Query and target Selection Process

  • Queries were sampled across IPC3 domains, targeting about 10 per domain; each query's target pool sampled approximately 20 full-text family documents from citation networks.The process produced 1,247 query families.Source: §3.3 · p. 10

3.4. Assigning Relevance via Citation Links

  • Citation-network families receive binary relevance 1, while randomly sampled non-cited families serve as negative examples with relevance 0.Cited families are treated as examiner-validated technical relationships; about 20 negatives per query were sampled for supervised learning and negative sampling.Source: §3.4 · p. 10

3.5. Domain Labeling: In-Domain vs. Out-of-Domain

  • A query-target pair is IN-domain when the families share at least one IPC3 code and OUT-domain when none overlap.The three-character granularity aims to balance domain specificity with sufficient evaluation coverage; primary and secondary classifications are included.Source: §3.5 · p. 11

3.6. Final Dataset Organization

  • The final dataset comprises 1,247 query families and 45,336 target families, organized with relevance and domain labels for flexible experiments.Its modular files support varied query representations, corpus subsets, and evaluation metrics.Source: §3.6 · p. 11

4. Dataset Statistics and Analysis

The dataset spans patent families across IPC3 domains, filing eras, text lengths, and jurisdictions. Its explicit IPC3-overlap labels identify a substantial out-of-domain share of relevant documents.

4. Dataset Statistics and Analysis

  • 1,247 queries and 45,336 target families are linked by 49,869 evaluation records, approximately 20 positive and 20 negative pairs per query.The dataset represents all major IPC3 sections, with query and target distributions shown for the top 50 IPC3 classes.Source: §4 · p. 13, §4.1 · p. 13, Figure 2 · p. 13

4.1. Domain Distribution and Coverage

  • Queries and targets span major IPC3 sections, with Figure 2 showing the frequency of the top 50 classes for each.This coverage supports retrieval experiments across multiple technological boundaries.Source: §4.1 · p. 13, Figure 2 · p. 13

4.2. Earliest Claim Date Distribution

  • Queries cluster in 2000–2015, while target families cover a wider range of earliest claim years.The older cited documents allow retrieval to be examined across publication eras.Source: §4.2 · pp. 13–14, Figure 3 · p. 14

4.3. Temporal Coverage and Citation Patterns

  • The dataset’s temporal scope is 1964–2023, and patent text length grew over 1990–2020.The authors relate modern patents’ greater text length to increased technical complexity and disclosure requirements.Source: §4.3 · p. 14, Figure 4 · p. 15

4.4. Text Length Analysis

  • Queries average approximately 20,448 tokens, compared with approximately 11,090 for targets.The authors suggest this difference may reflect the selection of highly cited query families and note that length variation calls for adaptive retrieval strategies.Source: §4.4 · pp. 15–16

4.5. Jurisdiction Coverage

  • US documents dominate the jurisdiction mix, alongside prominent Japanese, European, Korean, German, and Chinese representation.The text is presented in English at the family level, including families first filed in non-English jurisdictions.Source: §4.5 · p. 16, Table 10 · p. 16

4.6. In-Domain vs. Out-of-Domain Relevance

  • About 26% of relevant documents share no first-three-character IPC code with their query, defining a substantial cross-domain subset.IN/OUT labels are based on IPC3-code overlap; Table 12 presents average counts and percentages by domain relation.Source: §4.6 · p. 16, Table 12 · p. 17

5. Evaluation & Methods

The evaluation ranks candidate patent families across complete, in-domain, and out-of-domain subsets. It varies retrieval granularity, text fields, backends, passage aggregation, and rank fusion across 249 configurations.

5.1. Task Setup and Views

  • Retrieval ranks candidate families; IN shares at least one IPC3 code with a query, OUT shares none, and ALL contains the complete test set.Document retrieval uses one consolidated text unit per family; passage retrieval scores fixed-size windows and aggregates them into a family-level score.Source: §5.1 · p. 17

5.2. Metrics

  • NDCG@100 and Recall@100 are macro-averaged over queries in ALL, IN, and OUT, with rank 100 chosen to balance early precision and coverage.The cutoff reflects prior-art workflows in which examiners examine results beyond the first page.Source: §5.2 · p. 17

5.3. Backends and Compute Envelope

  • The lexical backend uses BM25 with default parameters, while dense retrieval uses one multilingual encoder with int8 vectors under a compute-constrained setup.The single-encoder design emphasizes retrieval choices such as granularity and field views rather than model selection.Source: §5.3 · pp. 17–18

5.4. Passage Slicing and Aggregation

  • Four passage-score aggregation strategies produce family-level scores: maxP, avg_top3, avgP, and sumP.They respectively select the highest score, average the three highest, average all, or sum all passage scores; avg_top3 uses N=3.Source: §5.4 · p. 18

5.5. Hybridization via Reciprocal Rank Fusion

  • RRF combines passage-level or document-level component rankers, with K selected by grid search on ALL NDCG@100.The chosen K values are 30 for passage-level hybrids and 60 for document-only hybrids.Source: §5.5 · p. 18

5.6. Design Space and Breadth

  • The design space varies query and corpus representations, passage aggregation, and passage length, yielding 249 unique configurations.The breadth-focused evaluation prioritizes systematic design comparisons over model variety.Source: §5.6 · p. 19

5.7. Protocol and Implementation

  • Evaluation runs use rank cutoff 100, log NDCG@100 and Recall@100 for each subset, and track timing and configuration parameters.Efficiency measurements cover scoring and aggregation time, excluding indexing and embedding preprocessing.Source: §5.7 · p. 19

6. Results

Passage-level methods lead on ALL and IN, but dense retrieval’s advantage nearly vanishes on OUT-domain queries. Title+Abstract+Claims ranks first among query representations across all three subsets.

6.1. Backend and Granularity Analysis

  • Dense methods’ IN-domain advantage over BM25 nearly disappears on OUT-domain queries (0.0564 versus 0.0003 NDCG@100); dense-passage retention is 15% from ALL to OUT (0.3839 to 0.0592), versus BM25-passage’s 18% (0.3275 to 0.0589).Source: §6.1 · p. 20
  • Table 15 reports dense-passage methods achieve 0.3381 NDCG@100 on ALL and 0.3839 on IN; across backends, passage configurations outperform document-level approaches by 0.020–0.036 NDCG@100.Source: §6.1 · p. 20

6.2. Query representation effectiveness

  • Title+Abstract+Claims yields the highest NDCG@100 on ALL (0.3381), IN (0.3839), and OUT (0.0592), followed by Title+Abstract.Figure 6 presents the macro-mean comparison across class-best configurations; the authors attribute the ranking to complementary information in claims and abstracts.Source: Figure 6 · p. 21, §6.2 · p. 22

6.3. Passage-length effects

Optimal passage length differs by backend: dense methods peak at shorter windows than BM25. Aggregation also varies by backend and domain subset, with maxP favored on OUT.

6.3. Passage-length effects

  • Dense methods peak around 1024–2048 tokens, while BM25 improves through 4096–8192 tokens before plateauing.Figure 7 compares NDCG@100 by passage length and evaluation subset. The authors relate the difference to context dilution in long dense passages and increased vocabulary coverage for BM25.Source: Figure 7 · p. 22, §6.3 · pp. 22–23

6.4. Aggregation strategies at passage level

  • Dense retrieval selects avg_top3 for IN (0.3839 NDCG@100) and maxP for OUT (0.0592), while BM25 selects maxP across all subsets.Tables 16 and 17 report aggregation best by subset for NDCG@100 and Recall@100. The authors connect dense avg_top3’s cross-subset consistency with robustness when domain classification is uncertain.Source: Table 16 · p. 23, §6.4 · p. 23

6.5. Effectiveness–efficiency frontier

The frontier analysis contrasts faster document-level retrieval with more effective but costlier passage-level methods. RRF improves effectiveness, with document-only fusion delivering particularly large gains without passage indexing overhead.

6.5. Effectiveness–efficiency frontier

  • Document-level methods execute in under 50s with moderate effectiveness, while passage-level methods reach higher effectiveness at 100–400s.Passage retrieval adds slicing, aggregation, and expanded-index costs, while potentially finding relevant content in long documents; reported execution times exclude preprocessing.Source: §6.5 · pp. 23–24, Figure 8 · p. 24

6.6. RRF hybridization (best overall)

  • Passage-capable RRF reaches 0.3475 NDCG@100 on ALL, beating dense-passage by +0.0094; gains are +0.0092 on IN and +0.0036 on OUT.RRF combines lexical and dense rankings; the reported recall gains are +0.0099 on ALL and +0.0112 on IN.Source: §6.6 · p. 24, Figure 9 · p. 25, Table 18 · p. 25
  • RRF improves effectiveness across evaluation subsets by combining complementary lexical and dense signals, with minimal tuning requirements.K=30 is described as balancing sensitivity to top ranks against dilution from higher values.Source: §6.6 · pp. 24–25

6.7. Document-only hybrid fusion

  • Document-only RRF gains +0.0269 NDCG@100 on ALL over the best single method, nearly 3× the passage-level fusion gain of +0.0094.The reported document-only fusion score is 0.3324 versus 0.3055 for dense-document; the figure reports +0.0261 on IN and +0.0061 on OUT.Source: §6.7 · p. 25, Figure 10 · p. 26, Table 20 · p. 26
  • Document-only fusion’s larger gains reflect greater rank diversity between BM25 and dense backends, and avoid passage-indexing overhead.The discussion attributes its increased RRF potential to larger performance disparities between document-level backends; K=60 is reported as the optimal constant.Source: §6.7 · pp. 25–26

6.8. OUT-of-domain challenge

OUT-domain retrieval degrades sharply when query and target families share no IPC3 classifications. Lexical methods retain a larger share of ALL performance than dense methods, while hybrid fusion offers only partial robustness.

6.8. OUT-of-domain challenge

  • BM25-passage retains 20.1% of ALL performance on OUT, versus 17.5% for dense-passage; document methods retain 19.3% and 15.8%, respectively.The table reports relative OUT-versus-ALL performance ratios at @100.Source: §6.8 · p. 27, Table 22 · p. 27
  • Dense methods are described as suffering disproportionate domain-shift degradation, while BM25’s lexical matching is more robust to the shift.The account attributes dense degradation to representations that fail to generalize across technological domains and BM25 resilience to exact term overlap.Source: §6.8 · p. 27
  • Hybrid fusion retains 18.0% of ALL performance for passage-level systems and 17.6% for document-only systems, indicating only partial mitigation.The conclusion recommends maxP aggregation, sparse-dense fusion, and document-only hybrid fusion when passage indexing is unaffordable.Source: §6.8 · p. 27

7. Discussion

The discussion identifies granularity, aggregation, and fusion as practical levers, but emphasizes that OUT-domain retrieval remains a distinct challenge. It also distinguishes deployment regimes by effectiveness and computational cost.

Global findings

  • Passage slicing can suppress long-document noise, while family-level aggregation preserves recall and rewards informative segments.avg_top3 is described as stabilizing dense matching when several high-quality passages exist, typical of IN scenarios; maxP avoids dilution from neutral passages, critical for OUT and BM25.Source: §7 · p. 27

OUT retrieval challenge

  • OUT queries lack shared IPC3 anchors, and near-miss families may be structurally similar without substantive term overlap.The discussion says lexical methods show greater resilience and hybrid fusion reintroduces token-level signals; it recommends maxP and sparse-dense fusion.Source: §7 · p. 28

Document-only RRF efficiency

  • Document-only RRF removes passage-indexing requirements while combining retrieval signals; K is selected by grid search on ALL NDCG@100.The tested K values are 10, 30, 60, and 100; selected values are 30 for passage-capable and 60 for document-only systems.Source: §7 · p. 28

Effectiveness-efficiency trade-off analysis

  • Dense-document methods offer effectiveness with minimal indexing costs, dense-passage methods peak at higher computational expense, and document-only fusion favors constrained deployments.BM25-passage occupies an intermediate position and is particularly competitive for OUT recall scenarios.Source: §7 · p. 28

8. Conclusion

The paper presents DAPFAM as a family-level benchmark with explicit IN/OUT partitions and a compute-aware evaluation of retrieval design choices. Its conclusion reports persistent domain-shift challenges alongside practical retrieval trade-offs, and identifies limits in dataset scope and evaluation settings.

Conclusion

  • DAPFAM evaluates cross-domain retrieval through systematic IN/OUT partitions, while the study varies retrieval granularity, representations, passage length, aggregation, and fusion.The evaluation constrains modeling capacity to a single multilingual encoder with int8 quantization.Source: §8 · p. 28
  • Passage-level retrieval outperforms document-level approaches for both backends; dense-passage leads on ALL and IN, while BM25-passage is a fallback for OUT.RRF gains are modest for passage-capable systems but substantially larger in document-only configurations that avoid passage indexing overhead.Source: §8 · p. 28
  • OUT-domain retrieval remains challenging: the partition amplifies lexical drift and drafting-style variation, while pure semantic approaches may retrieve legally irrelevant topical matches.The conclusion reports sparse lexical signals as showing superior degradation patterns under domain shift.Source: §8 · p. 28
  • DAPFAM’s metadata and family aggregation also support research on patent relationships, temporal dynamics, and cross-jurisdictional patterns.The conclusion names technological landscape analysis, citation pattern studies, and patent family evolution as additional applications.Source: §8 · p. 29
  • DAPFAM covers English-language Lens.org families, uses IPC3 partitions and citation-based labels, and evaluates one multilingual encoder on consumer hardware.The authors note possible jurisdictional and linguistic biases, alternative domain boundaries, incomplete coverage of technical relatedness or legal sufficiency, and limits to model-specific and enterprise-deployment insights.Source: §8 · p. 29
  • The authors identify encoder adaptation, robustness stress tests, and alternative domain-partitioning schemes as future research directions.Proposed stress tests include noisy patent sections and jurisdictional inconsistencies.Source: §8 · p. 29

Appendix — Complete run matrix

The appendix reports the complete run matrix across retrieval backends, granularities, query and corpus representations, and passage aggregators. Results are organized by ALL, IN, and OUT subsets, with NDCG@100 and Recall@100 reported for each configuration.

Appendix — Complete run matrix

  • The run matrix reports NDCG@100 and Recall@100 across ALL, IN, and OUT subsets for each configuration.The configurations span backend, granularity, query view, corpus view, aggregator, and passage length.Source: Appendix — Complete run matrix · p. 32
  • Query and document views use field combinations, and Full Text denotes title, abstract, claims, and description.Query fields include keywords, title, abstract, and claims; document corpus views include Full Text and selected field combinations.Source: Appendix — Complete run matrix · p. 32
  • Passage configurations specify window length and aggregation method, including maximum, mean, top-three mean, and sum.The notation uses pℓ for passage length ℓ and names the aggregators maxP, avgP, avg_topN, and sumP.Source: Appendix — Complete run matrix · p. 32
  • The appendix includes full BM25 document and passage results for ALL, IN, and OUT subsets.The full-result tables are identified for BM25 document configurations and BM25 passage configurations.Source: Appendix — Complete run matrix · pp. 32–33
  • The appendix includes full dense document and passage results for ALL, IN, and OUT subsets.The full-result tables are identified for dense document configurations and dense passage configurations.Source: Appendix — Complete run matrix · pp. 34–35

Related papers

Loading 2506.22141v2…