PaperlayerView exact revision on arXiv

Source-linked AI summary

PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding

Iliass Ayaou, Denis Cavallucci

arXiv:2510.22264v1cs.CLcs.AIcs.IR

TL;DR

Patent text embeddings face long, structured documents and asymmetric matching, while existing evaluation frameworks do not reflect real-world patent requirements. The paper introduces PatenTEB and the patembed model family, then reports strong external validation, including state-of-the-art patent clustering performance.

  • Problem

    Patent text embedding must address long, structured technical discourse and asymmetric matching, but existing evaluation frameworks do not reflect real-world patent requirements.

    Source: §1 · p. 1

  • Method

    The paper builds PatenTEB with leakage-resistant splits, domain-aware hard negatives, and asymmetric retrieval, classification, paraphrase, and clustering tasks, and develops patembed models.

    Source: §9 · p. 29

  • Results

    External validation reports state-of-the-art 0.494 V-measure on MTEB BigPatentClustering.v2 for patembed-base and 0.377 NDCG@100 for patembed-large.

    Source: §9 · p. 29

  • Takeaways & Limitations

    PatenTEB provides a 15-task framework for evaluating patent text embeddings across retrieval, classification, paraphrase, and clustering.

    Source: §9 · p. 29

    The authors identify potential bias from following DAPFAM's simple family aggregation methodology [1].

    Source: §8.3 · p. 28

Abstract

from arXiv · show

Patent text embeddings enable prior art search, technology landscaping, and patent analysis, yet existing benchmarks inadequately capture patent-specific challenges. We introduce PatenTEB, a comprehensive benchmark comprising 15 tasks across retrieval, classification, paraphrase, and clustering, with 2.06 million examples. PatenTEB employs domain-stratified splits, domain specific hard negative mining, and systematic coverage of asymmetric fragment-to-document matching scenarios absent from general embedding benchmarks. We develop the patembed model family through multi-task training, spanning 67M to 344M parameters with context lengths up to 4096 tokens. External validation shows strong generalization: patembed-base achieves state-of-the-art on MTEB BigPatentClustering.v2 (0.494 V-measure vs. 0.445 previous best), while patembed-large achieves 0.377 NDCG@100 on DAPFAM. Systematic ablations reveal that multi-task training improves external generalization despite minor benchmark costs, and that domain-pretrained initialization provides consistent advantages across task families. All resources will be made available at https://github.com/iliass-y/patenteb. Keywords: patent retrieval, sentence embeddings, multi-task learning, asymmetric retrieval, benchmark evaluation, contrastive learning.

1 Introduction

Patent text embedding faces long, structured technical documents, asymmetric matching, and cross-domain semantic challenges, while existing evaluation resources do not systematically reflect real-world patent requirements. The paper asks how to build a comprehensive evaluation framework and identify models and training strategies that perform well across tasks and generalize.

1 Introduction

  • Patent embeddings must handle long, structured technical discourse, asymmetric query-target matching, and semantic variation across technological domains.Patents often exceed 10,000 words and use specialized rhetorical patterns.Source: §1 · p. 1
  • Existing evaluation frameworks do not adequately reflect real-world patent text embedding requirements or systematically cover diverse downstream tasks.General-purpose benchmarks lack patent-specific evaluation, while existing patent resources address only narrower applications.Source: §1 · p. 1
  • The paper asks how to evaluate the full spectrum of patent text understanding and optimize models for benchmark performance and real-world generalization.The question includes preventing leakage, capturing realistic matching, maintaining performance across task families, and understanding benchmark-generalization trade-offs.Source: §1 · p. 1

external validation.

The work combines a domain-stratified, leakage-resistant 15-task benchmark with the patembed model family and systematic training ablations. External results include a state-of-the-art clustering score, while the experiments report cross-domain retrieval degradation and trade-offs associated with training choices.

external validation

  • PatenTEB covers 15 tasks across retrieval, classification, paraphrase, and clustering, using domain-stratified splits and leakage prevention.The patembed family is trained through multi-task learning on 13 tasks, with ablations of supervision, scale, and architecture.Source: external validation · pp. 1–2
  • Multi-task training with retrieval, classification, and paraphrase supervision marginally reduces benchmark scores but improves external generalization.The authors suggest benchmarks can inadvertently reward overfitting.Source: external validation · p. 2
  • Domain-pretrained initialization provides consistent advantages across task families, with the largest gains in semantic matching tasks.Source: external validation · p. 2
  • 3–6× degradation in cross-domain retrieval and paraphrase performance indicates a persistent challenge when matching patents across disjoint technological domains.The passage identifies vocabulary mismatch as indicating this challenge.Source: external validation · p. 2
  • 0.494 V-measure on MTEB BigPatentClustering.v2 versus the previous best of 0.445 establishes patembed-base's state-of-the-art external clustering result.The patembed family spans 67M–344M parameters.Source: external validation · p. 2

2 Related Work

This section situates PatenTEB and patembed among general embedding methods, patent encoders, existing datasets, and domain-pretraining work, highlighting gaps in patent-specific multi-task evaluation and fragment-based retrieval. It also frames patembed as a model family designed to share representations across task families, adapt to downstream objectives with prompts, and support systematic benchmark evaluation.

2.1 Foundational Sentence Embedding Methods

  • Sentence embedding methods established siamese contrastive embeddings and multi-task training, but MTEB and BEIR lack patent-specific structural, matching, and evaluation challenges.Sentence-BERT [30] introduced a siamese architecture for similarity computation, and the Universal Sentence Encoder [5] demonstrated multi-task training.Source: §2.1 · pp. 2–3

2.2 Patent-Specific Encoders

  • PatentSBERTa is limited to classification, while PAECTER’s title-abstract document similarity does not cover full-text or asymmetric retrieval; patembed multi-task fine-tunes across four task families.The patembed family includes 67M–344M parameter models and prompt-based task conditioning.Source: §2.2 · p. 3

2.3 Patent Datasets and Benchmarks

  • PatenTEB spans 15 tasks across retrieval, classification, paraphrase, and clustering, with leakage prevention, stratification across 109 IPC3 classes, and 2.06 million data points.Existing patent datasets offer single-task depth or lack embedding-specific evaluation, while general embedding benchmarks lack patent coverage.Source: §2.3 · p. 4

2.4 Domain-Specific Pretraining for Technical Text

  • Work on legal and scientific text shows benefits from specialized pretraining for domain vocabulary and citation structure; patembed-large initializes from BERT-for-Patents.Relevant examples include Legal-BERT, CaseLaw-BERT, SciBERT [3], and BioBERT [21].Source: §2.4 · p. 4

2.5 Positioning PatenTEB and patembed

  • PatenTEB includes five asymmetric tasks for fragment-based patent queries, including title→full and problem→solution, with deterministic fragment removal to prevent trivial lexical matching.Existing benchmarks focus on symmetric document-document matching despite patent searches often using fragments such as titles, problem statements, or desired effects.Source: §2.5 · p. 5
  • patembed trains jointly on 13 tasks—8 retrieval, 3 classification, and 2 paraphrase—learning shared representations across task families.This contrasts with prior work focused on individual task types or pretraining alone.Source: §2.5 · p. 5
  • Task-specific prompts guide representation learning, allowing one encoder to adapt to different downstream objectives without task-specific fine-tuning.The paper presents prompts as a way to serve multiple downstream applications with a single encoder.Source: §2.5 · p. 5
  • The model family spans 67M–344M parameters, using knowledge distillation and long-context training to support deployment across resource constraints.Existing work is described as offering single model sizes.Source: §2.5 · p. 5
  • Evaluation covers 15 PatenTEB tasks plus MTEB BigPatentClustering.v2 and DAPFAM, assessing generalization beyond training tasks.Prior work is characterized as evaluating on narrower task sets.Source: §2.5 · p. 5
  • Patent workflows require retrieval, classification, and paraphrase, while patent vocabulary differs substantially from other text sources [14].The section motivates multi-task training as more practical than separate models for each workflow and domain pretraining for technical terminology and formal legal language.Source: §2.5 · p. 5

3 Benchmark Construction

PatenTEB’s construction uses patent-family data, IPC3 domain stratification, and task-specific quality controls to cover patent search and analysis workflows across 15 tasks in four task families. It also uses stratified sampling to balance training across objectives and provides standardized data exports.

3 Benchmark Construction

  • Using DAPFAM [1], families are atomic analysis units, each assigned a dominant IPC3 code; requiring ≥100 families per class retains 109 technology domains, followed by an 80%/10%/10% IPC3-stratified split preventing family leakage.Source: §3 · p. 5

3.1 Symmetric Retrieval

  • Three symmetric retrieval objectives—IN-domain, OUT-domain, and MIXED-domain—test matching within, across, and between partially overlapping IPC3 areas.Hard negatives exclude citation-connected candidates and are selected by domain relationship to discourage reliance on superficial domain cues.Source: §3.1 · p. 7

3.2 Asymmetric Retrieval

  • Five asymmetric tasks retrieve full patent documents from partial queries, including titles, problems, effects, solutions, and substances.To prevent trivial lexical matching, targets are reconstructed with query fragments removed; effect and problem segments are less prevalent, so those tasks have fewer examples.Source: §3.2 · pp. 7–8

3.3 Classification

  • Classification covers 109-domain Text→IPC3 prediction, citation-trajectory Bloom classes, and binary NLI-oldnew citation-direction prediction.Bloom labels distinguish early, late, and normal citation trajectories within each IPC3 domain.Source: §3.3 · p. 8

3.4 Paraphrase Detection

  • Problem and solution paraphrase tasks use same-segment IN-domain citation pairs as positives and cross-domain pairs as negatives, with positives set to 14% per split.Each task contains 150,000 training pairs and about 18,000 validation and test pairs.Source: §3.4 · p. 8

3.5 Clustering

  • Two evaluation-only clustering tasks group families by complete IPC-code sets or inventor identifiers, testing technological similarity and inventor-identity signals.IPC clusters retain sizes 200–1000; inventor clusters retain sizes 100–1000.Source: §3.5 · pp. 8–9
  • 2,057,286 data points across 15 tasks comprise 1,556,751 training, 181,215 validation, and 319,320 test examples.The benchmark covers four task families; Table 1 provides train/validation/test sizes for all tasks.Source: §3 · p. 7, §3.5 · p. 9, Table 1 · p. 9

3.6 Stratified Sampling and Export

  • Conservative stratified subsampling is applied when certain tasks would otherwise overwhelm multi-task training.Source: §3.6 · p. 10
  • The sampling mix preserves minimum representation per domain and label to balance training across objectives while respecting computational constraints.Source: §3.6 · p. 10
  • All data exports use Apache Parquet format with standardized splits.Source: §3.6 · p. 10

4 Models and Training

The patembed family specializes patent text encoders through multi-task learning and distillation, with variants designed for different deployment and context-length needs. Training combines task-specific losses under a uniformly weighted objective.

4.1 Model Variants

  • The six core variants include 344M-parameter patembed-large and distilled 12-layer patembed-base, plus smaller compressed models.patembed-large produces 1024-dimensional embeddings; patembed-base produces 768-dimensional embeddings.Source: §4.1 · p. 10

4.2 Long-Context Variants

  • Extended-context variants provide windows of 1024, 2048, and 4096 tokens, initialized from gte-modernbert-base.These variants are intended for processing larger patent text segments.Source: §4.2 · p. 10

4.3 Multi-Task Training Framework

  • Multi-task training combines retrieval and paraphrase contrastive objectives with classification losses to support diverse patent tasks.The framework uses shared representations across tasks.Source: §4.3 · p. 11

4.3.1 Loss Functions.

  • Retrieval uses multiple-negatives ranking loss, paraphrase uses online contrastive loss, single-text classification uses batch-hard triplet loss, and pair classification uses cross-entropy.Retrieval objectives cover symmetric and fragment-to-full asymmetric tasks.Source: §4.3.1 · p. 11

4.3.2 Training Objective.

  • The global objective uniformly sums losses over 13 training tasks, while the two clustering tasks are evaluation-only.Exhaustive task-weight tuning was computationally expensive, motivating future exploration of automated weight adaptation.Source: §4.3.2 · p. 12

4.4 Distillation Pipeline

Smaller patembed variants are distilled from patembed-large using teacher embeddings, layer subsampling, and learned projection. Task-specific prompts and a one-epoch training setup complete the pipeline.

4.4 Distillation Pipeline

  • Student variants subsample patembed-large’s layers and learn projections to smaller embedding dimensions by minimizing mean squared error to projected teacher embeddings.The pipeline caches embeddings for 2,318,732 unique training texts and uses incremental PCA to initialize projections.Source: §4.4 · p. 12

4.5 Prompt Design and Input Processing

  • Distinct query and target prompt prefixes provide task-specific representations, particularly for tasks with contradictory objectives.Examples include IN versus OUT retrieval and same-problem versus same-solution paraphrase.Source: §4.5 · p. 12

4.6 Optimization and Hyperparameters

  • Training runs for one epoch with gradient accumulation yielding an effective batch size of 128, constrained by GPU memory.The selected recipe uses paged 8-bit AdamW with cosine learning-rate decay, 10% warmup, and gradient clipping at 1.0.Source: §4.6 · pp. 12–13
  • Learning rates vary by initialization: 1 × 10−5 for BERT-for-Patents, 5 × 10−5 for gte-modernbert-base, and 1 × 10−4 for distilled models.Table 4 reports one-epoch training times for selected variants on a single 48GB NVIDIA A40 GPU.Source: §4.6 · p. 13, Table 3 · p. 13, Table 4 · p. 14

5 Evaluation Protocol

The protocol gives equal weight to each task’s primary metric in an Overall Score, then evaluates models under prompt conditions matching their training regime. It specifies task metrics, implementation choices, and reproducibility controls.

5 Evaluation Protocol

  • Overall Score is the unweighted mean of each model’s primary metric across all 15 tasks, treating tasks equally.The primary metric varies by task.Source: §5 · p. 14

5.1 Metric Definitions

  • Retrieval uses NDCG@10, paraphrase uses Pearson correlation, classification uses Macro-F1, and clustering uses V-measure.Source: §5 · p. 14

5.2 Prompt Handling

  • Patembed models are evaluated with their task-specific prompts, while baselines not designed for instruction-following are evaluated without prompts.External validation also tests every model with and without prompts.Source: §5.2 · p. 15

5.3 Evaluation Implementation Details

  • Retrieval ranks embeddings by cosine similarity, classification fits logistic regression on a stratified 20% training subset, and clustering uses MiniBatchKMeans.Paraphrase evaluation computes Pearson’s r between predicted cosine similarities.Source: §5.3 · p. 15

5.4 Evaluation Protocol and Reproducibility

  • Inference caps sequences at 8192 tokens because of memory constraints, after preprocessing patent texts to combine title, abstract, and first claim.Source: §5.4 · p. 15
  • Evaluations use fixed seeds and single deterministic runs on fixed test sets, with robustness assessed on independent benchmarks and through ablations.Source: §5.4 · p. 15, §5.3 · p. 15

6 Results

Patembed models lead the PatenTEB results, with fine-tuning and patent-domain initialization associated with strong performance across tasks. External evaluations show competitive clustering and retrieval results, while cross-domain retrieval and deployment efficiency remain important considerations.

6.1 Overall Performance on PatenTEB

  • 0.654 Overall Score makes patembed-large the top model across PatenTEB’s 15 tasks; patembed-base reaches 0.645 with lower computational cost.Source: Table 5 · p. 16, §6.1 · p. 16
  • With identical multi-task training, patembed-large initialized from BERT-for-Patents scores 0.654 versus 0.628 for the gte-modernbert-base-initialized variant.Fine-tuning gains are largest on retrieval and paraphrase.Source: §6.1 · pp. 16–17
  • Patembed-large performs best on 12 of 15 tasks, with the largest full-training gains in retrieval and paraphrase; classification differences are smaller.Clustering scores vary only slightly within the patembed family, at 0.584–0.625 V-measure.Source: §6.1 · pp. 16–17, Figure 2 · p. 17

6.2 External Validation

  • 0.494 V-measure on BigPatentClustering.v2 surpasses the previous best of 0.445, while patembed-large achieves the best DAPFAM score of 0.377 NDCG@100.The BigPatent result is reported for patembed-base; the DAPFAM result is for patembed-large.Source: §6.2 · p. 19
  • Retrieval difficulty rises from shared to disjoint IPC3 domains: patembed-large scores 0.512 (IN), 0.443 (MIXED), and 0.172 (OUT).The passages describe out-of-domain retrieval as challenging even for specialized models.Source: §6.2 · p. 19

6.3 Efficiency Analysis

  • Patembed-base_small achieves 97.7% of patembed-large’s Overall Score at roughly 40% of its inference time.The patembed family is reported on the accuracy–speed Pareto frontier.Source: §6.3 · pp. 19–20

7 Ablations and Robustness

The ablations weigh in-domain benchmark scores against external generalization and examine data scale, deployment choices, and input robustness. Full multi-task training favors external results, while the other studies identify where reduced data, dimensions, or input content retain performance and where layer pruning does not.

7.1 Supervision Ablations

  • 0.458 V-measure for full multi-task training on BigPatent clustering exceeds the no-classification variant’s 0.396 and retrieval-only variant’s 0.386, despite slightly higher in-domain scores for those variants.The authors select the full 13-task model as flagship because it generalizes better externally.Source: Table 6 · p. 21, §7.1 · p. 22
  • Removing prompts lowers Overall Score from 0.654 to 0.636 and retrieval score from 0.646 to 0.615; asymmetric retrieval benefits most, while paraphrase and classification are negatively affected.Source: §7.1 · p. 22

7.2 Distillation Data Scaling

  • 0.546 at sp10k rises to 0.589 at sp150k, with diminishing returns beyond roughly half the data; sp70k reaches 98.3% of full performance.sp150k represents the full training set; the caption reports diminishing returns beyond sp80k.Source: Table 7 · p. 24, §7.2 · p. 24
  • Retrieval continues improving through full data, from 0.507 to 0.570, while paraphrase rises from 0.819 to 0.852 and classification from 0.430 to 0.442.Source: §7.2 · p. 24

7.3 Deployment Optimizations

  • D256 embeddings retain 96.6% of full performance with a 4× storage reduction; D128 retains 93.4% with an 8× reduction.Embedding truncation is applied at inference without retraining.Source: §7.3 · p. 24, Table 8 · p. 25
  • Keeping 16 of 24 layers yields only 0.283 Overall Score, while keeping 23 yields 0.644, or 98.4% of full performance.The results identify layers 17–20 as critical to performance.Source: §7.3 · p. 25, Table 9 · p. 26

7.4 Structural Robustness

  • Title-only input retains 88.5% of full performance; removing separators as well yields 88.4%, while the text stresses that an abstract or claims section remains important.Source: §7.4 · pp. 26–27
  • Removing separators retains 99.8% of full performance, and removing either abstract or claims alone retains approximately 98.8%.The remaining substantive section provides sufficient semantic signal.Source: Table 10 · p. 26, §7.4 · p. 26

8 Discussion

The discussion finds that full multi-task training trades a small benchmark-score reduction for better external generalization, while patent retrieval remains substantially harder across disjoint technology domains. It also highlights dataset and labeling biases and recommends validating models beyond a single benchmark.

8.1 Key Findings

  • −0.004 Overall Score (0.654 versus 0.658) accompanies full 13-task training, which nevertheless improves external generalization (+0.062 V-measure on BigPatent).The full variant outperforms externally despite its slightly lower benchmark score.Source: §8.1 · pp. 27–28

8.2 Cross-Domain Performance Gap

  • 0.512 NDCG@10 on retrieval_IN versus 0.172 on retrieval_OUT (2.98× gap), and 0.428 versus 0.069 NDCG@100 on DAPFAM (6.20× gap), show the cross-domain challenge.Different IPC3 domains use specialized vocabularies with little lexical overlap, and embedding-based matching struggles with abstract problem-solution patterns without explicit knowledge incorporation.Source: §8.2 · p. 28
  • 0.494 versus 0.458 V-measure on BigPatent favors patembed-base (193M) over patembed-large (344M), but the comparison is from one external task without significance testing.The authors note that patembed-large outperforms patembed-base on DAPFAM, so the size effect is not systematic.Source: §8.2 · p. 28

8.3 Limitations and Potential Biases

  • Family aggregation treats simple-family members as one invention despite possible Lens.org linking errors; IPC assignments can also diverge across examiners and offices and change over time.These inconsistencies may affect domain-stratified splits and IPC classification tasks.Source: §8.3 · p. 28
  • Patent citations are imperfect relevance labels: strategic, examiner-added, and self-citations, plus incomplete prior-art discovery, introduce noise.Citation practices also vary across technology domains and time periods.Source: §8.3 · p. 28
  • Patents from 1980 onward are filtered by citation counts, creating recency bias; the bloom classification task is particularly susceptible to temporal effects.Recent patents have had less time to accumulate citations.Source: §8.3 · p. 29

8.4 Practical Implications

  • Patent-trained patembed-base (193M) outperforms general Qwen3-Embedding-0.6B (600M) despite being 3× smaller, while the persistent cross-domain gap points to domain-aware system design.The discussion suggests that hybrid systems could use separate encoders per domain or a general encoder to help bridge domains.Source: §8.4 · p. 29
  • +0.018 Overall Score from prompting is a modest but non-negligible gain, and prompts can condition one model for different behaviors without separate models per use case.The discussion describes prompting as a lightweight way to disambiguate tasks and simplify deployment pipelines.Source: §8.4 · p. 29

9 Conclusion

The conclusion presents PatenTEB and the patembed family as a broad evaluation and modeling effort for patent text embeddings. External results and ablations show that benchmark performance, external generalization, domain-pretrained initialization, and prompt-based fine-tuning do not always move together.

9 Conclusion

  • PatenTEB covers retrieval, classification, paraphrase, and clustering with leakage-resistant splits, domain-aware hard negatives, and asymmetric retrieval; patembed spans 67M to 344M parameters.The models are trained through multi-task learning.Source: §9 · p. 29
  • 0.494 V-measure on MTEB BigPatentClustering.v2 is state of the art, while patembed-large attains 0.377 NDCG@100 on DAPFAM.The patembed family achieves an Overall Score of 0.654 through multi-task learning.Source: §9 · p. 29
  • Full multi-task training improves external generalization despite a marginal benchmark cost, indicating that benchmark optimization and deployment performance can diverge.The conclusion also reports consistent gains across tasks from domain-pretrained initialization and a modest overall benefit from prompt-based fine-tuning.Source: §9 · p. 29
  • A moderate-size model outperformed a larger one on an external task, an observed capacity-generalization inversion that the conclusion says underscores external validation in model selection.The conclusion frames this as a reason to validate beyond benchmark performance.Source: §9 · p. 30
  • The authors identify hybrid or knowledge-informed approaches to cross-domain matching, multilingual patents, and larger models under domain constraints as future directions.They envision PatenTEB supporting more robust and generalizable experimentation for patent text embeddings.Source: §9 · p. 30

A Supplementary material

The supplementary material provides task, model, prompt, and evaluation specifications alongside full benchmark and external-validation results. Its tables document the scope and settings needed to interpret the reported experiments.

Supplementary material

  • Table 11 lists task-specific prompt prefixes for training and evaluation, with distinct query and document fields in the format “[prefix] [text content]”.Table 12 specifies each task’s objective, family, metric, and training loss; clustering tasks are evaluation-only and have no training loss.Source: Table 11 · p. 32, Table 12 · p. 33
  • Table 13 details patembed variants’ transformer layers, hidden dimensions, context lengths, embedding dimensions, and parameter counts.Table 14 specifies evaluation hyperparameters, random seeds, and key technical configurations.Source: Table 13 · p. 33, Table 14 · p. 34
  • Table 15 reports per-task results across all 15 tasks, using NDCG@10 for retrieval, Pearson for paraphrase, Macro-F1 for classification, and V-measure for clustering.Bold indicates the best result per task.Source: Table 15 · p. 34
  • Table 16 gives external-validation results for MTEB BigPatent (V-measure) and DAPFAM (NDCG@100), including prompted and unprompted configurations and sensitivity deltas.Bold indicates the best result per benchmark.Source: Table 16 · p. 35

Related papers

Loading 2510.22264v1…