PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Embedding: Two-Stage Training for Multilingual Patent Prior-Art Retrieval

Wentao Wu, Xu Wang, Linqing Chen, Fenhong Zhu, Hanmeng Zhong, Yuanxiang Huangfu

arXiv:2610.29521v1cs.IR

TL;DR

X-category prior-art retrieval must find disclosures relevant to novelty or inventive step, beyond topical similarity across languages and patent views. X-Embedding first aligns patent content across sections and multilingual family members, then trains for X-category retrieval. On the evaluated collection, it improves aggregate retrieval over direct supervised fine-tuning.

  • Problem

    X-category retrieval must identify examination-relevant disclosures beyond patents that are merely close in topic, despite differences in wording and presentation.

    Source: §1 · p. 1

  • Method

    X-Embedding first aligns cross-section and cross-language patent-family content, then trains retrieval relevance using examination-report X citations.

    Source: §1 · p. 2

  • Results

    89.09% overall macro Hit@100 at 64 dimensions, exceeding direct supervised fine-tuning by 3.19 percentage points under the same second-stage training conditions.

    Source: §7 · p. 11

  • Takeaways & Limitations

    On the evaluated collection, patent-domain adaptation before retrieval training improves aggregate X-category retrieval, with gains varying across languages, views, and cutoffs.

    Source: §7 · p. 11

    Evaluation covers one collection and one run per variant, so differences do not establish statistical significance or an advantage at equal total computation.

    Source: §6.3 · p. 11

Abstract

from arXiv · show

Multilingual patent prior-art retrieval requires identifying X-category references that are particularly relevant to the novelty or inventive step of a claimed invention, beyond topical similarity across languages and document views. We present X-Embedding, a two-stage framework that first learns patent-domain representations and then optimizes X-category retrieval relevance. The first stage aligns complementary sections within patents and corresponding sections across different-language patent-family members, learning from both patent structure and multilingual expressions. The second stage learns X-category relevance using examination-report X citations as positives and negatives drawn from retrieval candidates and A-category citations. Matryoshka representation learning (MRL) enables 64- and 1024-dimensional embeddings within one model. Evaluation on a common Chinese, English, and Japanese collection with four query views yields 89.09% overall macro Hit@100 at 64 dimensions, exceeding Qwen3-Embedding-0.6B and RoBERTa by 21.25 and 3.68 percentage points, respectively. Under the same second-stage training and evaluation conditions, two-stage training outperforms direct supervised fine-tuning by 3.19 points, with gains of 5.34 and 4.72 points for English and Japanese and higher scores in nine of twelve language-view combinations. Overall macro Hit@10 and Hit@20 also improve by 3.71 and 3.79 points, respectively. These results support sequential patent-domain and task adaptation for improving aggregate X-category retrieval on this multilingual collection.

1 Introduction

X-category retrieval targets references especially relevant to novelty or inventive step, rather than patents that are merely similar in topic. X-Embedding tests whether patent-content alignment before X-citation supervision improves retrieval.

1 Introduction

  • X-category retrieval must connect technical content to examination-relevant disclosures despite different wording and presentation, not merely match terminology or technological fields.Similar components can differ in the connections, operating conditions, or step sequences that define an invention.Source: §1 · p. 1
  • The proposed sequence first aligns patent sections and multilingual family members, then learns X-category relevance from examination-report citations.The authors compare this approach with direct supervised fine-tuning under matched second-stage training and evaluation conditions.Source: §1 · p. 2
  • 3.19 percentage points higher overall macro Hit@100 at 64 dimensions than direct supervised fine-tuning, with gains in nine of twelve language–view combinations.Hit@10 and Hit@20 also improve in all three query-language groups.Source: §1 · p. 2

2 Related Work

Prior work spans multiple patent-representation objectives and evaluation settings; X-Embedding distinguishes its sequential supervision design and evaluates against a matched direct-supervision control.

2.1 Patent representations and retrieval relevance

  • Patent models address technological distance and classification, claim–passage matching (Risch et al., 2021), citation supervision (Ghosh et al., 2024), and graph retrieval (Daniell et al., 2025).These tasks use different relevance units; performance on them does not directly characterize complete-document retrieval, for which this study uses examination-report X citations.Source: §2.1 · p. 3

2.2 Citation supervision and multi-view alignment

  • Related approaches use section self-supervision (Zuo et al., 2026), jointly train citation retrieval and multi-view alignment in QaECTER (Djemmal et al., 2026), or evaluate twelve query representations in Sophia-bench.Sophia-bench uses English text, including translated patent-family representatives, and a pool of cited and citing documents in XY and A categories.Source: §2.2 · p. 3

2.3 Patent embedding evaluation and multi-stage training

  • Patent embedding evaluations vary by task, query–document representation, domain, and size; X-Embedding uses X-category labels, Chinese, English, and Japanese queries, and TAC candidates, unlike Sophia-bench.General-purpose multi-stage retrieval training has also been explored by GTE (Li et al., 2023). Scores from different collections and relevance definitions are not directly comparable.Source: §2.3 · p. 3
  • Unlike joint QaECTER training (Djemmal et al., 2026) and R4’s mixed pair sources (Yousefiramandi and Cooney, 2026), X-Embedding stages cross-section and cross-language family alignment before X-citation retrieval training.A direct-supervision control with matched second-stage settings tests the aggregate benefit of adding patent-domain adaptation.Source: §2.3 · p. 3

3 Background

The background defines the retrieval target as recorded X-category references and evaluates separate query views against a common candidate representation.

3.1 X-category patent retrieval

  • An X-category reference is particularly relevant when taken alone and may prejudice novelty or inventive step; for novelty, it must disclose the necessary claim elements together in their required arrangement.Training positives are X-category citations recorded in examination search reports.Source: §3.1 · p. 4

3.2 Multilingual, multi-view retrieval

  • Each abstract, claim, description, or TAC view is a separate query, while candidates use TAC representations and the recorded X-reference set is shared across views.Evaluation is stratified by query language and view; the view determines which source-patent information is available to the encoder.Source: §3.2 · p. 4

4 Method

X-Embedding uses a shared encoder in sequential training: weak patent-view alignment initializes X-category retrieval supervision, with Matryoshka representations supporting multiple dimensions. The method combines within-patent and cross-language family correspondences, examination-report citations, and selected negatives.

4.1 Framework overview

  • X-Embedding uses a shared Qwen3-Embedding-0.6B encoder for sequential weakly supervised patent-view training and task-relevance adaptation, with independently encoded queries and candidates.Retrieval uses normalized representations of matching dimensions; experiments use 64 dimensions.Source: §4.1 · p. 4, Figure 1 · p. 5

4.2 Stage I: weakly supervised patent-view alignment

  • Stage I aligns abstract–claim and claim–description pairs within patents, plus abstract–abstract and claim–claim pairs across different-language family members.It uses approximately 25.3 million weakly supervised pairs and an InfoNCE objective that draws paired texts closer while comparing them with other batch texts.Source: §4.2 · pp. 4–5

4.3 Stage II: supervised prior-art retrieval

  • Stage II ranks X-category references above comparison candidates using query–document supervision and examination-report X citations as positives.Negatives include retrieval candidates and A-category citations; score-stratified sampling and language-prioritized selection produce examples with eight valid negatives, while the objective promotes the X-category target over remaining candidates.Source: §4.3 · pp. 5–6

4.4 Matryoshka representation learning for dimension truncation

  • Matryoshka representation learning trains the 64-dimensional prefix and full 1024-dimensional representation within the same encoder, enabling retrieval at either size without dimension-specific retraining.All experiments evaluate 64-dimensional representations.Source: §4.4 · p. 6

5 Experiments

The experiments compare X-Embedding with baselines under a shared multilingual evaluation setup and examine how results vary by language, query view, and retrieval depth. The complete model leads the reported overall comparisons and has the highest overall macro Hit@K among the Qwen-based models at all six cutoffs.

5.1 Experimental setup

  • The evaluation uses approximately 300,000 TAC candidate patents and Chinese, English, and Japanese queries in ABS, CLAIM, DESC, or TAC views, excluding each query patent and its family from candidates.Effective query counts are stratified by IPC section A–H; the supplied passages give 103 Chinese and 151 English queries per view, and 125–148 Japanese queries depending on view.Source: §5.1 · pp. 6–7, Table 1 · p. 7
  • The direct-supervision control starts from Qwen3-Embedding-0.6B and matches the two-stage model's second-stage data, negative construction, training budget, and validation-based selection.The comparison is designed to assess the effect of preceding supervised retrieval training with Stage I.Source: §5.1 · p. 8

5.2 Retrieval with 64-dimensional representations

  • 21.25 and 3.68 percentage points: the complete model exceeds Qwen3-Embedding-0.6B and RoBERTa, respectively, on overall macro Hit@100.Stage I scores 72.06 overall, 4.22 points above Qwen3-Embedding-0.6B; the complete model is 17.03 points above Stage I.Source: Table 2 · p. 8, §5.2 · p. 8
  • Two-stage training gains vary across language–view pairs: English improves in all four views, while Japanese gains are concentrated in ABS, CLAIM, and DESC and Chinese results are mixed.Relative to RoBERTa, the two-stage model improves Chinese and English means by 5.78 and 7.37 points, while its Japanese mean is 2.09 points lower.Source: §5.2 · p. 8, Table 3 · p. 9, Figure 2 · p. 9

5.3 Retrieval across cutoffs

  • At all six evaluated cutoffs, X-Embedding has the highest overall macro Hit@K among the four Qwen-based models, with gains of 1.90–4.75 percentage points over direct supervised fine-tuning.Overall macro Hit@10 and Hit@20 are 66.62% and 74.67%, gains of 3.71 and 3.79 points, respectively.Source: §5.3 · p. 9, Figure 3 · p. 10
  • Two-stage training improves all three query-language groups at Hit@10 and Hit@20, with the largest gains for Japanese; Chinese improves at these cutoffs despite slightly lower Hit@100.The Japanese gains are 6.15 points at Hit@10 and 5.98 points at Hit@20.Source: §5.3 · pp. 9–10, Table 4 · p. 10

6 Discussion

The controlled comparison supports an aggregate benefit from initializing supervised fine-tuning with patent-domain adaptation, even though Stage I alone changes retrieval unevenly. Gains vary by language, query view, and retrieval depth, and the evaluation has scope and interpretation limits.

6.1 The role of patent-domain adaptation

  • Under the same second-stage conditions, patent-domain adaptation improves aggregate retrieval over direct supervised fine-tuning, although Stage I alone changes retrieval unevenly across languages.The controlled comparison supports the aggregate benefit of the training sequence on the evaluated collection.Source: §6.1 · p. 10

6.2 Language and query-view differences

  • 14.21, 16.06, and 6.55 percentage-point DESC Hit@100 gains over RoBERTa occur for Chinese, English, and Japanese, but architecture, pretraining, and input length confound attribution.The direct-supervision comparison holds encoder architecture and second-stage training conditions fixed, more specifically evaluating patent-domain adaptation.Source: §6.2 · pp. 10–11
  • Japanese ABS has 125 effective queries, versus 148 for each other view, so view-level scores reflect both input representation and query-set composition.Language-level results use different patent query sets and do not by themselves isolate cross-language transfer.Source: §6.2 · p. 11
  • Training gains vary by query language, view, and retrieval depth; Chinese gains at shorter cutoffs do not coincide with Hit@100 improvement, while Japanese gains over direct fine-tuning coexist with a RoBERTa gap.Aggregate gains should therefore be interpreted alongside language- and view-specific results.Source: §6.2 · p. 11

6.3 Limitations

  • The evaluation uses one patent collection, 64-dimensional representations, and one run per variant; it establishes neither statistical significance nor an advantage under equal total computation.Stage I increases total training computation; 1024-dimensional retrieval, other collections, and separate ablations remain untested.Source: §6.3 · p. 11
  • Language-stratified evaluation lacks query–candidate language-direction estimates, and incomplete X-category annotations may leave relevant documents among retrieval-derived negatives.These limits constrain interpretation of cross-language transfer and retrieval beyond annotated reference sets.Source: §6.3 · p. 11

7 Conclusion

X-Embedding combines weakly supervised patent-view alignment with X-category retrieval training. On the evaluated collection, it improves aggregate Hit@100 over direct supervised fine-tuning and exceeds the base encoder and RoBERTa overall, though gains vary across languages, views, and cutoffs.

7 Conclusion

  • X-Embedding combines weakly supervised patent-view alignment with X-category retrieval training.The approach is evaluated on a Chinese, English, and Japanese collection.Source: §7 · p. 11
  • 89.09% overall macro Hit@100 at 64 dimensions exceeds direct supervised fine-tuning by 3.19 percentage points under the same second-stage training conditions.Overall macro Hit@10 and Hit@20 improve by 3.71 and 3.79 points, respectively.Source: §7 · p. 11
  • The model exceeds Qwen3-Embedding-0.6B and RoBERTa in overall macro Hit@100, but Japanese ABS, CLAIM, and TAC remain below RoBERTa.At Hit@100, gains are concentrated in English and Japanese, while Chinese performance decreases slightly.Source: §7 · p. 11

Related papers

Loading 2610.29521v1…