PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonalQuery-Benchmark:Evaluating Retrieval Robustness to Personalized User Expression Differences

Wenyu Liang, Hongzhi Yin, Tongtong Wu, Fan Liu, Kexin Dai, Weiqing Wang

arXiv:2610.32164v1cs.IR

TL;DR

Existing personalized retrieval emphasizes user preferences and intent, while benchmarks offer limited support for evaluating differences in how users express the same need. PQB models personalized syntax and writing errors from user histories and evaluates retrieval systems under these variations. It finds ranking variation despite fixed information needs, with different sensitivities across models and no simple correspondence between effectiveness and robustness.

  • Problem

    Existing personalized retrieval mainly models user preferences and intent, leaving less attention to expression differences between users with the same information need.

    Source: §1 · p. 1

  • Method

    PQB models personalized syntactic expression and writing errors from user histories and evaluates retrieval systems under these variations.

    Source: to personalized expression differences? · p. 3, §5 · p. 23

  • Results

    Even with the information need held fixed, personalized expression differences produce ranking variation, model sensitivities differ by variation type, and overall effectiveness does not map simply to robustness.

    Source: to personalized expression differences? · p. 2, §4.7 · p. 22

  • Takeaways & Limitations

    PQB provides a benchmark and evaluation framework for systematically studying personalized expression robustness.

    Source: to personalized expression differences? · p. 2, §5 · p. 23

    Sample retention depends only on historical expression evidence observable before encoder training, not learned representations, Gaussian quality, or retrieval outcomes.

    Source: §3.2.2 · p. 7

Abstract

from arXiv · show

Existing personalized information retrieval methods mainly model differences in user needs through user preferences and query intent, while paying little attention to the personalized expression differences that arise when different users express the same retrieval need. Existing real query data typically lack stable user identities, long-term expression histories, and intent-matched counterparts, making it difficult to systematically evaluate the impact of personalized expression differences on retrieval outcomes. To this end, we propose PersonalQuery-Benchmark (PQB). PQB models personalized syntactic expression and personalized writing errors from users' historical texts, constructs personalized syntactic expression queries and personalized writing-error queries, and validates benchmark quality through LLM assessment and independent human evaluation. PQB covers three Amazon product domains with 6,516 query instances in total, and is evaluated on five retrieval models representing lexical, learned sparse, dense bi-encoder, late-interaction, and hybrid architectures, together with two LLM-based ranking models. Results show that even when the information need is held fixed, personalized expression differences still induce measurable ranking variation; different retrieval models exhibit clearly different sensitivities to syntactic variation and writing-error variation, while higher overall retrieval effectiveness does not necessarily correspond to stronger robustness against expression variation. PQB provides a new benchmark and evaluation framework for systematically studying personalized expression robustness. The associated code and dataset are available at the following address: https://github.com/cpxjazz123/PersonalQuery-Benchmark.

1 Introduction

Personalized retrieval research emphasizes what users want, but users can express the same need differently, potentially affecting retrieval. The paper asks how robust retrieval systems are to these expression differences, which existing benchmarks do not directly support.

1 Introduction

  • Existing personalized retrieval mainly models user preferences, behavior, and intent, while giving less attention to how users express the same need.Expression differences can affect lexical matching, query representation, and downstream ranking.Source: §1 · p. 1
  • Personalized expression differences are linguistic differences between users that do not depend on changes in underlying intent.Source: §1 · p. 1
  • Existing personalized retrieval datasets emphasize preferences, interactions, or intent, while real-query datasets typically lack stable user identities and sufficient historical expression evidence.Source: §1 · pp. 1–2
  • Fig. 1 illustrates syntactic-expression and writing-error variations under matched product intent and core product attributes.Source: Figure 1 · p. 2
  • The paper asks how robust retrieval systems are when users express the same retrieval need differently.Source: §1 · p. 2

to personalized expression differences?

The paper introduces PQB to evaluate retrieval robustness to personalized expression differences, modeling syntactic expression and writing errors from user histories while holding product need fixed. It validates the benchmark and finds that expression variation affects retrieval outcomes across diverse models.

to personalized expression differences?

  • PQB defines personalized expression through two complementary dimensions: users’ long-term writing-error habits and their syntactic organization of the same semantic content.The writing-error dimension includes stable differences in error frequency and preferred mechanisms; syntax varies in phrase organization, modifiers, and clause construction.Source: to personalized expression differences? · p. 2
  • PQB builds from Amazon review histories and validates syntactic attribution, writing-error consistency, query quality, and evaluation agreement through LLM and human evaluation.Source: to personalized expression differences? · pp. 2–3
  • Across three Amazon domains, personalized expression differences produce ranking variation, with model sensitivities differing by variation type and overall effectiveness not mapping simply to robustness.Source: to personalized expression differences? · p. 2
  • Evaluations span five retrieval architectures and two LLM-based ranking models.Source: to personalized expression differences? · pp. 2–3
  • The benchmark covers three Amazon product domains and 6,516 query instances.Source: to personalized expression differences? · p. 3
  • For syntax, PQB learns user-specific patterns from historical expressions under fixed product context and fits a low-rank-plus-diagonal Gaussian to capture characteristic positions and within-user variation.Source: to personalized expression differences? · p. 3
  • For writing errors, PQB aligns historical-review error annotations, retains primarily surface-form changes, and derives each user’s error frequency and mechanism preferences.Source: to personalized expression differences? · p. 3
  • Personalized syntactic modeling improves user attribution over tested alternatives, and personalized error sampling outperforms global, uniform, character-random, and wrong-user-profile baselines on attribution.Source: to personalized expression differences? · p. 3

2 Related Work

Prior work studies query variation, writing errors, personalization, and controlled generation, but typically treats them separately. PQB combines user-history-based expression modeling with controlled comparisons that hold product and retrieval intent fixed.

2 Related Work

  • Prior work separately studies query formulation, writing errors, user histories, and controlled generation; PQB unifies them to examine personalized expression as an independent retrieval-ranking variable.Source: §2 · p. 3, §2.3 · p. 5

2.1 Query Variation and Controlled Query Generation

  • Query-variation research links structural differences to retrieval behavior but rarely connects query structure to individual users’ long-term expression patterns.Source: §2.1 · p. 4
  • Existing controlled query generation uses prompts, generic rewrites, or predefined attributes, whereas PQB derives personalization signals and selection constraints from historical user texts.Source: §2.1 · p. 4

2.2 Writing Errors and Retrieval Performance

  • Prior writing-error robustness studies generally use population- or task-level perturbations; PQB uses user-specific profiles to form paired queries while keeping target product and intent fixed.Source: §2.2 · p. 4

2.3 Personalized Retrieval and User Modeling

  • Unlike personalized retrieval that adjusts relevance using preferences or behavior, PQB treats expression history as a variable and compares cross-user syntax and same-user error variants.Source: §2.3 · pp. 4–5

3 Benchmark Construction

This section describes how PQB constructs personalized syntactic and writing-error queries using users’ historical expression, product semantics, and error and syntactic-context patterns. Writing-error variants are limited and rejected when their semantic similarity to the source query is below 0.9.

3.1 Dataset Construction Framework

  • PQB links Amazon Reviews 2023 product metadata and ASIN retrieval targets with associated users’ review histories across Musical Instruments, Video Games, and Baby Products.Product attributes constrain query content, while historical reviews provide evidence for modeling users’ syntactic patterns and writing-error habits.Source: §3.1 · p. 5, §3.1.1 · p. 6

3.2 Expression-Difference Quantification

  • PQB represents historical syntax with 21,737-dimensional structural-rule vectors, models user-specific syntactic distributions, and retains profiles that pass numerical, held-out calibration, and stability checks.Same-product supervised contrastive learning distinguishes users’ syntactic patterns; a Gaussian captures both typical style and within-user variation, while held-out calibration is the primary filtering stage.Source: §3.2.1 · p. 7, §3.2.3 · p. 8, §3.2.5 · p. 8

3.3 Personalized Syntactic Expression Query Generation

  • PQB selects personalized syntactic queries from shared product-specific candidate pools, preserving five core terms and requiring pairwise semantic similarity of at least 0.9.It selects user-pair queries that fit each user’s Gaussian syntactic profile, then favors feasible pairs with the largest Bhattacharyya distance between their profiles.Source: §3.3.1 · p. 10, §3.3.2 · pp. 11–12

3.4 Personalized Writing-Error Generation

  • PQB constructs error profiles by using GECToR-corrected historical reviews and ERRANT alignments to record error types, transformations, syntactic contexts, and transformation frequencies.These profiles inform personalized writing-error query generation.Source: §3.4.1 · pp. 12–13
  • PQB estimates user-specific, context-conditioned error-mechanism distributions while retaining users’ overall mechanism preferences and observed character transformations.The profile records historical edits and relates error mechanisms to syntactic contexts, addressing individual differences that global or context-independent distributions would miss.Source: §3.4.1 · pp. 12–13
  • PQB perturbs personalized syntactic queries using context-matched historical character transformations, backing off across syntactic and token contexts before sampling from the user’s overall error preferences.It modifies at most three token positions and rejects variants with semantic similarity below 0.9, measured using all-mpnet-base-v2.Source: §3.4 · p. 12, §3.4.2 · p. 13

4 Retrieval Benchmark Evaluation

This section evaluates how personalized syntax and writing errors affect retrieval for paired queries with fixed targets, alongside PQB’s benchmark quality; results show that both alter retrieval outcomes across systems, while validation supports the quality of the queries and user-specific modeling.

4.1 Experimental Setup

  • The evaluation compares five first-stage retrieval models and two LLM-based rerankers, measuring hit-status and reciprocal-rank differences for paired queries with the same target.Flip@K captures hit-status disagreement among pairs where at least one query retrieves the target; Mean RRstd captures rank variation, while Deg@K and MRR Degradation assess writing-error effects.Source: §4.1.1 · p. 14, §4.1.2 · pp. 14–15

4.2 Personalized Syntactic Expression Differences and Retrieval Robustness

  • All seven retrieval configurations show nonzero Flip@K and Mean RRstd under syntactic variation, with disagreement patterns differing by architecture and retrieval depth.The paired queries retain the same target product and five core terms; their differences lie mainly in function words, modification relations, and syntactic organization. Flip@K is non-monotonic across cutoffs, so a single depth does not capture robustness.Source: §4.2.2 · pp. 16–17, Table 2 · p. 16

4.3 Personalized Writing Errors and Retrieval Robustness

  • All seven configurations show positive Deg@K and MRR Degradation under personalized writing errors, with losses spanning hit rates and target-product rankings.Average Deg@K generally increases with retrieval depth; BM25 has relatively pronounced hit-rate degradation, while BGE-M3 Hybrid has the lowest cross-domain average degradation. LLM-based rankers also degrade, including the ranking-specialized Qwen3-Reranker-8B.Source: §4.3.2 · pp. 18–19, Table 3 · p. 18

4.4 Benchmark Quality Validation

  • Human and LLM evaluations place syntactic attribution above random choice and find personalized writing-error patterns consistent with users’ historical habits.The study also reports that PQB maintains high language quality while preserving personalized expression; its quality assessment includes query ratings and human–human and LLM–human agreement.Source: §4.4.2 · p. 20, Table 4 · p. 20, Table 5 · p. 20
  • LLM and human evaluations find high naturalness and syntactic quality, with good Human–Human and LLM–Human agreement across evaluation dimensions.These results support the reliability of PQB’s query-quality evaluations and indicate that personalized expression modeling does not markedly reduce query language quality.Source: §4.4.2 · p. 20

4.5 User-Specific Syntactic Modeling Comparison

  • The rank-one-plus-diagonal Gaussian achieves the best results across all three user-attribution measures, outperforming alternative syntactic representations in identifying users from held-out expressions.The reported trends also show comparable user-attribution results for a single embedding and shared isotropic Gaussian, with discrimination decreasing across the user-diagonal, vMF, rank-two, and k-nearest-neighbor alternatives.Source: §4.5 · p. 21, Table 6 · p. 21

4.6 Personalized Writing-Error Profile Evaluation

  • Personalized writing-error sampling performs best across all three attribution metrics, outperforming global, uniform, character-random, and wrong-user-profile baselines.Its advantage over global and wrong-user-profile sampling is specifically reported for Top-1 accuracy and MRR.Source: §4.6 · p. 22, Table 7 · p. 22

4.7 Summary of Overall Experimental Results

  • Across RQ1–RQ2, syntactic differences changed retrieval results and personalized writing errors degraded hit rates and ranking across depths, with model robustness varying by variation type.RQ3–RQ5 further found that generated queries retained user-specific characteristics and linguistic quality, the rank-one-plus-diagonal Gaussian led the three syntactic-attribution metrics, and personalized error sampling led the three sampling metrics.Source: §4.7 · p. 22

5 Conclusion

PQB evaluates personalized syntactic expression and writing-error queries under a fixed target product. Both forms of expression variation are associated with retrieval changes, while the paper identifies broader coverage and richer histories as future work.

5 Conclusion

  • Across evaluated models, personalized syntactic expressions change target-product hit status and ranking, while personalized writing errors degrade retrieval performance.The reported phenomena appear across multiple retrieval models.Source: §5 · p. 23
  • PQB holds the target product and five core retrieval terms and their frequencies fixed while evaluating seven retrieval configurations across three product domains.It generates personalized syntactic queries and corresponding writing-error queries.Source: §5 · p. 23
  • Future work will broaden expression-difference coverage and incorporate richer user histories and real query data for further validation.Source: §5 · p. 23

A.1 History Threshold Sensitivity

The appendix reports how minimum-history thresholds relate to eligible user–product pairs and same-product cohort coverage.

A.1 History Threshold Sensitivity

A.2 Syntax Encoder Configuration

The appendix specifies the syntax encoder configuration and notes two properties of its learned representations.

A.2 Syntax Encoder Configuration

A.3 Gaussian Validity Filter Thresholds

The appendix lists Gaussian validity-filter thresholds and results, followed by the configuration for candidate query generation.

A.3 Gaussian Validity Filter Thresholds

  • Table 10 lists the Gaussian validity-filter thresholds.Source: Table 10 · p. 27
  • Table 11 reports Gaussian validity-filter results across three domains, with percentages relative to each domain’s initial Gaussian-profile cohort.Source: Table 11 · p. 28

A.4 Candidate Query Generation Configuration

A.5 Writing-Error Construction Statistics

This section reports selected-user writing-error profiles and the resulting query statistics across product domains, then lists retrieval and reranking configurations.

A.5 Writing-Error Construction Statistics

  • Table 13 reports personalized writing-error profile statistics for selected users, rather than the broader upstream population.The profiles are reported across product domains.Source: Table 13 · p. 28, §A.5 · p. 28
  • Table 14 reports the construction statistics of emitted personalized writing-error queries across three product domains.Source: §A.5 · p. 28, Table 14 · p. 29

A.6 Retrieval and Reranking Configurations

B Additional Absolute Effectiveness Results

This section presents absolute ranking effectiveness for canonical personalized syntactic-expression queries across three product domains and an equal-weight domain average.

B.1 Absolute Ranking Effectiveness

  • Table 16 reports absolute ranking effectiveness for canonical personalized syntactic-expression queries across three product domains and their all-domain average.Source: Table 16 · p. 30, §B.1 · p. 30
  • The all-domain average is the equal-weight average of the three domains.Source: §B.1 · p. 30
  • Within each metric column, the best and worst values across all seven models are marked red and blue, respectively.Source: §B.1 · p. 30

B.2 Absolute Ranking Effectiveness on Personalized Syntactic Expression and Personalized Writing-Error Queries

This section reports absolute effectiveness on both query sides of the same paired records, distinguishing clean personalized syntactic-expression queries from personalized writing-error queries.

  • Table 17 reports absolute effectiveness on both sides of the same paired records.Source: §B.2 · p. 30, Table 17 · p. 31
  • Clean columns report personalized syntactic-expression queries, while Error columns report personalized writing-error queries.Source: §B.2 · p. 31
  • Best and worst values in each metric column are marked red and blue across all seven evaluated models.Source: §B.2 · p. 31
Loading 2610.32164v1…