PaperlayerView exact revision on arXiv

Source-linked AI summary

Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search

Filippos Ventirozos, Tom Armitage, Mahmoud Oshagh, Peter A. Appleby, Matthew Shardlow

arXiv:2610.35102v1cs.CLcs.IR

TL;DR

Natural-language e-commerce parsers need labelled data, while synthetic-query diversity can conflict with faithfulness to structured intent. This paper tests persona-conditioned generation for automotive search and finds consistent IPIP parser gains alongside a syntax-related shift toward real traffic.

  • Problem

    Natural-language e-commerce parsers require extensive labelled data, while useful synthetic-query diversity can conflict with faithfulness to structured intent.

    Source: §1 · p. 1

  • Method

    The study generates synthetic queries from structured automotive intents with persona-conditioned prompts and evaluates parser utility, reference alignment, diversity, and audited failures.

    Source: §1 · p. 1, §3 · p. 2, §6.2 · p. 7

  • Results

    IPIP adds +1.1 to +2.1 F1 points in all four matched contrasts, while syntax moves generations toward real traffic and induces lexical diversity.

    Source: §1 · p. 1, §7 · p. 8

  • Takeaways & Limitations

    Alignment sKL tracks parser utility when its reference matches evaluation queries, whereas alignment to real traffic measures realism.

    Source: §7 · p. 9

    The study covers one UK automotive marketplace and schema, and the audit estimates hallucination conditional on universal parser failure rather than across the full corpus.

    Source: §9 · pp. 9–10

Abstract

from arXiv · show

Synthetic supervision can bootstrap semantic parsers in new search domains, but prompt-induced diversity is useful only when generations are faithful to the structured intent. We explore this for cold-start e-commerce search: 118,098 synthetic pairs produced by 6 LLMs with 9 persona-conditioned prompt families on a 47-field UK automotive schema, against 1,214 human-annotated and 15,304 production queries. We show that the IPIP personality bundle improves parser F1 in all four matched contrasts (Holm-corrected $p<.001$), and all three bundles have a positive main effect over the complete $2^3$ grid. Syntax conditioning is the axis that moves generations towards real traffic. It also induces lexical diversity. Alignment sKL, the divergence between von Mises-Fisher distributions fitted to synthetic and human-annotated queries, is an unlabelled proxy for parser F1 ($\rho=-0.917$ against human queries written for simulated shopper scenarios, lower is closer). An LLM committee, checked against a blinded author pass, triages the 927 queries missed by every parser and labels 37% as hallucinated. Dropped constraints dominate. The four syntax-bearing families have the largest hallucinated shares, 38 to 42%, while IPIP alone has the lowest at 28%.

1 Introduction

Natural-language e-commerce search requires parsers that map queries to structured fields, but training them needs extensive labelled data. This study tests whether persona-conditioned synthetic queries can vary usefully while remaining faithful to structured intent.

1 Introduction

  • The study examines synthetic supervision for UK automotive search, where queries map to a 47-field schema with long-tail combinations of constraints.The synthetic task is to turn structured intent into plausible natural-language queries for training a smaller downstream model.Source: §1 · p. 1
  • Prompting for greater lexical and structural variation can introduce unsupported attributes, reverse inequality directions, or produce repetitive templates when prompts are tightly constrained.The study varies prompts along IPIP personality, adjective, syntax, and UK demographic axes.Source: §1 · p. 1
  • Every conditioned parser beats the vanilla prompt; the complete 2^3 grid gives positive main effects for all three bundles, and IPIP adds +1.1 to +2.1 F1 points in four matched contrasts.The paper also reports that dropped constraints dominate audited defects, followed by unsupported additions and direction reversals.Source: §1 · pp. 1–2

2 Related Work

Prior work studies persona-conditioned variation and synthetic supervision for search, but often evaluates generated queries through diversity or downstream performance. This paper examines faithfulness against structured e-commerce intent alongside those measures.

2 Related Work

  • Persona-generation research commonly measures surface diversity or downstream accuracy, while whether persona-induced variation remains faithful to structured intent is still open.The cited work also identifies prompt length as a live confound in persona manipulation.Source: §2 · p. 2
  • E-commerce synthetic-query studies evaluate downstream retrieval or parser metrics, and rarely audit generated queries against the catalogue fields they encode.The passage describes synthetic supervision in cold-start product search and compound-constraint parsing.Source: §2 · p. 2
  • This study applies the question of induced variation to a structured e-commerce schema using diversity, human-reference alignment, parser utility, and an LLM-judge audit.The two closest prior works named are Kambhatla et al. (2025) and Schaffelder and Gatt (2026).Source: §2 · p. 2

3 Synthetic Dataset

The synthetic dataset pairs structured automotive search intents with generated natural-language queries under controlled prompt configurations. The study compares prompt families for internal diversity as well as their contribution to the corpus.

3 Synthetic Dataset

  • Queries were generated from a proprietary Autotrader UK dataset of 13M search intents collected over seven days, including field-value constraints and operator-valued fields.Operator words such as “under,” “within,” and “from” signal relations between values and fields in query text.Source: §3 · p. 2
  • Each LLM uses the same intents across all nine prompt settings, making configurations directly comparable within a generator.Intent sets differ between LLMs.Source: §3 · p. 2

3.1 Prompt Types and Motivation

  • The text-style bundles combine IPIP traits, syntax modes, and guarded adjective prompts; the UK demographic bundle remains standalone.The demographic prompt draws 3–15 attributes from a postcode profile, while the adjective prompts include a semantic-fit guard.Source: §3.1 · pp. 2–3

3.2 Diversity

  • Syntax-bearing ps, pas, as, and s lead on both diversity metrics, while syntax-free p, a, and pa remain within 0.003 Distinct-2 of n.The figure reports configuration means across six generators and shows ±1 standard deviation across LLMs; cross-LLM spread grows with syntax.Source: §3.2 · p. 3, Figure 1 · p. 4
  • The demographics-only u family is the only syntax-free prompt with a Distinct-2 lead over n, and that lead is small..Source: §3.2 · p. 4

4 Human Dataset

The study builds human query references through two annotation protocols and adds unstimulated production traffic for corpus comparison. It examines how reference construction relates to synthetic-query alignment and parser utility.

4 Human Dataset

  • The human dataset contains 1,214 query-intent pairs from a free-form website protocol and a fixed-intent persona-card protocol, with the latter supplying 87% of the corpus.The fixed-intent protocol trades some ecological validity for breadth and intent matching.Source: §4 · p. 4

4.1 Protocol 1: Verification through the Partner’s Website Front

  • Website-front annotators expressed their own free-form queries through the partner’s filters, yielding 156 pairs, while persona-card annotators wrote queries from fixed intents, yielding 1,058 pairs.The website-front queries were written without a schema or persona stimulus in view.Source: §4.1 · p. 4, §4.2 · p. 4

4.3 External Reference: Live Production Traffic

  • The unstimulated production reference contains 15,304 screened queries, used for alignment and corpus comparison but not parser scoring.It was drawn from one month of deployed-search traffic.Source: §4.3 · p. 4

4.4 Data Statistics

  • 47.3% of production queries are two words or shorter, against 0.4% of human-reference queries and 0.9% of synthetic queries.Synthetic queries average 17.6 words and 5.96 fields, while human queries average 14.3 words and 3.17 fields.Source: §4.4 · pp. 4–5

4.5 Alignment to Human and Production References

  • Alignment sKL is the symmetric KL divergence between fitted synthetic and reference distributions; lower values indicate closer alignment.The study also reports synthetic-side concentration as Spread κ, with lower values indicating broader distributions.Source: §4.5 · p. 5
  • The personality-plus-syntax configurations are closest to every reference, but persona-card wording shifts rankings; alignment sKL tracks parser F1 against persona-card queries (ρ = −0.917, p < .001) and not production traffic (ρ = −0.350, p = .36).Website-front and production references rank configurations similarly (Kendall τ = 0.944); the full in-house reference correlation is ρ = −0.800.Source: §4.5 · p. 5

5 Methodology

The methodology trains configuration-specific parsers and a mixed-corpus baseline, then uses an LLM committee to triage whether universally missed queries drifted from their intended constraints. It also compares synthetic-query alignment across human and production references, with within-reference rankings needed for interpretation.

5.1 Per-Configuration Parser

  • Each configuration-specific Flan-T5-Large parser trains on 10,910 examples, with 1,212 for validation; a mixed parser uses the same training volume sampled across all nine configurations.All parsers use best-checkpoint selection on validation F1 and are evaluated on the 1,214-pair human reference and a reserved synthetic test set; tuple-level F1 requires both field and value to match.Source: §5.1 · p. 5

5.2 LLM-as-Judge Committee

  • Three frontier-model judges independently assess whether each generated query hallucinates the seed intent, and majority vote supplies the label.The cleanroom prompt exposes only the seed field–value tuples and query, hiding model identity, prompt configuration, and parser predictions.Source: §5.2 · pp. 5–6
  • Alignment sKL compares each synthetic distribution with a reference distribution; Table 1 reports within-column ranks because absolute sKL is not comparable across references.Spread κ measures synthetic-side concentration, with lower values indicating a broader cloud; Table 1 orders rows by in-house tuple F1.Source: Table 1 · p. 6

6 Results

Conditioned prompts outperform the vanilla parser on the held-out human reference, with IPIP improving F1 in all four matched contrasts. The audit of universal parser failures finds that hallucination labels are conditional on this subset, and that dropped constraints and syntax-bearing prompts are prominent patterns.

6.1 Parser Utility on Human Eval

  • Every conditioned configuration beats vanilla by 0.9–3.6 F1 points, while IPIP improves F1 in all four matched contrasts by +1.08 to +2.14 points.All four IPIP confidence intervals exclude zero and have Holm-corrected p-values below .001; Table 2 reports tuple-level F1, precision, and recall on the held-out human reference.Source: Table 2 · p. 6, §6.1 · p. 6
  • Across the complete bundle grid, all three bundles have positive main effects, but their pairwise and three-way interactions are negative.The pas parser scores only 0.22 F1 points above pa and 0.27 above mixed, and both differences are statistically indistinguishable.Source: §6.1 · pp. 6–7
  • The IPIP gains remain positive across five human-reference subsets split by collection protocol and intent size; 16 of 20 confidence intervals exclude zero.Source: §6.1 · p. 7

6.2 Conditional Universal-Fail Audit

  • The audit estimates hallucination only among the 927 queries missed by every parser, about 10% of the sampled pool; the committee labels 37.2% of them hallucinated.This conditional estimate is not the overall hallucination rate; Table 3 reports majority-vote labels within the universal-failure subset.Source: §6.2 · p. 7, Table 3 · p. 8
  • Generator counts among universal failures are not directly comparable because each LLM verbalised a different intent set.Configurations share intent indices within each generator, but Opus-4.6’s 415 failures and GPT-5.4’s 62 therefore reflect different intents rather than a generator ranking.Source: §6.2 · p. 7
  • Hallucinated shares among universal failures range from 28.2% for p to 42.3% for pas, with syntax-bearing configurations in the upper half of every judge’s ordering.Noun-phrase and telegraphic modes have the largest reported shares, 43.4% and 41.8%; post-modified noun phrases have the smallest, 25.7%.Source: §6.2 · pp. 7–8
  • A blinded author audit agrees with the committee majority on 83% of 100 sampled cases; dropped constraints dominate the tagged hallucination rationales at 57.3%.The author identified omitted constraints, omitted search radii, and altered values; the committee is treated as a triage instrument rather than a definitive scorer.Source: §6.2 · p. 7

7 Discussion

The discussion separates the effects of the prompt axes across utility, realism, diversity, and faithfulness, showing that gains on one dimension can carry trade-offs on another.

Syntax (s)

  • Syntax moves generations towards real traffic and increases diversity, but deleted operator words are a common hallucination cue; combining p, a, and s turns both dials.Syntax has the smallest main effect (+0.76 F1); its four configurations have the highest hallucinated shares.Source: §7 · pp. 8–9

Personality (p)

  • +1.73 F1 is the personality bundle’s main effect, with positive gains in all four matched contrasts and the lowest hallucination share at 28.2%.Personality lengthens queries and broadens wording while preserving sentence structure; its utility gain does not come from moving closer to real traffic.Source: §7 · p. 8

Adjectives (a)

  • +2.20 precision, +0.15 recall, and +1.34 F1 are the adjectives bundle’s main effects, with most precision gains concentrated on Gemini-3.1 and Opus-4.6.The bundle alone is furthest from real traffic, but under syntax it moves closer; 33% of its hard cases are hallucinated.Source: §7 · p. 9

Demographics (u)

  • Demographics has no matched contrasts, ranks third on F1, and produces persona leakage when Gemini-3.1 adds a persona’s town or price preference without intent support.Its 37% hallucination share equals the pooled rate, and UK exposure is a caveat.Source: §7 · p. 9

Diversity and alignment as proxies

  • Alignment sKL tracks parser F1 against its own human reference, whereas Distinct-2 and Self-BLEU-4 do not; use a named reference and hand-check samples for faithfulness.The reported association is ρ = −0.917; synthetic κ can flag generations bunching up.Source: §7 · p. 9

Generator identity

  • Hallucination patterns vary by generator: Qwen3 has zero hard-case hallucinations, while Hunyuan-A13B and DeepSeek exceed half and account for nearly all direction reversals.These are descriptive generator-level patterns, and prompt recipes behave differently across models.Source: §7 · p. 9

8 Conclusion

The conclusion finds that persona-conditioned prompts improve parser performance, with different prompt axes serving utility, realism, and precision needs. It recommends targeted deployment choices and auditing each new generator for hallucinations.

8 Conclusion

  • +1.1 to +2.1 F1 for IPIP in all four matched contrasts; every conditioned parser beats vanilla, while pas, pa, and mixed form a statistically indistinguishable top group.Adjectives and syntax add smaller gains that partly overlap.Source: §8 · p. 9
  • Use IPIP as the default, adjectives when the parser over-extracts, and syntax when realism matters, checking that operator words survive.Alignment sKL is reported as a label-free proxy for parser F1 (ρ = −0.917 on the persona-card reference).Source: §8 · p. 9
  • Each new generator needs a hallucination audit of sampled generations; syntax-bearing prompts may improve realism while risking operator-word loss.The study compares 9 prompt configurations across 6 generators at matched training volume, using human-annotated and production queries.Source: §8 · p. 9

9 Limitations

The study’s evidence is bounded by one UK automotive marketplace and schema, and several design and evaluation choices constrain how its results should be interpreted.

Domain and distribution

  • Findings are evidence about one UK automotive marketplace and schema, despite automotive search serving as a stress test for varied field types.The study does not establish whether the findings transfer to other domains or schemas.Source: §9 · p. 9

UK-cultural exposure confound

  • Differences under UK demographic prompting may reflect unknown model exposure to British data, not demographic prompt design alone.The authors avoid attributing behavior to model nationality; the unobserved factor is training and alignment data composition.Source: §9 · pp. 9–10

Generator comparisons

  • Generator-level patterns are descriptive because generators differ in intent slices, vendor stacks, decoding controls, and closed-model implementations.Per-generator diversity trends and audited-pool counts are diagnostics, not risk estimates or a definitive leaderboard.Source: §9 · p. 10

Conditional faithfulness audit

  • The audit estimates hallucination only among universal parser failures, and cannot see systematic drift already absorbed by every parser.Its judges overlap with generator families; the 100-case author sample cannot estimate full-corpus label noise or support per-configuration rate comparisons.Source: §9 · p. 10

Reasoning-model and scale restriction

  • Prompt rankings should not be applied directly to smaller or non-reasoning generators without repeating the alignment, utility, and faithfulness checks.The pool contains relatively large reasoning-capable LLMs, which may respond differently to the prompt axes than other models.Source: §9 · p. 10

Prompt-axis sampling

  • Prompt-axis effects are not separated from instruction length because bundles differ in wording and no length-matched placebo was run.Alternative priors or bundle designs could shift absolute scores, so results concern these operationalizations rather than an exhaustive search over sampling policies.Source: §9 · p. 10

Ethics Statement

The statement documents data handling and risks, then describes the prompt-family legend and how IPIP, UK demographic, syntax, and adjective cues are operationalized.

Ethics Statement

  • Production query strings were screened for identifiers and sensitive details, with matching rows discarded; demographic prompts use synthetic area-level descriptors and should not guide profiling or targeting.Unfaithful queries could alter constraints and affect listings shown, so the authors recommend filtering or repair and user-facing confirmation, correction, or fallback mechanisms.Source: Ethics Statement · pp. 10–11

A Prompt Configuration Legend

  • The legend distinguishes four axes—UK demographics, IPIP personality cues, adjective injection, and syntax—with u as the only UK-axis family.The other eight families combine subsets of IPIP, adjective, and syntax axes.Source: Table 4 · p. 13

B IPIP Linguistic Mapping

  • IPIP Big Five traits steer linguistic cues: openness affects exploratory wording, conscientiousness constraint marking, extraversion enthusiasm, agreeableness politeness, and neuroticism tentativeness.Neutral traits contribute no text, keeping conditioning to at most five short cue strings per call.Source: B IPIP Linguistic Mapping · p. 13

C UK Demographic Persona Sampling

  • UK demographic prompts draw synthetic profiles from postcode-indexed area attributes, sampling one of 111 areas and 3–15 attributes for each generation call.The profiles use banded aggregates rather than individual-level records.Source: C UK Demographic Persona Sampling · p. 14

D Syntactic Mode Sampling

  • The syntax axis samples six query modes, from noun phrases and telegraphic fragments to questions, imperatives, post-modified noun phrases, and navigational fragments.Each mode receives a mode-specific instruction and two examples drawn from its curated example bank.Source: D Syntactic Mode Sampling · p. 14

E Attribute-Style Adjective Sampling

F Generator Pool Specification

The generator pool combines six reasoning-mode models with provider-recommended inference settings, while prompt-conditioned diversity responses vary across generators.

F Generator Pool Specification

  • The six reasoning-mode LLMs span closed-source frontier APIs and open-weight reasoning models.The exact model identifiers are listed in Table 8.Source: F Generator Pool Specification · p. 15
  • Each generator follows its provider’s recommended inference regime, with open-weight decoding parameters varied within vendor safe bands.For closed reasoning APIs, documented reasoning effort is the only varied parameter.Source: F Generator Pool Specification · p. 15
  • DeepSeek and GPT-5.4 are nearly flat across prompt configurations, while Hunyuan rises slightly and Qwen3 becomes more repetitive than under the vanilla prompt.Table 9 reports per-generator Self-BLEU-4, where lower values indicate greater diversity.Source: F Generator Pool Specification · p. 15

G Human Reference Data: Collection and Validation

The appendices document human-reference validation, corpus comparisons, robustness analyses, prompt-axis measurements, and the protocol and reliability checks for the universal-failure audit.

G Human Reference Data: Collection and Validation

  • The human reference combines persona-card and website-front queries, with slot-value F1 of 0.85 and 0.67 respectively; website-front disagreements mainly concern slot selection.Potentially inconsistent persona-card items were manually reviewed, while website-front disagreements were adjudicated and some pairs dropped.Source: G Human Reference Data: Collection and… · p. 15

H Corpus Comparison

  • Production field counts cover only 23 parity-shared fields, while the study schema has 47; on the parity basis, the human–production difference is 0.30 fields.Production length differences are attributed to real traffic’s short lookups, and its distinct-type rate of 0.705 indicates recurring strings.Source: H Corpus Comparison · p. 16, Table 10 · p. 16

I Protocol- and Complexity-Stratified Robustness

  • Paired bootstrap contrasts resample the same human queries for both configurations, and the protocol- and field-count-stratified table tests whether IPIP effects persist across those subsets.The nine training configurations share the same 12,122-intent multiset and field-count distribution.Source: I Protocol- and Complexity-Stratified… · p. 16, Table 12 · p. 17

K LLM Judge Protocol and Reliability

  • The committee labels 345 of 927 universal failures hallucinated by majority vote, but its audit is a triage signal rather than a parser-scale faithfulness score.The manual author audit’s 83% agreement is adjudicated after a blinded pass and correction pass, not independent agreement.Source: K LLM Judge Protocol and Reliability · pp. 17–18, Table 16 · p. 19

J Surface Anatomy of the Prompt Axes

  • Surface measurements distinguish syntax’s removal of grammatical glue, including operator words, from IPIP’s added annotator-register glue and vocabulary breadth.Alignment sKL contrasts find that adding syntax moves generations toward real production traffic in all four contrasts; the table presents negative changes as closer.Source: Table 14 · p. 18, Table 15 · p. 18

Related papers

Loading 2610.35102v1…