Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

1 Introduction

• Temperature 0 is not a guarantee of repeatable model outputs, and hosted deployments may change over time.The study asks how many outputs change, whether query operators amplify them, and what this means for benchmark conclusions and engine design.Source: §1 · p. 1
• In Movie Q3, gpt-oss-120B returned four counts across ten executions, while Llama-3.3-70B returned different stable counts on two inference stacks.The quality metric ranged from 0.64 to 0.93 for gpt-oss-120B; Llama-3.3-70B scored 0.43 and 0.36 on the two stacks, respectively.Source: §1 · p. 1
• The study reanalyzes SemBench results and re-executes SemBench, LRO-Bench, and UDA-Bench workloads through their own harnesses, logging every model call.The unchanged plans, prompts, and scoring let the logs connect answer changes to individual operator outputs.Source: §1 · p. 2
• Its contributions include a possible-worlds model, repeated-execution tests of benchmark findings, per-item logs and propagation analysis, and replays of voting and proxy cascades.The paper presents these as experimental insights into existing engines, plans, remedies, and benchmark practices, not new methods.Source: §1 · p. 2
• About one in five published SemBench results changes quality across executions, and plans can be deterministic; several per-query comparisons and implementation recommendations remain unsettled.Scenario rankings are stable, but many LRO-Bench recommendations are not settled by the executions run.Source: §1 · p. 2

2 Background and Related Work

• Semantic-query systems offer filters, maps, joins, rankings, and aggregations, while optimizers place, order, estimate, and reduce the cost of these operators.Prior systems use proxies and cascades, some with statistical guarantees; they generally treat the model as returning one answer per item.Source: §2 · p. 2
• Benchmarks report single executions or averages without per-item variance or tests of comparison stability; this study holds queries and plans fixed and tests published conclusions.Alrashed et al. [1] argue strict execution equality is the wrong criterion for generated text-to-SQL results with AI operators; this study measures per-item and per-answer variance.Source: §2 · p. 2
• Temperature-zero variation has been reported in code generation and LLM benchmarks, but prior work measures free-form outputs or aggregate accuracy rather than judgments inside query plans.Dynamic batching can change floating-point reduction order and request logits; reduced precision can amplify the effect [54].Source: §2 · pp. 2–3
• The paper connects varying semantic outputs to probabilistic relations, noisy crowdsourced judgments, view materialization, and provenance.It treats an LLM operator as a worker that can disagree with itself and measures what redundancy buys.Source: §2 · p. 3

3 A Possible-Worlds Model and Measures

• An execution is a possible world of item outputs drawn from per-item distributions, which the upper plan maps to a query answer.Answer reproduction is the probability that two independent executions return the same answer; item disagreement is the probability that an item differs between them.Source: §3 · p. 3
• For answers that expose every item, reproduction is approximately e^(-nδ̄), so it declines exponentially with the number of items; COUNT and LIMIT behave differently.COUNT changes can cancel and reproduction depends on count spread; LIMIT depends only on the prefix through the kth qualifying row.Source: §3 · p. 3
• For plans such as ranking by mean score, the model estimates reproduction by simulating pairs of worlds through the plan.Table 1 reports which items decide each workload answer and how often the closed-form or simulated prediction is within 0.15 of measured reproduction.Source: §3 · p. 3, Table 1 · p. 3
• The measures include item flip rate over k executions, pairwise disagreement, answer reproduction, quality range, and a plan-based independence prediction.The decision margin is the log-probability gap between the two most likely tokens at the first output-determining token; contestedness compares an item’s majority outputs across other deployments.Source: §3 · p. 3

4 Study Design

• The study frames five questions about changes in published results, operator outputs, answers, execution conditions, and the benefits of repetition and calibrated plans.It restricts the study to text and tabular data and uses each benchmark’s own harness.Source: §4 · p. 3, §4.1 · p. 3

4.1 Workloads

• SemBench workloads cover Movie and Medical queries, including filters, maps, and joins; LRO-Bench covers five LLM-enhanced relational operators; UDA-Bench runs SQL over extracted document attributes.The SemBench joins use an exact 1,560-pair sample, rather than the benchmark’s randomized cascade.Source: §4.1 · pp. 3–4

4.2 Models, serving, and protocol

• The protocol runs benchmark plans and prompts at temperature 0, logs every model call, and compares deployments, including managed and vLLM services of the same model.The two Llama deployments received identical prompts on 7,544 shared requests; the study logged 1,219,338 calls.Source: §4.2 · p. 4

5 RQ1: Published Benchmark Results

• 21% of SemBench’s 101 text-query system cells changed quality across five executions, including 12 of 50 cells with deterministic plans.Excluding three cells that recorded a no-row failed execution, 20% of all cells still changed.Source: §5 · p. 5
• 7 of 112 text-query system pairs whose mean qualities differed were ordered the other way by at least one execution.For 7 of 12 variable text-query pairs, separating the mean-quality difference at 95% required more than SemBench’s five executions per system.Source: §5 · p. 5
• Across all six scenarios, the best system by mean quality remained best in every execution, and single-execution rankings matched mean rankings in 26 of 29 executions.The study concludes that per-query results should be reported with their spread.Source: §5 · p. 5

6 RQ2: How Many Items Change?

• 0.07% to 10.2% of SemBench filter and map items changed over five executions, depending on the model, rising to 24.0% on one query.For gpt-oss-120B, the pooled rate rose from 5.2% over two executions to 10.2% over five and 13.0% over ten.Source: §6 · pp. 5–6
• On the same SemBench filters and maps, GPT-4o changed 7.5% of items and Claude Sonnet 5 changed 4.1% over five executions.Both rates fall within the range of the six open-weight deployments; GPT-4o’s content filter rejected every attempt on four queries.Source: §6 · pp. 5–6, Table 6 · p. 6
• For gpt-oss-120B, item flip rates were 3.4% on Boolean filters, 17.5% on 1–5 scores, and 24.0% on 24-way diagnosis.The study reports that graded judgments and free text change more than lookups of stated facts, regardless of output-space size.Source: §6 · p. 6, Figure 3 · p. 6

7.1 SemBench filters and maps

• For 16 of 17 SemBench cells whose answers expose every item, e^-x was within 0.15 of measured reproduction; for answers using LIMIT, aggregates, or rankings, 32 of 36 reproduced at least as often as that curve predicts.Using the first five executions to predict the next five was within 0.15 on 8 of 10 queries; rare changes caused a large miss on Movie Q2.Source: §7.1 · pp. 6–7, Figure 4 · p. 7
• On Movie Q1, gpt-oss-120B’s quality stayed at 1.00 across ten executions even though only 0.53 of execution pairs returned the same five reviews.The quality of Movie Q3 ranged from 0.64 to 0.93.Source: §7.1 · p. 7

7.3 Relational operators and their implementations

• On LRO-Bench, gpt-oss-120B’s answers differed across executions on 413 of 726 answered queries (57%); 16 of 28 implementation-recommendation tests had intervals containing zero.Single executions disagreed on the better implementation in six tests; only ALL over ONE for filters on columns and tables held on every deployment.Source: §7.3 · p. 7, Table 7 · p. 8, Table 10 · p. 9

7.2 Semantic joins

• On the full-pair join Q7, Qwen-72B and Llama-70B (V) changed only 0.3% and 0.2% of verdicts yet reproduced answers in 0.10 and 0.20 of pairs.Q5 and Q6, which return only the first ten matches, reproduced in every execution on the same deployments.Source: §7.2 · p. 7, Table 8 · p. 8

7.4 SQL over extracted tables

• On UDA-Bench, 4.0% of Llama-70B (M)’s extracted values changed while the result tables differed on 83 of 123 queries (67%).For gpt-oss-120B, 7.5% of extracted values and 68% of answers changed; the exposure model was within 0.15 on 70 of 123 queries.Source: Table 9 · p. 8, §7.4 · p. 8

7.5 Three levels of reproducibility

• Across SemBench, LRO-Bench, and UDA-Bench, answers were identical over five executions on 48%, 59%, and 19% of queries, respectively.Quality was identical on 52%, 71%, and 33%, so benchmarks reporting reproducibility should state whether they measure operator outputs, answers, or quality.Source: §7.5 · pp. 8–9, Figure 5 · p. 9
• Aggregates, rankings, full joins, and SQL over extracted tables can turn a few percent of changed items into changes to most answers, while LIMIT absorbs some changes and quality metrics hide others.The study finds that implementation recommendations inherit variance and that many LRO-Bench recommendations are not settled by its executions.Source: §7.5 · p. 9

8 RQ4: Which Items Change, and Why

• 77% of changing SemBench items are contested, versus 34% of stable items; flip rates rise with the number of other deployments dissenting.For gpt-oss-120B, the rate rises from 3.2% when all other deployments agree to 37.5% when three dissent; for Llama-70B (M), it rises from 0.19% to 10.7%.Source: §8 · p. 9
• Identical duplicated requests disagree within an execution about as often as across executions, so a single answer can combine contradictory outputs.For gpt-oss-120B, the rates are 6.0% within executions and 6.1% across executions; 91 of 533 duplicated inputs differ within at least one execution.Source: §8 · p. 9
• Decision margin and reasoning length predict which items change: Granite-8B’s margin has AUC 0.93, while gpt-oss-120B’s trace length has AUC 0.84.The figure plots later-execution changes against first-execution margin and item flip rate against cross-deployment disagreement; longer traces also correlate with more diverging pairs (0.57).Source: §8 · pp. 9–10, Figure 6 · p. 10
• Granite-8B’s fitted noise scale is 0.28 nats, and items with margins below one nat account for 78% of its changes.Those low-margin items are 7.8% of Granite-8B’s items; the fitted model includes a 0.26% floor.Source: §8 · p. 10
• The same model on managed service and vLLM differs on 1.8% of Movie items, while at least one LRO-Bench execution fails on 59% versus 9% of queries.On Movie Q3, the two stacks return different quality scores, 0.43 and 0.36; across campaigns, managed-service pairs disagree on 1.9% of items.Source: §8 · p. 10
• No LRO-Bench operator implementation is most stable for every operator, and the tested client-setting comparisons show no detectable effect on disagreement.Across whole-operand, per-element, and hybrid implementations, gpt-oss-120B differs on 35%–97% of answered queries; the setting intervals rule out only large effects.Source: §8 · pp. 10–11

9 RQ5: What Repetition and Calibration Buy

• A vote of three removes 36% of pairwise disagreement at up to 2.13 calls per item, and a vote of five removes 48%, but combined answers are not reproducible.The ten executions form only two groups of five, so the Vote 5 estimate rests on one pair; majority outputs remain wrong on 17.9%–30.7% of labelled items, depending on deployment.Source: §9 · p. 11, Table 12 · p. 11
• Re-asking the 20% of items with the longest first reasoning traces achieves 64% of a full vote’s disagreement reduction at 1.4 calls per item.Re-asking the longest-reasoning 30% achieves 80% of the reduction at 1.6 calls per item, while a random 20% achieves 29%.Source: §9 · p. 11
• On Medical Q1, the cascade misses its 90% recall target in 2.2% of calibrations against another oracle execution, below the allowed 5%.The same 400 rows labelled by two oracle executions produce identical returned rows in only 24% of cases; the table reports recall violations and row-set reproducibility.Source: §9 · pp. 11–12, Table 13 · p. 12
• Five-execution uncertainty bounds do not bracket later set answers: on Medical Q1, later runs omit 1.6 certain patients on average and add 3.0 rows not even possible.For the count of Movie Q3, the central 95% interval [9, 13] contains every later count but excludes the gold count of 14.Source: §9 · p. 12
• Recording operator outputs makes an answer reproducible; stored outputs take about 39 bytes each, and deduplication saves 6.5% of calls in the measured execution.One gpt-oss-120B execution of ten SemBench filters and maps produces 348 KB of outputs.Source: §9 · p. 12

10 Lessons

• Report per-query quality spread, answer reproduction, stated-execution flip rates, failures, per-item outputs, and model deployment; base execution counts and recommendations on results across runs.Five executions did not suffice for 7 of 112 published text comparisons, and 6 of the tested LRO-Bench recommendation tests could go either way on one execution.Source: §10 · p. 12
• Treat stored semantic columns as samples, deduplicate within-execution calls, and estimate answer stability from item disagreement and plan shape.Answers exposing n items reproduce with probability about e^-nδ; record calibration labels and reuse recorded operator outputs when reproducing an answer.Source: §10 · p. 12

11 Threats to Validity

• RQ2–RQ5 use six open-weight deployments and two commercial models for SemBench filters and maps; the deployment comparison rests on one model served two ways.Per-item logs require an engine-client hook, and only margins from deployments returning log-probabilities are available.Source: §11 · p. 12
• The study covers text and tabular workloads; its joins use a 40-review sample, and the cascade replay fixes proxy scores to one execution.That fixed proxy scoring understates the variance of a real cascade.Source: §11 · p. 13
• Five executions limit detectable flip rates and give answer-reproduction granularity of 0.1, while measured flip rates continue to grow with more executions.On five Movie queries, Granite-8B flips 3.1% of items at five executions and 5.2% at 40.Source: §11 · p. 13

• Item-output changes depend on the model and operator, and plans amplify them exponentially in the items an answer exposes.Source: §12 · p. 13
• These changes unsettle per-query system comparisons and implementation recommendations.Source: §12 · p. 13
• Benchmarks should report quality spreads, answer reproduction, and failure rates; engines needing stable answers should record the operator outputs that produced them.Source: §12 · p. 13