PaperlayerView exact revision on arXiv

Source-linked AI summary

100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models

Yeounoh Chung, Rushabh Desai, Jian He, Yu Xiao, Thibaud Hottelier, Yves-Laurent Kom Samo, Pushkar Khadilkar, Xianshun Chen, Sam Idicula, Fatma Özcan, Alon Halevy, Yannis Papakonstantinou

arXiv:2603.15970v6cs.DBcs.AI

TL;DR

LLM-based AI queries expand semantic querying across structured and unstructured data, but their cost and latency are challenging for large-scale analytics and low-latency database workloads. This paper evaluates lightweight proxy models over embeddings, using online and offline training strategies. It reports 100x latency and cost improvements with accuracy similar to LLM-based counterparts for BigQuery and AlloyDB setups.

  • Problem

    LLM-based semantic operators can be prohibitively costly at scale, while their latency restricts practical use in HTAP workloads.

    Source: §1 · p. 1

  • Method

    The paper evaluates lightweight non-LLM proxy models over embedding vectors for semantic operators, with online training for ad hoc queries and offline training for known or recurring tasks.

    Source: §1 · p. 1, §4 · p. 3, §4.1 · p. 4

  • Results

    Across BigQuery OLAP and AlloyDB HTAP setups, proxy models provide 100x latency and cost improvements while achieving similar accuracy to LLM-based counterparts.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The authors conclude that large-scale OLAP and ultra-low-latency HTAP with semantic operators are possible using proxies, with costly LLM calls reserved for hard cases when necessary.

    Source: §7 · p. 14

    For semantic ranking, the study focuses on proxy substitution, while existing cross-attention re-rankers are effective in many settings and their integration is identified as future work.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Several data warehouse and database providers have recently introduced extensions to SQL called AI Queries, enabling users to specify functions and conditions in SQL that are evaluated by LLMs, thereby broadening significantly the kinds of queries one can express over the combination of structured and unstructured data. LLMs offer remarkable semantic reasoning capabilities, making them an essential tool for complex and nuanced queries that blend structured and unstructured data. While extremely powerful, these AI queries can become prohibitively costly when invoked thousands of times. This paper provides an extensive evaluation of a recent AI query approximation approach that enables low cost analytics and database applications to benefit from AI queries. The approach delivers >100x cost and latency reduction for the semantic filter operator and also important gains for semantic ranking. The cost and performance gains come from utilizing cheap and accurate proxy models over embedding vectors. We show that despite the massive gains in latency and cost, these proxy models preserve accuracy and occasionally improve accuracy across various benchmark datasets, including the extended Amazon reviews benchmark that has 10M rows. We present an OLAP-friendly architecture within Google BigQuery for this approach for purely online (ad hoc) queries, and a low-latency HTAP database-friendly architecture in AlloyDB that could further improve the latency by moving the proxy model training offline. We present techniques that accelerate the proxy model training.

1 Introduction

LLM-based AI operators bring semantic querying to structured and unstructured data, but their cost and latency constrain large-scale OLAP and low-latency HTAP workloads. The paper evaluates lightweight proxy models as a way to reduce LLM use while retaining accuracy.

1 Introduction

  • LLM costs make semantic processing economically unfeasible on millions of OLAP rows, while LLM-call latency restricts practical HTAP applications.Prior integration work often assumes small tables or does not adequately address these performance and cost implications.Source: §1 · p. 1
  • The approach uses cheap ML proxy models over embedding vectors for semantic operations reducible to classification, reserving LLM calls for cases requiring their reasoning.The prototype evaluates semantic filter and rank operators in BigQuery and AlloyDB across benchmarks, including one with 10 million rows.Source: §1 · pp. 1–2
  • On the 10-million-row AI.IF query, online proxies deliver 329x latency improvement and 728x cost savings, while offline proxies deliver 991x latency improvement and 792x cost savings.The comparison is against BigQuery's distributed LLM-based AI query engine.Source: Table 1 · p. 2, §1 · p. 2
  • Earlier work reported proxy approaches falling short on complex tasks, while this paper argues that conventional ML can work well when applied to suitable problems.The introduction cites prior findings that proxies can struggle when substituted blindly for LLMs.Source: §1 · pp. 1–2

2 Related Works

Prior AI-query systems largely optimize LLM execution, while treating non-LLM proxy models as baselines with insufficient accuracy. This work instead studies aggressive proxy substitution for high-volume semantic filtering and ranking, with LLM fallback below a preset approximation threshold.

2 Related Works

  • Prior systems reduce costly LLM calls through row skipping, operator reordering, or model cascading, while often considering proxy models insufficiently accurate.The cited prior work includes UQE and AI-query optimization systems.Source: §2 · p. 2
  • This work substitutes lightweight proxies for LLM invocations across the table and falls back to an LLM when proxy approximation falls below a predetermined threshold.Its stated focus is high-volume semantic filtering and ranking rather than merely reducing LLM calls or replacing them with smaller LLMs.Source: §2 · p. 2

3 A Case for Lightweight Proxy Model For Efficient AI Query Engine

The section makes the case for simple proxy models as a low-cost alternative for suitable semantic tasks, then defines proxy approximation and illustrates its performance and scaling behavior.

3 A Case for Lightweight Proxy Model For Efficient AI Query Engine

  • The paper positions proxy models as alternatives to optimization efforts focused on making LLM-based implementations more efficient.Those efforts emphasize model cascading and algorithmic improvements to LLM-based operations.Source: §3 · pp. 2–3
  • Proxy models can achieve comparable or better results at significantly lower cost and latency in certain scenarios.Table 2 compares embedding-plus-logistic-regression proxies with an LLM baseline for AI.IF on the Kaggle SPAM email classification dataset.Source: Table 2 · p. 3, §3 · p. 3
  • For this particular dataset, offline and online proxy models and the LLM baseline both score above 90% accuracy; proxy models yield slightly superior accuracy, at 97%.A proxy model is a text-embedding-based Logistic Regression classifier trained with 200 examples labeled with gemini-2.5.Source: §3 · p. 3
  • Online proxy latency gains grow with table size as sampling, labeling, and training overhead is amortized; offline latency reduction remains static across table sizes.Both proxy prediction time and LLM inference time grow proportionally with data size.Source: §3 · p. 3
  • A proxy model is a simpler model that approximates a complex model's behavior for a relevant subset of inputs and a specific task or query.The prototype applies lightweight proxy approximation to reduce AI-query latency and cost, with LLM fallback when approximation quality is insufficient.Source: §3 · p. 3

4 Lightweight Proxy Approximation

The system automates proxy training and selects between proxy models and LLMs for each semantic operator using measured proxy quality. It supports online training for ad hoc analytics and offline training for known workloads.

4 Lightweight Proxy Approximation

  • Proxy models are automatically trained and deployed only when evaluation shows they meet a quality threshold; otherwise, the system falls back to the LLM baseline.The workflow parses each operator, prompt, and column reference, then automates embedding generation, sampling, LLM labeling, and model fitting.Source: §4 · p. 3, §4.3 · p. 4, §4.4 · pp. 4–5

4.1 Online and Offline Proxy Model Training

  • Online training runs during BigQuery query execution, while offline training prepares models for known or recurring AlloyDB workloads outside the query path.Offline models require periodic retraining as data changes, and their performance on filtered data slices needs monitoring.Source: §4.1 · pp. 3–4

4.2 Imbalanced Label Training

  • Imbalanced labels make representative sampling and proxy quality difficult; weighted training is the default, with SMOTE used when minority examples are insufficient.Other hyperparameters use defaults because tuning does not significantly change results.Source: §4.2 · p. 4

4.3 Automatic Proxy Model Evaluation

  • For online training, the engine evaluates proxy models on the LLM-labeled training sample and uses the LLM for remaining rows if quality thresholds are not met.Offline models can instead be evaluated on pre-curated, LLM-labeled test data for known workloads.Source: §4.3 · p. 4

4.4 Adaptive Proxy Model Selection

  • For each operator, selection compares proxy quality with the LLM baseline on relevant samples and chooses a qualifying proxy or the LLM.The selection considers operator type, prompt, and data column, using online evaluation results or a pre-curated test dataset.Source: §4.4 · pp. 4–5, Figure 1 · p. 5

5 Detailed Analysis of Proxy Approximation

The evaluation implements proxy-based semantic filtering and ranking in BigQuery and AlloyDB, whose online OLAP and offline-trained HTAP configurations represent different workload needs. It emphasizes BigQuery results that include online training overhead.

5 Detailed Analysis of Proxy Approximation

  • The study evaluates proxy approximations for semantic filtering (AI.IF) and semantic ranking (AI.RANK), focusing most micro-benchmarks on AI.IF.The authors relate ranking to classification over input-column embeddings and assess performance across classification and information-retrieval benchmarks.Source: §5 · p. 5, §5.1 · p. 7

5.1 Evaluation Setup

  • BigQuery uses online model training for large-scale OLAP, while AlloyDB prepares models offline for known query patterns and latency-sensitive HTAP workloads.The evaluation reports BigQuery latency breakdowns including online training overhead, alongside separate AlloyDB latency results.Source: §5.1 · p. 5, §5 · p. 5
  • The filtering benchmarks turn multicategory datasets into one-vs-rest queries, while ranking benchmarks assess relevant-document retrieval in heavily imbalanced corpora.Ranking corpora may contain hundreds of thousands or millions of documents, with few relevant documents per query.Source: Table 3 · p. 6, Table 4 · p. 6
  • The evaluation compares proxy models with pure Gemini LLM execution and does not directly compare against plan-optimization techniques.The stated focus is proxy approximation over tables with millions of rows; datasets and operators are evaluated using classification and information-retrieval benchmarks.Source: §5.1 · pp. 6–7

5.2 Fast Semantic Filtering

Proxy filtering closely matches LLM quality across benchmarks while reducing cost and latency, with performance depending on dataset scale and whether embeddings are precomputed. Offline training removes training overhead from the online path, while online optimization also permits fallback after proxy evaluation.

5.2 Fast Semantic Filtering

  • Proxy filtering achieves near-perfect relative accuracy against the LLM and slightly exceeds it on some datasets, but also inherits errors on difficult queries when LLM predictions are inaccurate.The cited comparison is macro F1 and relative accuracy for semantic filtering; multi-label results average one-vs-rest tasks.Source: §5.2 · p. 7, Table 5 · p. 8
  • On FEVER, proxy relative accuracy is 0.917 across 6.6k claims, with training under a minute; LOTUS reports ∼190s per 1,000 claims, though hardware prevents direct comparison.The proxy inference takes a few seconds, and its training includes 1,000-sample sampling, labeling, and fitting.Source: §5.2 · p. 7
  • With pre-computed embeddings, proxy models deliver orders-of-magnitude speedups and cost savings over pure LLM execution; on-the-fly embeddings yield 5x speed-up and 2.5x lower cost.The on-the-fly cost reduction remains constant across data sizes, while pre-computed embedding gains scale with larger tables.Source: §5.2 · p. 7, Table 6 · p. 8
  • Offline-trained AlloyDB proxies provide latency gains that remain constant across table sizes because both proxy and LLM processing times grow linearly with table size.The offline architecture uses pre-trained models and a fixed number of workers in one database VM instance.Source: §5.2 · p. 7, Table 7 · p. 8
  • Sampling and prediction are the only optimization steps that scale with input size, while BigQuery parallelism keeps prediction's time fraction stable from 1m to 10m rows.Sampling and training overhead is small enough to support eager proxy training and later LLM fallback; the cited futile-training overhead is mostly sampling at 25%.Source: Figure 2 · p. 7, §5.2 · p. 7

5.3 Fast Semantic Ranking

Proxy-based semantic ranking is evaluated against cross-attention re-ranking and LLM ranking, with task complexity shaping which approach performs best. The proxy reformulates ranking as classification and can fall back to LLM ranking when its quality constraint is not met.

5.3 Fast Semantic Ranking

  • Proxy converts ranking into classification using binary relevance labels or rubric scores, while adaptive selection uses LLM ranking when proxy quality misses a threshold.For TREC-DL-2022, the proxy is trained as a multi-label classifier; Figure 3 reports better online Proxy performance after more than 120 labeled samples.Source: §5.3 · p. 8, Figure 3 · p. 9
  • Five of six datasets favor Re-Ranker; for 500 candidates, it costs 2.5% of Proxy and 1% of LLM, while nuanced TREC-DL-2022 favors LLM and Proxy.On TREC-DL-2022, Proxy costs roughly 40% of the LLM baseline.Source: Table 8 · p. 8, Table 9 · p. 9, §5.3 · p. 9
  • For each query, the system prefilters 500 documents by embedding similarity and labels 200 with LLM to train Proxy, eliminating 300 LLM calls per query.The filtered set is re-ranked to retrieve the top 10 documents.Source: §5.3 · p. 9
  • On HellaSwag, Proxy maintains nDCG@10 within 54% relative accuracy of LLM while avoiding 300 LLM calls; sparse-relevance datasets trigger fallback to LLM.The paper attributes the fallback need to insufficient minority-class examples for learning the LLM decision function.Source: §5.3 · p. 9

5.4 Impact of Sampling Strategy

Sampling strategy affects whether proxy training data can be balanced, while imbalanced-label techniques affect proxy robustness. Random sampling can suffice for less imbalanced data, but highly skewed labels may require more targeted sampling or training adjustments.

5.4 Impact of Sampling Strategy

  • With one relevant document among 5K, a balanced training sample is nearly impossible and imbalance rises with sample size; with enough relevant documents, stratified sampling can balance the sample.Top-K sampling can produce higher imbalance ratios by favoring one class according to query-document relevance.Source: §5.4 · p. 10, Figure 4 · p. 10
  • Random sampling can perform well on tables that are not heavily imbalanced, while active-learning stratified sampling can yield more robust samples at higher sampling cost.For datasets with high population imbalance, Top-K and Random can both fail to create balanced training data.Source: §5.4 · pp. 9–10

5.5 Imbalanced Data Label Challenge

  • Imbalanced Standard training is not robust, whereas imbalance-aware techniques improve robustness; at imbalance ratios ≥50, downsampling can perform best by preserving minority-class samples.SMOTE can help when minority examples are scarce, but synthetic or reweighted samples can change class distributions and decision boundaries.Source: §5.5 · p. 10
  • At a fixed imbalance ratio of 10, training overheads vary by technique, but training remains ≤2% of end-to-end proxy optimization, leaving room for costlier methods when minority examples are scarce.Balanced and Downsampled are relatively cheaper than other techniques; overhead can rise with table size and imbalance ratio.Source: §5.5 · p. 10, Table 11 · p. 10

5.6 Role of Embedding Quality

Embedding model and dimension influence proxy classification quality, and the lowest-cost embedding is not necessarily the most accurate. The reported comparisons also connect embedding distinctiveness with downstream proxy performance.

5.6 Role of Embedding Quality

  • Gecko provides consistently high-quality embeddings and outperforms 3072-dimensional Gemini on Tweet Sentiment, while Gemma embeddings yield the lowest proxy performance.Gecko and Gemma support dimensions up to 768; Gemini supports up to 3072.Source: §5.6 · p. 10, Figure 6 · p. 11
  • Gemma is the fastest and most cost-efficient embedding option, while Gecko can match Gemini effectiveness at lower cost; 3072-dimensional Gemini embeddings also increase storage costs.Reduced-dimension embeddings compress model representations but lower embedding quality across the compared models.Source: §5.6 · p. 11, Table 12 · p. 11
  • PCA separability results show more distinctive, better-separated Gecko clusters than Gemma clusters, helping explain Gemma’s weaker proxy results; Emotion classification remains difficult because six clusters overlap.The PCA visualization uses PC1 and PC2, with separability defined as average inter-class distance divided by average intra-class variance.Source: §5.6 · p. 11, Figure 7 · p. 12

5.7 Online VS. Offline Training and Model Selection

Offline hyperparameter tuning did not materially improve classification performance, while Logistic Regression offers a favorable training-speed and F1 balance. For relationally filtered slices, a general proxy trained on the full table can perform reliably, though its robustness depends on the data and LLM labeling quality.

5.7 Online VS. Offline Training and Model Selection

  • Tuning produced negligible or slightly negative F1 changes across benchmarks; Logistic Regression retained 0.867 F1 after tuning, without improvement.The authors conclude that this removes the main rationale for iterative, resource-intensive online hyperparameter optimization.Source: §5.7 · p. 12, Table 13 · p. 12
  • Logistic Regression is the fastest model and tied for the highest F1; SVM takes 49.84x the LR training latency, versus 1.42x for RF and 3.18x for XGB.The latency ratios are normalized to LR training latency of 1x.Source: §5.7 · p. 12, Table 13 · p. 12

5.8 Proxy Model Robustness Across Relational Filter Predicates

  • A general proxy trained on the whole table performs reliably on most California Housing slices when LLM labels are accurate; slice-specific proxies may perform worse from insufficient or imbalanced data.The results are data-dependent, and proxy accuracy is low when LLM accuracy on the slice is low.Source: §5.8 · pp. 12–13, Table 14 · p. 13

6 Discussion and Lessons for Future Research

The discussion recommends a diverse set of specialized models for semantic operators and outlines extensions to classification and semantic joins. It also identifies sampling and scaling constraints that future systems must address.

6.1 Optimize Using Diverse Array of Models

  • Future query engines should use specialized proxies beyond simple classifiers and a single LLM baseline, including logistic regression, fine-tuned or small LLMs, and cross-attention re-rankers for ranking.Model cascading offers a precedent, but the paper identifies using a diverse array of cheaper models as a research avenue.Source: §6.1 · p. 13

6.2 Proxies for other AI Query Operators

  • AI.IF proxies can extend to multi-class AI.CLASSIFY, but accurate training requires representative samples for every class, especially rare or minority classes.The paper notes that multi-label classification may require more randomly sampled examples as the number of labels increases.Source: §6.2 · p. 13
  • A naïve semantic join proxy still requires O(N×M) inferences, so the paper identifies pre-filtering, such as vector similarity, as necessary to limit candidate pairs.Joining two tables of 10,000 rows would require 100 million inferences; join optimization is outside this work's scope.Source: §6.2 · p. 13
  • Proxy-based joins depend on workload characteristics such as join selectivity and predicate complexity, with low match density making sampling and training difficult.The paper cites SciFact as an example of the needle-in-a-haystack challenge.Source: §6.2 · p. 13

7 Conclusion & Future Work

The paper evaluates non-LLM proxy models for semantic operators in BigQuery OLAP and AlloyDB HTAP settings. It reports roughly 100x latency and cost improvements with similar accuracy, and points to mixed-model optimizers as future work.

7 Conclusion & Future Work

  • 100x latency and cost improvements with similar accuracy were demonstrated for proxy models across the BigQuery OLAP and AlloyDB HTAP setups.Source: §7 · p. 13
  • The gains come from reducing AI.IF and AI.RANK to binary classification, using careful stratified sampling, and employing effective embedding models.Source: §7 · p. 14
  • Future optimizers should use low-cost proxies and fall back to costly LLM calls for specific hard cases, with automated model mixes left for future evaluation.Source: §7 · p. 14
Loading 2603.15970v6…