PaperlayerView exact revision on arXiv

Source-linked AI summary

QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing

Yueying Li, Zhongle Xie, Ke Chen, Lidan Shou

arXiv:2610.09894v1cs.DBcs.AI

TL;DR

In-database predictive queries apply Transformer models to SQL-selected tuples, but query context can reveal distinct FFN activation patterns that inference pipelines do not use. QCATS selects context-aligned slices before inference through query-level routing. It improves throughput and memory use while preserving predictive accuracy compared with dense baselines.

  • Problem

    In-database predictive queries apply Transformers to SQL-selected tuples, whose distinct query contexts induce different FFN activation patterns.

    Source: §I · p. 1

  • Method

    QCATS profiles a fine-tuned dense model to construct context-aligned expert slices and uses query context to select sparse parameter combinations before inference.

    Source: §I · p. 2

  • Results

    QCATS improves inference throughput and reduces memory consumption while preserving predictive accuracy compared with dense baselines.

    Source: §VII · p. 12

  • Takeaways & Limitations

    Query-level context can select experts effectively for database-oriented tasks, matching MoEBERT performance with a simpler routing strategy.

    Source: C. System Effectiveness (RQ2) · p. 10

Abstract

from arXiv · show

In-database predictive query processing increasingly applies Transformer-based models within relational pipelines. However, existing in-database inference typically exposes only tuple-level model inputs to the inference runtime, leaving relational predicates and metadata statistics invisible to neural execution planning. In this paper, we propose QCATS, a query context-aware transformer slicing framework that enables efficient sparse inference inside database systems. QCATS executes at query granularity: instead of routing individual tokens or tuples during inference, it uses query predicates and metadata statistics to pre-select context-aligned FFN slices before model execution. The framework comprises offline expert construction and lightweight query-level routing that dynamically selects experts during execution. QCATS further introduces system optimizations, including asynchronous CPU-GPU pipelines and routing-aware batching. Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42x latency reduction while preserving prediction accuracy comparable to dense baselines.

I. INTRODUCTION

Predictive queries expose relational context that current inference pipelines often discard, even though query-selected data can produce distinct FFN activation patterns. QCATS uses that context to select model slices and execute inference at query granularity, alongside database-oriented system optimizations.

I. INTRODUCTION

  • Distinct query contexts activate different FFN neuron subsets, motivating context-aligned slices rather than using the full model uniformly.The paper reports different activation patterns across query contexts and identifies the pattern using average activations for tuples selected by eight queries.Source: §I · p. 1
  • Black-box inference UDFs discard relational constraints and metadata, so models run dense inference for each tuple despite query-level locality.The paper identifies resulting computation and memory waste in predictive query processing.Source: §I · pp. 1–2
  • QCATS preselects model parameters from query context before inference, aligning sparse execution with database workloads and avoiding runtime control-flow divergence.It treats inference as a query physical operator specialized using optimizer-visible relational context.Source: §I · p. 2
  • QCATS decomposes fine-tuned dense FFN layers into shared and semantic components, then routes query contexts to sparse expert combinations without training separate expert networks from scratch.Routing moves conditional computation from token-level dynamic decisions to query-level static decisions.Source: §I · p. 2
  • Asynchronous CPU–GPU execution and routing-aware batching address preprocessing bottlenecks, padding overhead, and pipeline stalls in predictive-query workloads.Long-tailed text lengths contribute to padding overhead under conventional batching.Source: §I · p. 2

A. Predictive Query

A predictive query filters tuples using relational constraints and applies a Transformer to their text. QCATS uses the resulting query context to select a unified sparse execution configuration instead of routing each token independently.

A. Predictive Query

  • A predictive query applies a Transformer to text in the subset selected by relational constraints such as WHERE, JOIN, and GROUP BY.The query appends a prediction column to the metadata-constrained subset.Source: A. Predictive Query · p. 3
  • Query context combines explicit relational constraints with statistics over the post-constraint subset, specifying the semantic scope of inference.The context can be materialized into dense features through feature engineering or lightweight encoders.Source: A. Predictive Query · p. 3

B. Core Idea and Goal

  • QCATS uses query-selected tuples’ semantic consistency to apply one execution configuration across the subset, rather than routing each input token independently.The query context can be computed before inference, enabling static execution paths and homogeneous batch construction.Source: B. Core Idea and Goal · p. 3
  • The goal is to reduce accumulated inference cost over the selected tuples while lowering latency and peak memory and preserving predictive performance within a small tolerance.The focus is not autoregressive generation or serving-specific Transformer mechanisms.Source: B. Core Idea and Goal · p. 3

III. QCATS

QCATS converts a fine-tuned dense Transformer into context-aligned expert slices offline, then uses query-time context to select slices for sparse inference. Its expert construction separates broadly useful FFN neurons from semantic-group-specific neurons.

III. QCATS

  • QCATS shifts inference from dense execution to query-adaptive sparse execution through offline expert construction and lightweight query-time routing.The offline library is built once; query-time inference selects slices using relational constraints and metadata statistics.Source: §III · p. 4

A. Stage I: Expert Construction

  • Expert construction profiles FFN activations on representative samples from semantic groups, then separates shared neurons from group-specific neurons.FastText embeddings and K-Means identify groups; samples closest to each centroid are retained to reduce noise.Source: A. Stage I: Expert Construction · p. 4
  • The shared set uses neurons consistently active across semantic groups, while semantic sets use neurons selectively active in one group and suppressed in others.The General Importance Score selects shared neurons; the Semantic Score ranks group-specific neurons against background activity.Source: A. Stage I: Expert Construction · p. 4
  • QCATS reconstructs each FFN expert slice from the shared set and its corresponding semantic set, applying neuron selection across up, gate, and down projections.Each slice therefore retains general linguistic capabilities alongside domain expertise.Source: A. Stage I: Expert Construction · p. 4

B. Stage II: Context-Aware Inference

QCATS extracts relational constraints and metadata statistics to form query context, then uses that context to route inference to preconstructed expert slices. It rewrites query plans and featurizes selected metadata and predicates for routing.

B. Stage II: Context-Aware Inference

  • QCATS extracts relational constraints and metadata statistics from each query, maps the resulting context to expert weights, and performs inference with selected slices.The query-time path comprises query processing, context featurization, expert routing, and sliced model execution.Source: B. Stage II: Context-Aware Inference · p. 5

1) Query Processing:

  • When a neural UDF appears in predicates, QCATS defers its evaluation past lightweight constraints and joins so those operators define the explicit context Φ.The UDF then evaluates on the reduced subset, and routing features come from the same subset that the UDF processes.Source: B. Stage II: Context-Aware Inference · p. 5
  • For GROUP BY queries, QCATS batches each pre-aggregated subgroup independently, assigning dedicated slices to dominant groups and sharing a route for the tail.Dominant groups are identified by cardinality within the query-filtered subset.Source: B. Stage II: Context-Aware Inference · p. 5, 1) Query Processing · p. 5

2) Query Context Featurization:

  • QCATS selects informative metadata columns offline using global relevance and local disambiguation scores, prioritizing metadata that distinguishes textually similar samples with different labels.The local score is based on metadata discrepancies among text-embedding nearest-neighbor conflict pairs.Source: 2) Query Context Featurization · p. 5
  • Schema-agnostic featurization encodes selected-column distributions as implicit metadata features and relevant SQL predicates as explicit query features.Numerical values use discretization-based encoding, categorical fields use frequency-based encoding, and the explicit predicate feature set is empty when no relevant predicates exist.Source: 2) Query Context Featurization · p. 6

3) Context-Aware Expert Routing:

QCATS uses a decoupled context-aware router to select expert combinations before inference. It represents relational metadata and query intent, then converts the resulting context into a bounded Top-K execution plan.

3) Context-Aware Expert Routing:

  • The decoupled, lightweight router maps query context to expert combinations before inference, unlike token-level, layer-wise MoE routing.The design targets structured, non-sequential database content.Source: 3) Context-Aware Expert Routing · p. 6
  • QCATS discards absolute positional encodings and uses Structural Type Embedding to represent permutation-invariant database columns without imposing artificial order.A two-layer Transformer encoder processes implicit metadata features into a global representation, while a decoder attends to metadata using query predicates or a global intent embedding.Source: 3) Context-Aware Expert Routing · p. 6
  • Masked mean pooling converts decoder output into a fixed-length context embedding, and a Top-K gating network activates the most relevant expert slices.The number K controls the trade-off between computational sparsity and retained semantic nuance.Source: 3) Context-Aware Expert Routing · p. 6
  • The selected expert configuration yields a deterministic, hardware-friendly execution path that eliminates redundant parameter computations.The sparse gating vector contains routing weights for the selected experts.Source: 3) Context-Aware Expert Routing · pp. 6–7

4) Sliced Model Execution:

QCATS executes query-selected, preconstructed FFN expert slices to reduce computation and activation memory. Its decoupled training aligns query contexts with experts using dense-model activations, then specializes the experts using fixed routing.

4) Sliced Model Execution:

  • Query-level execution retrieves selected Top-K expert tensors and restricts FFN computation to those slices, reducing intermediate activations and peak VRAM.Unlike token-level MoE, it uses preconstructed slices and avoids runtime expert construction and dynamic control flow.Source: 4) Sliced Model Execution · p. 7
  • QCATS combines shared linguistic neurons with specialized experts, while weakly constrained queries can use weighted multi-expert collaboration.The shared neuron set is described as preserving fundamental linguistic capabilities and a stable performance floor across query contexts.Source: 4) Sliced Model Execution · p. 7

C. Decoupled Training Strategy

  • Jointly optimizing the router and expert library risks unstable training because router learning needs specialized experts while specialization needs consistent routing.The proposed response is a decoupled two-phase strategy: establish routing first, then use it to specialize experts.Source: C. Decoupled Training Strategy · p. 7

1) Phase I: Router Semantic Alignment via Activation Distillation:

  • Phase I uses the fine-tuned dense model’s FFN activations as oracle signals to align query contexts with experts, excluding shared neurons from routing targets.Shared neurons are active across most inputs and provide weak routing signals.Source: 1) Phase I: Router Semantic Alignment… · p. 7
  • Calibration balances relative activation improvement against raw magnitude, preserving strong signals while emphasizing context-specific spikes.The calibration addresses disparities in absolute activation magnitudes across experts; η controls the weight of the original signal.Source: 1) Phase I: Router Semantic Alignment… · p. 7

2) Phase II: Context-Driven Joint Expert Training:

Phase II freezes the context-aware router and trains experts according to its assignments. Optional distillation uses the dense model as a reference to help preserve its representational capacity.

2) Phase II: Context-Driven Joint Expert Training:

  • Phase II freezes the router as a coordinator and trains experts under its fixed routing policy, separating router learning from expert optimization.The stated aim is a stable training environment in which experts specialize in the semantic subspaces identified during Phase I.Source: 2) Phase II: Context-Driven Joint… · p. 7
  • Only selected experts participate in computation and parameter updates, restricting gradients to active experts and adapting them to their designated contexts.The routing weights are fixed during this selective activation scheme.Source: 2) Phase II: Context-Driven Joint… · p. 8
  • A single routing vector is broadcast across FFN layers, creating vertically aligned expert pathways rather than independent layer-wise routing.The paper motivates this global consistency strategy with reported correlations in routing patterns across consecutive Transformer layers.Source: 2) Phase II: Context-Driven Joint… · p. 8
  • Optional dual-granularity distillation aligns student and teacher hidden states and output probabilities to preserve the dense model’s representational capacity.The objectives use hidden-state mean squared error and symmetric KL divergence; the paper says QCATS retains dense-model accuracy under constrained execution budgets.Source: 2) Phase II: Context-Driven Joint… · p. 8

IV. SYSTEM-LEVEL OPTIMIZATION

QCATS combines routing-aware, length-aware batching with an asynchronous CPU–GPU producer-consumer pipeline to address divergent execution paths, padding, and preprocessing bottlenecks. The evaluation spans four predictive-query datasets and asks about system efficiency, effectiveness, ablations, and robustness.

IV. SYSTEM-LEVEL OPTIMIZATION

  • Mixed query contexts can diverge in expert paths, long-tailed text lengths increase padding, and faster GPU inference can make CPU preprocessing a system bottleneck.These are the system-level inefficiencies the proposed optimizations target.Source: §IV · p. 8
  • An asynchronous producer-consumer pipeline overlaps CPU preparation of routing-aware batches with GPU sparse inference through a threadsafe prefetch buffer.Hierarchical bucketing groups inputs first by routing outcomes and then by text length, with batch sizes adjusted to sequence length and GPU budget.Source: §IV · p. 8
  • Routing-consistent, length-homogeneous batches reduce padding waste, while the pipeline overlaps preprocessing and inference so throughput is primarily model-execution-bound.The paper states that this combination eliminates pipeline stalls.Source: §IV · p. 8, 1) Datasets and Tasks · p. 9

V. EXPERIMENTS

  • The experiments frame questions about latency and GPU memory, accuracy under parameter reduction, component contributions, sensitivity, and transfer robustness.The setup uses four real-world datasets, including Amazon Review, Kickstarter, and OSHA in the supplied passages.Source: §V · p. 9, 1) Datasets and Tasks · p. 9

1) Datasets and Tasks:

The evaluation uses generated predictive-query workloads across four datasets and compares QCATS with dense and sparse baselines. The reported results describe lower latency and peak memory, with memory savings attributed mainly to reduced activation memory and padding.

1) Datasets and Tasks:

  • CVE (Cybersecurity) is used to predict CVSS severity from vulnerability descriptions, and the dataset contains 11 structured metadata columns.Source: A. Experimental Setup · p. 9

2) Query Workload Generation:

  • The workload generator creates 50 SQL templates per dataset, with one to three predicates and options for grouping, sampled joins, and non-context filters.Context-column predicates use categorical equality or numerical ranges, combined only with AND.Source: 2) Query Workload Generation · p. 9

3) Baselines and Implementation:

  • Comparisons include dense BERT and Qwen models, MoEfication and MoEBERT where available, and QCATS; sparse methods activate 25% of FFN parameters.For Qwen, the passage compares Qwen-base directly with QCATS because no standard MoEfication implementation exists.Source: 3) Baselines and Implementation · p. 9

2) Peak GPU Memory (VRAM) Consumption:

  • QCATS substantially reduces peak GPU memory, mainly by allocating memory for activated expert subsets and reducing padding through bucketing.The stated static weights occupy about 220 MB for BERT and 1.2 GB for Qwen; the passage attributes the savings mainly to activation memory.Source: Figure 9 · p. 10, 2) Peak GPU Memory (VRAM) Consumption · p. 10

4) Evaluation Settings and Metrics:

  • The evaluation executes about 600 queries per dataset, retrieves one million rows in total, and measures end-to-end latency, peak GPU memory, accuracy, and weighted F1.Average batch size is aligned across methods; effectiveness is measured on an aggregated test set formed by merging query results across workloads.Source: 4) Evaluation Settings and Metrics · p. 10

B. System Efficiency (RQ1)

1) End-to-End Latency:

  • QCATS achieves end-to-end latency speedup through query-level routing, with pipelining further amplifying the gain by overlapping CPU preprocessing and GPU inference.MoEfication’s logical sparsity does not necessarily translate into efficient physical GPU execution.Source: 1) End-to-End Latency · p. 10

C. System Effectiveness (RQ2)

QCATS preserves predictive performance while reducing inference cost through query-level routing and system optimizations. The evaluations examine its effectiveness, component contributions, and sensitivity to design choices.

C. System Effectiveness (RQ2)

  • QCATS preserves task accuracy while activating a subset of FFN parameters, consistently rivaling or outperforming dense baselines and matching MoEBERT.The comparison covers accuracy and weighted F1-score across four datasets.Source: C. System Effectiveness (RQ2) · p. 10, Table III · p. 11

1) Latency Breakdown and Bottleneck Analysis:

  • For 100,000-row queries, QCATS adds negligible router overhead, while sparsification and query-level routing reduce inference latency and hierarchical bucketing reduces padding.With GPU inference substantially shortened, CPU preprocessing remains a non-negligible bottleneck for lightweight models such as BERT.Source: 1) Latency Breakdown and Bottleneck… · pp. 10–11, Table II · p. 11

2) Context-Aware Router and Knowledge Distillation:

  • Removing routing causes a clear performance drop, whereas routed variants remain comparable to the dense baseline; without distillation, a QCATS variant surpasses the teacher on Qwen-CVE.The ablation indicates routing is critical for expert specialization, while distillation is not the decisive factor behind the performance gain.Source: 2) Context-Aware Router and Knowledge… · p. 11, Figure 11 · p. 11

1) Expert granularity:

  • At a fixed 25% activation ratio, expert granularity leaves accuracy stable, but finer granularity increases latency; N4K1 is selected for throughput without compromising predictive performance.For context columns, small M weakens context alignment and routing accuracy, while large M adds redundant attributes and preprocessing overhead.Source: 1) Expert granularity · p. 11, 2) Context Columns · p. 11, Figure 12 · p. 11

2) Context Columns:

QCATS reduces latency across diverse SQL workloads while limiting accuracy impact, and its query-based slicing remains comparable to dense baselines in a similar-domain zero-shot transfer. The paper positions query-level Transformer execution as a way to connect database context with model internals.

3) QCATS under diverse SQL workloads:

4) Case Study: Transferability to Similar Domains:

  • In zero-shot transfer from Amazon to Flipkart, QCATS achieves performance comparable to dense baselines using Product Category and Product Price as routing contexts.The transferability comparison is reported in Table IV.Source: 4) Case Study: Transferability to… · p. 12

VI. RELATED WORK

  • QCATS differs from prior database-aware approaches by using query context to deterministically select FFN neuron slices within a single Transformer backbone for text prediction.The related-work discussion distinguishes this from instance-level adaptation, resource-budget slicing, and expert or model selection for structured-data analytics.Source: §VI · p. 12

VII. CONCLUSION

  • The conclusion reports that query-level conditional execution aligns sparse neural inference with database semantics while improving throughput and reducing memory consumption against dense baselines.The authors identify larger backbones, more realistic workloads, and distribution shifts as future evaluation directions.Source: §VII · pp. 12–13

APPENDIX

The appendix details a query-context activation analysis and implementation choices for QCATS. Its analysis finds that neuron activation patterns vary with query context, motivating context-aware expert construction.

QUERY CONTEXT ANALYSIS

  • The activation benchmark uses 16 Kickstarter queries, samples 200 tuples per query, and compares overlap among the top 768 neurons under a 25% activation setting.The queries comprise eight seeds and eight perturbed variants; the overlap matrix shows greater similarity for semantically related query pairs and divergence among unrelated groups.Source: QUERY CONTEXT ANALYSIS · pp. 14–15, Figure 15 · p. 15

HYPERPARAMETER SETTINGS

  • Expert construction partitions data into four semantic groups and profiles 500 representative instances per group; inference uses N4K1, activating one of four experts.The configuration fixes activation at 25%; half the neurons are shared and half context-specific, motivated by approximately 50% overlap across contexts.Source: HYPERPARAMETER SETTINGS · p. 15
  • The feature-selection stage chooses the top three metadata columns, while the router uses a 512-dimensional Transformer block and a three-layer MLP.The paper notes that the optimal number of context columns may vary across datasets.Source: HYPERPARAMETER SETTINGS · p. 15
  • Router training uses η = 0.02 and τ = 1.0 for semantic alignment, followed by joint expert training with a distillation-loss weight α = 5.0.The calibration coefficient is chosen to align selected expert indices with expert-score rankings while preventing under-utilization of lower-scoring experts.Source: HYPERPARAMETER SETTINGS · p. 15
Loading 2610.09894v1…