PaperlayerView exact revision on arXiv

Source-linked AI summary

ByteX: A Unified AI Search Engine at ByteDance

Yao Tian, Yuncheng Lu, Liyao Xiong, Yuming Xu, Hao Zhang, Weichen Zhao, Xi Zhao, Bo Kuang, Dongyu Wang, Jiehui Li, Yakun Li, Lei Zhang

arXiv:2608.30607v2cs.DB

TL;DR

ByteX addresses the ingestion-memory and all-in-memory serving constraints of trillion-vector AI search within an OpenSearch-compatible platform. It combines SymRaBitQ-based quantization-aware indexing with adaptive memory–SSD storage and fine-grained caching. Across benchmarks and deployments, it reports lower construction memory, faster building, and lower storage cost while preserving recall and latency.

  • Problem

    Trillion-scale AI search makes graph construction memory-intensive and entirely memory-resident vector serving economically infeasible while requiring hybrid retrieval and OpenSearch compatibility.

  • Method

    ByteX combines a SymRaBitQ-based quantization-aware vector kernel with adaptive memory–SSD storage, record-level caching, and unified hybrid query execution.

  • Results

    Up to 80% lower graph-build peak memory, 60% faster build time, and 7.3× lower storage cost were achieved while preserving recall and latency.

  • Takeaways & Limitations

    ByteX supports PB-scale ingestion-heavy and latency-sensitive online serving workloads within one OpenSearch-compatible operational ecosystem.

  • Takeaways & Limitations

    ByteX must support differing memory-backed and SSD-backed deployment conditions, spanning 105–212 TiB memory and 2–3 PB SSD workloads.

Abstract

from arXiv · show

Since 2016, ByteX has been the foundation of ByteDance's search infrastructure, scaling to more than 7,000 clusters and 300 PB of indexed data. Driven by the demands of AI workloads, ByteX has evolved from a text search engine into a unified AI search system supporting vector retrieval, lexical matching, and predicate filtering. Its largest deployment indexes nearly one trillion high-dimensional vectors. This scale exposes two central bottlenecks in AI-era retrieval: memory-intensive graph-index construction under sustained ingestion, and the prohibitive cost of keeping vector indexes entirely in memory. ByteX addresses these bottlenecks with two techniques. First, it introduces a quantization-aware vector kernel based on SymRaBitQ, a new symmetric quantization scheme with tight theoretical guarantees that allows index construction to run directly in the quantized space accurately and efficiently without retaining a copy of full-precision vectors. Second, it provides a hybrid storage engine that supports memory-resident, hybrid, and SSD-resident deployments, with fine-grained record-level caching to trade memory for latency under operational control. On large-scale benchmarks, ByteX improves throughput by up to 3x, reduces indexing memory by 80%, and lowers operating cost by 86% compared with prior systems, while supporting trillion-vector scale, write-heavy or latency-sensitive workloads in production.

1 Introduction

ByteX frames trillion-scale AI search as a system-level challenge spanning ingestion, storage, serving, and hybrid query execution. It addresses these constraints within an OpenSearch-compatible platform using quantization-aware indexing and adaptive memory–SSD storage.

  • 7,000+ clusters and 300+ PB of storage have driven ByteX from text search toward AI search with vector, lexical, predicate, and reranking operators.
  • Trillion-scale vector growth is driven by finer retrieval granularity, higher embedding dimensionality, and longer retention periods.
  • A 2048-dimensional FP32 vector occupies 8 KB, while graph indexing 1 million vectors requires at least 8 GB of memory and over one hour.
  • At 10+ billion vectors ingested daily and peaks above 700K vectors/s, retaining full-precision vectors during construction can consume tens of terabytes.
  • ByteX integrates SymRaBitQ into indexing to construct graphs directly in quantized space, eliminating full-precision copies and reducing construction cost.
  • ByteX combines memory-resident, hybrid, and SSD-resident execution with record-level caching while preserving OpenSearch-compatible hybrid retrieval and operational tooling.

2 Production Workload & Challenges

ByteX production workloads combine hybrid retrieval, online latency requirements, OpenSearch-native execution, and deployments spanning memory to SSD. At trillion-vector scale, ingestion and serving share DRAM, creating coupled memory and cache-efficiency challenges.

  • Customers adopted ByteX as an OpenSearch-compatible platform before incrementally enabling vector search across existing queries and workflows.
  • Production workloads: Hybrid workloads combine vector similarity with lexical, scalar, and domain-specific predicates, followed by ranking and LLM-based recall.Their cost includes materialization, filtering, merging, and downstream reranking, not only ANN traversal.
  • Production workloads: ByteX must preserve OpenSearch-native ingestion, merging, replication, recovery, monitoring, DSL, filtering, ranking, and shard-level execution while supporting mixed storage tiers.Production deployments span 105–212 TiB memory-backed workloads and 2–3 PB SSD-backed workloads.
  • Challenges: At trillion-vector scale, DRAM is the shared bottleneck between ingestion and serving.Naïve graph construction can peak at 27 TB against a 13.8 TB DRAM budget, before reserving memory for serving caches.
  • Challenges: Existing disk-based ANN systems reduce serving-time DRAM but leave the ingestion–serving memory conflict unresolved.Their asymmetric quantization still materializes full-precision vectors during graph construction, while generic page caches are too coarse for graph search.
  • Challenges: ByteX converts DRAM freed during ingestion into higher serving-cache efficiency while preserving OpenSearch compatibility and rich predicates, filtering, and ranking.

3 System Overview

ByteX preserves OpenSearch interfaces while redesigning vector representation, storage placement, and hybrid query execution for production-scale vectors. Its architecture supports the same index across DRAM, hybrid memory–SSD, and SSD-resident deployments.

  • ByteX preserves clients, APIs, SDKs, ingestion pipelines, dashboards, and lifecycle tools so existing deployments can adopt it without rewriting workflows.
  • ByteX treats vectors as first-class citizens by co-designing representation, storage layout, and hybrid query execution within OpenSearch.
  • The redesign combines a quantization-aware vector kernel, an adaptive storage engine, and placement-independent segment serving.The same index can serve from DRAM, hybrid memory–SSD, or SSD-resident storage without reindexing.
  • ByteX executes BM25, vector search, predicates, range filters, score fusion, and reranking together as one distributed query plan.DRAM segments use in-memory traversal, while SSD segments use record-level buffering and asynchronous traversal to overlap I/O with distance computation.

4 Quantization-Aware Vector Kernel

ByteX introduces SymRaBitQ to support both data-to-query and data-to-data distance evaluation in quantized space. The same primitive serves graph and IVF construction, merging, and search while reducing construction cost.

  • SymRaBitQ: SymRaBitQ replaces asymmetric quantization with a symmetric scheme in which both operands can be represented by compact codes.This unifies data-to-query and data-to-data evaluation in one quantized-space primitive.
  • SymRaBitQ: SymRaBitQ is the distance primitive used by the ByteX vector kernel for encoded residual vectors.Stored vectors and queries are represented relative to a centroid when used in IVF or partitioned graph indexes.
  • Estimator: For data-to-data comparisons, both denominator terms are precomputed; for query-to-data comparisons, only the query term is computed online once.
  • Estimator: The estimator’s numerator is computed directly from integer codes, with 1-bit computation reducible to sign agreement and bit operations with popcount.ByteX uses a 5-bit construction code.
  • System implications: SymRaBitQ extends quantization benefits from query-time estimation to graph construction and segment merging by eliminating raw-vector accesses for data-to-data computations.
  • Results: B=5 preserves more than 98% distance-estimation accuracy with less than 0.01 recall loss under the same search parameters.On GIST-1M, SymRaBitQ reduces peak build memory by 80% and total build time by 60%.
  • Index primitives: ByteX applies SymRaBitQ to IVF list construction, list scanning, and k-means clustering in quantized space.

5 Vector Storage Engine

ByteX manages vector segments across memory-only, hybrid memory–disk, and disk-resident paths. Fine-grained record caching and asynchronous execution reduce avoidable SSD traffic and overlap remaining I/O with search computation.

  • ByteX makes newly written segments searchable at refresh, bounds query fanout through merges, upgrades indexes from scan to IVF to graph search, and selects storage placement per segment.
  • ByteX combines fine-grained caching with asynchronous graph traversal to optimize disk-resident serving.
  • Fine-Grained Record Cache: Record-level caching stores compact vector payloads and bounded neighbor lists, avoiding the incidental bytes loaded by page-granular caching.A vertex-ID mapping array provides O(1) record access and tracks disk residency, cache state, and offsets.
  • Asynchronous execution: Each SSD-resident ANN search runs as a coroutine that yields after submitting a required read when no ready candidate can progress.
  • Asynchronous execution: Coroutine scheduling advances other ready searches while suspended reads complete, overlapping SSD latency with vector-search computation.The record cache reduces avoidable SSD traffic, while scheduling hides remaining read latency.

6 Hybrid Query Engine

ByteX integrates vector retrieval with lexical matching, structured predicates, ranking, and reranking in an OpenSearch-compatible distributed query engine. It supports adaptive filtering modes for different predicate selectivities.

  • ByteX’s coordinator parses the DSL, routes requests to shards, and composes vector operators with the broader search pipeline.
  • Filtered Vector Search: ByteX supports pre-filtering, post-filtering, and in-filtering for structured predicates such as tenants, categories, tags, timestamps, and access policies.Pre-filtering suits highly selective conditions, while post-filtering suits low-selectivity predicates.

7 Experiment

Experiments evaluate ByteX across indexing cost, high-recall serving, complex retrieval, operating cost, SSD runtime, and scaling. Results show substantial gains in quantized construction, throughput, memory efficiency, caching, asynchronous execution, and cost control.

  • Indexing: 80%: SymRaBitQ cuts peak memory from 3.75 GB to 0.75 GB on GIST 1M while reducing build time from 587 s to 237 s (60%).On Cohere 100M, build time falls from 23,826 s to 12,496 s (48%), with peak memory dropping from 453 GB to 90 GB (80%).
  • Indexing: 49%: SymRaBitQ-enhanced IVF reduces GIST 1M peak memory from 3.7 GB to 1.9 GB while cutting build time from 176 s to 129 s (27%).
  • Vector Search Performance: 21,445 QPS: ByteX reaches around 90% recall@10 on the in-house 100M dataset, outperforming Milvus by 9.5×, PostgreSQL by 5.5×, and Elasticsearch by 2.3×.On Cohere, ByteX reaches 23,013 QPS at around 92% recall@10.
  • Storage Runtime Analysis: 97.7%: ByteX’s record-level cache hit rate on GIST reduces SSD accesses to 51.90 per query versus 80.9 for DiskANN.Each query visits 105.99 graph records on average.
  • Storage Runtime Analysis: 15K QPS: At 32 threads, ByteX-async delivers roughly 1.8× ByteX-sync throughput and 2× DiskANN throughput, with approximately 1.7 ms average latency.ByteX-sync and DiskANN reach about 3.5 ms and 4 ms, respectively.

8 Lessons

ByteX’s lessons emphasize innovating within compatibility boundaries and quantizing recurring index-building work, while integrating retrieval operators into one distributed plan.

  • Lesson 1: Innovate within compatibility boundary: ByteX preserves OpenSearch-facing behavior while redesigning the vector kernel, storage, and index lifecycle underneath.This keeps existing APIs, dashboards, alerts, access controls, and operational playbooks usable.
  • Lesson 2: Quantize the build path, not just the query path: SymRaBitQ quantizes graph construction and merge, reducing recurring memory and computation competition between ingestion and serving.
  • One distributed plan executes structured predicates, lexical matching, and reranking without materializing large intermediate results.The integrated plan enables end-to-end scheduling unavailable to a multi-service pipeline.

9 Related Work

Vector search methods include LSH-based, graph-based, and quantization-based approaches. Modern vector databases commonly use graph-based methods such as HNSW and DiskANN for approximate nearest-neighbor search.

  • Vector Search Algorithms: Approximate nearest-neighbor methods are broadly categorized as LSH-based, graph-based, or quantization-based.
  • Vector Search Algorithms: HNSW and DiskANN build approximate proximity graphs and are highly effective in modern vector databases.

10 Conclusion

ByteX evolves production search into an OpenSearch-compatible AI search engine by combining quantization-aware vector processing, adaptive storage, and unified hybrid query execution. Its design targets memory-efficient ingestion, flexible vector placement, and integrated retrieval workloads at production scale.

  • ByteX combines a quantization-aware vector kernel, adaptive storage engine, and hybrid query engine for production-scale AI search.The system preserves OpenSearch-compatible APIs while supporting vector similarity, lexical relevance, structured predicates, score fusion, and reranking.
  • Write and indexing path: ByteX upgrades segment representations progressively from direct scan to IVF and then graph indexes as segments mature.Graph-indexed segments are constructed and merged in quantized space, then placed in memory-resident, hybrid memory–SSD, or SSD-resident modes.
  • Read and serving path: ByteX jointly executes BM25 retrieval, vector search, structured predicates, score fusion, and reranking within distributed OpenSearch-style query plans.Applications can combine these operations without issuing separate retrieval requests or merging results across services.
  • SymRaBitQ lets distance estimation, graph construction, and segment merge operate in the quantized space without repeated full-precision materialization.Its symmetric estimator supports both data-to-query and quantized data-to-data comparisons, replacing distance calculations throughout graph construction with SymDist_B.

F The Unbiasedness of SymRaBitQ

The analysis establishes that SymRaBitQ is approximately unbiased in typical high-dimensional settings. Empirical distance estimates also track real distances closely across the evaluated bit settings, with limited deviation at low distances.

  • The unbiasedness argument uses concentration properties of normalized Gaussian Gram matrices to analyze the random orthogonal transformation.The proof begins by recalling a standard concentration fact for Gaussian matrices and treating the orthogonal matrix as a normalized Gaussian matrix.
  • SymRaBitQ is approximately unbiased in typical high-dimensional settings because unrelated vectors are usually nearly orthogonal.When t = 0, the estimator is unbiased; for small nonzero t, the remaining error is negligible as t approaches 0.
  • Across B = 1 and B = 5, estimated distances concentrate around the real-distance y = x line, with slope near 1 and intercept near 0.The remaining deviation is limited in the low-distance region.

G Controlled Benchmark Configuration

The controlled benchmark compares distributed and single-node systems under specified resource, shard, replica, and index configurations. Its cost model charges the serving footprint that physically stores or serves the in-house dataset.

  • Table 5 specifies resource and index configurations for Elasticsearch, ByteX, PostgreSQL, Milvus, and PostgreSQL/VectorChord.Distributed systems use one shard with three copies for Wiki and GIST, and three shards without replicas for Cohere and the in-house dataset.
  • The operating-cost comparison charges only the serving tier that physically stores or serves the in-house dataset.Auxiliary components are excluded because their overhead is shared or amortized and does not scale directly with dataset size.
  • ByteX Disk-only uses 3 × 2C16GiB data nodes because only minimal index structures remain in memory while the remaining data reside on disk.All charged nodes use the same commercial cloud instance family unless otherwise stated.

I Additional Latency Results

Figures 20–23 provide additional latency–recall curves under the same controlled benchmark setting. They cover concurrent and serial execution with average, P95, and P99 latency views.

  • The additional results include concurrent average latency, concurrent P95 latency, serial P95 latency, and serial P99 latency.Each figure presents a latency–recall trade-off under its named execution and percentile condition.
Loading 2608.30607v2…