Source-linked AI summary
Bang for the Buck: Vector Search on Cloud CPUs
Leonardo Kuffo, Peter Boncz
TL;DR
Cloud vector-database users lack benchmarks to guide CPU selection across architectures and microarchitectures. The paper experimentally compares cloud CPUs across vector-search scenarios, finding that careful microarchitecture choice can yield about 3x more queries per dollar in AWS.
Problem
Cloud vector-database users lack benchmarks comparing vector-search performance across CPU architectures and microarchitectures, making instance selection difficult.
Source: §1 · p. 1Method
The study experimentally evaluates cloud CPUs using QPS and QP$ across vector-search scenarios and microarchitectures.
Source: §1 · p. 1Results
CPU and search-algorithm differences can yield ≈3x more QP$ with careful microarchitecture choice; Graviton3 and Zen4 offer good value in most AWS scenarios.
Source: §6 · p. 7Takeaways & Limitations
Users should benchmark their use case because the study covers HNSW, IVF, and full scans, while other indexes, quantizers, and distance metrics exist.
Source: §5 · p. 7The evaluation focuses on HNSW, IVF, and full scans at different quantization levels, rather than the wider variety of indexes, quantization techniques, and distance metrics.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Vector databases have emerged as a new type of systems that support efficient querying of high-dimensional vectors. Many of these offer their database as a service in the cloud. However, the variety of available CPUs and the lack of vector search benchmarks across CPUs make it difficult for users to choose one. In this study, we show that CPU microarchitectures available in the cloud perform significantly differently across vector search scenarios. For instance, in an IVF index on float32 vectors, AMD's Zen4 gives almost 3x more queries per second (QPS) compared to Intel's Sapphire Rapids, but for HNSW indexes, the tables turn. However, when looking at the number of queries per dollar (QP$), Graviton3 is the best option for most indexes and quantization settings, even over Graviton4 (Table 1). With this work, we hope to guide users in getting the best "bang for the buck" when deploying vector search systems.
1 INTRODUCTION
Vector databases have grown with high-dimensional vector data, and their central task—similarity search at scale—often relies on approximate indexes and quantization. The paper addresses the lack of cross-CPU vector-search benchmarks and finds that the best-performing and best-value CPUs depend on the search scenario.
1 INTRODUCTION
- Vector databases have emerged as systems for storing, managing, and querying high-dimensional vectors, with adoption also appearing in established database systems.Examples include native vector capabilities and extensions.Source: §1 · p. 1
- At scale, exact vector similarity search requires substantial computation and storage, motivating approximate indexes, quantization, and distance-evaluation optimizations.These approaches trade exactness for answers considered good enough in many applications.Source: §1 · p. 1
- For IVF, Zen4 delivers almost 3x QPS versus Sapphire Rapids; HNSW reverses the ranking, while Graviton3 leads QP$.Table 1 compares QPS and QP$ across AWS instances and vector-search scenarios.Source: Table 1 · p. 1, §1 · p. 1
- Cloud SaaS offerings raise the question of which instance best suits vector search, but benchmarks comparing CPU architectures and microarchitectures are lacking.Existing examples cited include vendor benchmarks on Intel CPUs and USearch benchmarks on AWS Graviton3.Source: §1 · p. 1
- The study evaluates cloud CPUs using QPS and QP$ across vector-search scenarios, finding that data-access patterns and cache performance matter alongside SIMD capabilities.Many scenarios are memory-bound.Source: §1 · p. 1
2 PRELIMINARIES
The preliminaries define approximate vector search and introduce indexes, quantization, and SIMD as tools for searching high-dimensional collections. They describe the trade-offs and implementation considerations that shape these techniques.
2.1 Approximate Vector Similarity Search
- VSS seeks the k most similar vectors under δ; approximate search returns R-hat, whose quality is measured by overlap with exact results R.Computing distances to every vector is costly, motivating approximate indexes and quantization.Source: §2.1 · p. 2
2.2 Approximate Indexes
- Approximate indexes evaluate fewer vectors while seeking high recall; HNSW navigates layered graphs, whereas IVF probes nearest-centroid buckets and trades speed for recall.IVF uses buckets and typically sets the number of centroids in the order of √n; HNSW’s upper layers enable fast traversal before local search below.Source: §2.2 · p. 2
2.3 Quantization Techniques
- BQ maps float32 values to bits, SQ codes them as integers, and PQ encodes subvectors with codebook centroids, allowing variable compression but requiring decoding before distance calculations.More aggressive quantization reduces storage and can increase throughput, but can reduce recall; the figure overviews the techniques.Source: §2.3 · p. 2, Figure 2 · p. 2
2.4 Single Instruction Multiple Data (SIMD)
- SIMD speeds distance kernels, but wider registers alone do not ensure faster database workloads, and some quantized formats require costly alignment.Libraries may use specialized quantized kernels or decode vectors to float32; the latter avoids complex code but does not match symmetric-kernel performance.Source: §2.4 · p. 3
3 BANG FOR THE BUCK
Cloud vector-search performance varies by microarchitecture, search algorithm, and quantization setting. The experiments compare QPS and QP$ across cloud CPUs, and explain the differences through memory access patterns and distance-kernel efficiency.
3.1 IVF
- Zen4 leads QPS on unquantized IVF, while Graviton3 ties it in QP$; at the highest recall, Zen4 and Intel perform well on SQ, but Gravitons underperform there and shine on 1-bit vectors.FAISS's scalar decoding on ARM for SQ contrasts with its specialized NEON binary-quantization kernel.Source: Figure 3 · p. 4, §3.1 · p. 4, Table 4 · p. 4
3.2 HNSW
- On unquantized HNSW, Intel Z leads QPS but not QP$, where Zen3 and Graviton3 perform best on high-dimensional vectors; across quantized settings, QP$ leaders vary by type and dimensionality.Intel Z has the highest QPS in nearly all quantized settings, while the architecture gap is less evident than in IVF.Source: Table 5 · p. 4, Figure 4 · p. 5, §3.2 · p. 5
3.3 Full Scans
- Full-scan QP$ leadership varies with representation and library: Zen4 and Graviton3 lead for float32, Gravitons lead for float16 and USearch SQ, and Intel leads for PQ.For 8-bit vectors, Gravitons lead in USearch while Zen4 and Intel perform well across FAISS SQ settings; Gravitons underperform in FAISS SQ.Source: §3.3 · p. 5, Table 6 · p. 5
3.4 Why the differences across microarchs?
- SIMD specialization alone does not predict the winner: data-access patterns and distance-kernel efficiency both shape performance.SPR and Zen4 each have 1024-bit FP execution throughput per cycle, while Gravitons have 512 bits; the effective throughput depends on the kernel and instruction mix.Source: §3.4 · pp. 5–6
- Sequential access in IVF and full scans disadvantages CPUs with higher L3/DRAM latency and lower read bandwidth; as quantization narrows vectors, the performance gap between SPR and Zen4 closes.For data larger than L3, sequential-access latency on SPR is almost 2x that of the Zens and almost 4x that of the Gravitons.Source: §3.4 · p. 6
- HNSW accesses fewer vectors and caches upper-layer vectors efficiently, aligning with SPR's highest QPS across HNSW scenarios; PQ also benefits from efficient codebook caching.SPR's larger caches, higher L1/L2 read bandwidth, and higher SIMD LOAD throughput are identified as advantages for these access patterns.Source: §3.4 · p. 6
- For float32 full scans, SPR is 10% faster than Zen4 when data fits in L2, but its performance degrades after data spills to L3 and DRAM.At DRAM size, SPR delivers 30% less performance than Zen3 and Graviton4, and 2x less than Zen4.Source: §3.4 · p. 6
- Graviton3's 256-bit SVE registers versus Graviton4's 128-bit registers help explain why Graviton3 can outperform it: with SVE, Graviton3 is 31% faster, while with NEON, Graviton4 is 10% faster.The study attributes Graviton3's advantage to register width and LOAD throughput; SVE is 37% faster than NEON on Graviton3 in float32 full scans.Source: §3.4 · p. 6
4 BANG FOR THE BUCK: TAKEWAYS
Graviton3 gives the best queries-per-dollar value, even over Graviton4, while Zen4 remains a strong option for IVF and full scans. The CPU rankings also reflect price and workload-specific strengths.
4 BANG FOR THE BUCK: TAKEWAYS
- The ranking uses five tiers for QPS and QP$ on OpenAI/1536, then aggregates scores with +1 for each (+) and -1 for each (-).Source: §4 · p. 7
- Graviton3 gives the best QP$ overall, even over Graviton4, and loses ground in the scoring system only because FAISS lacks symmetric kernels.Its strengths include varied SIMD capabilities, high read throughput, low sequential memory latency at L3/DRAM, and low price.Source: §4 · p. 7
- Sapphire Rapids has the lowest QP$ score despite the best HNSW QPS score, because its price lowers its value.Source: §4 · p. 7
- Zen4 is a solid choice for IVF and full scans, especially with float32 and bfloat vectors, while Zen3 benefits from low price and low memory-access latencies but excels nowhere.Source: §4 · p. 7
5 DISCUSSION
The discussion emphasizes that cloud CPU choice can be tailored to the search algorithm and that the study’s conclusions have defined workload and hardware boundaries. It encourages data-driven benchmarking and attention to data-access patterns and storage design.
5 DISCUSSION
- Because cloud storage and compute are decoupled, users can switch microarchitectures by search algorithm, potentially saving costs in serverless vector search where each millisecond counts toward billing.Source: §5 · p. 7
- The study covers HNSW, IVF, and full scans across quantization levels, not hybrid indexes, residual quantizers, or cosine and inner-product metrics.The authors encourage data-driven benchmarks for each use case.Source: §5 · p. 7
- The tested microarchitectures will become outdated; the discussion also stresses data-access patterns, storage designs, and SIMD-optimized implementations for future algorithms.It notes Emerald Rapids and Zen5 as successors to Sapphire Rapids and Zen4, respectively.Source: §5 · p. 7
6 CONCLUSIONS AND FUTURE WORK
The conclusion reports that CPU value depends on more than SIMD capabilities, and that careful microarchitecture choice can yield substantially more queries per dollar. Future work includes broader workload, GPU, and VecDB benchmarking.
6 CONCLUSIONS AND FUTURE WORK
- ≈3x more QP$ is possible through careful microarchitecture choice, with data-access patterns and vector size also affecting performance beyond CPU SIMD capabilities.In AWS, Graviton3 (r7g) and Zen4 (r7a) offer good value in most vector-search scenarios.Source: §6 · p. 7
- Future benchmarks could cover interleaved updates and queries, multithreaded batch queries, index construction, GPU microarchitectures, and QP$ comparisons across VecDBs.The authors call for more established benchmarks for vector research.Source: §6 · p. 7