PaperlayerView exact revision on arXiv

Source-linked AI summary

A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware

Ayushman Bhattacharya, Nihal Gazi

arXiv:2609.05463v1cs.DCcs.PF

TL;DR

OreoLook addresses redundant work and lost context in low-cost, multi-turn web search by combining session persistence, semantic query deduplication, and cross-session URL embedding reuse. Its local infrastructure runs on commodity CPU hardware while remote provider inference synthesizes answers. In the evaluated production snapshot, the system reported an 89.3% aggregate Redis keyspace hit rate, while semantic near-duplicate hits avoided 3–8 seconds of wall-clock time per successful call.

  • Problem

    Growing multi-turn web-search usage caused lost session context, repeated full-pipeline work for rephrased queries, and redundant URL embeddings across sessions.

  • Method

    The paper presents a three-layer caching architecture combining a Redis session window with compressed disk overflow, a per-session semantic query cache, and a global URL embedding cache.

  • Results

    89.3% aggregate Redis keyspace hit rate was reported, and successful semantic cache hits avoided 3–8 seconds of wall-clock time per full pipeline call.

  • Takeaways & Limitations

    Bounded hot context, disk-backed archival, semantic deduplication, and cross-session embedding reuse support long-running sessions without retaining full histories in memory.

  • Takeaways & Limitations

    The evaluation uses one historical production snapshot and hardware configuration, and Redis keyspace hits are not query-level cache hits.

Abstract

from arXiv · show

AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers grounded in live web results. We developed OreoLook (formerly lixSearch), an open-source answer engine using automated browser agents and provider-routed LLM inference. Its local search, caching, session-management, and embedding stack runs on commodity CPU hardware; answer synthesis is performed by a remote inference provider. As usage grew, sessions lost context, equivalent queries triggered redundant work, and URLs were repeatedly embedded across sessions. We present a three-layer caching architecture: (1) a Session Context Window maintaining a rolling window of recent messages in Redis with automatic overflow to Huffman-compressed disk archives; (2) a Semantic Query Cache catches rephrasings via cosine similarity on embedding vectors, eliminating redundant LLM invocations; and (3) a URL Embedding Cache that deduplicates embedding computations across sessions. Deployed on a single 8-vCPU Intel Cascade Lake server (2 GHz, 32 GB RAM) running 30 Hypercorn worker processes across three containerized replicas, the evaluated system reported an 89.3% aggregate Redis keyspace hit rate with 0.1 ms read latency and just 1.38 MB of memory overhead. A background LRU eviction daemon migrates idle sessions from Redis to disk and re-hydrates them on demand, enabling conversations that can be resumed hours or days later under the configured retention policy.

I. INTRODUCTION

OreoLook began as a low-cost browser-agent and provider-routed LLM search pipeline, but growing multi-turn usage exposed problems with context retention, repeated rephrased queries, and duplicated URL embeddings. The resulting three-layer architecture targets these problems with session context, semantic query caching, and cross-session embedding reuse.

  • B. The First Version: Raw Search + LLM: OreoLook used automated browser agents and provider-routed LLM inference instead of a proprietary search API, reducing per-query cost to approximately $0.02.The initial system searched directly through headless browsers, fetched pages, and sent content to remote LLM inference.
  • B. The First Version: Raw Search + LLM: Three production problems motivated the architecture: lost conversational context, redundant full-pipeline execution for rephrased queries, and repeated embedding of popular URLs across sessions.These failures threatened the cost advantage as conversations lengthened and usage grew.
  • I. INTRODUCTION: OreoLook’s local search, caching, and embedding infrastructure runs on commodity hardware while answer synthesis remains provider-routed.The implementation separates local infrastructure from remote inference.
  • I. INTRODUCTION: The system was designed as a lightweight unified alternative because existing tools addressed individual concerns rather than all three together.The paper positions the coordinator as an integration layer spanning session persistence, semantic deduplication, and embedding reuse.
  • I. INTRODUCTION: The architecture unifies a Redis-backed session window with Huffman disk overflow, a semantic query cache, and a global URL embedding cache.The three layers address context persistence, query rephrasing, and cross-session embedding reuse respectively.

II. RELATED WORK

Prior systems address conversation memory, semantic caching, Redis-backed caching, or archival separately, whereas OreoLook presents a unified lightweight architecture spanning session persistence, semantic deduplication, and embedding reuse.

  • No single prior system addresses session persistence, semantic deduplication, and embedding reuse in one unified lightweight package.
  • Unlike in-process memory modules from LangChain and LlamaIndex, this system persists conversation state in Redis and archives it to disk across restarts and replicas.
  • GPTCache provides semantic LLM caching but lacks per-session isolation, whereas this design targets multi-user search assistants with session-scoped caching.
  • Redis-backed frameworks typically provide flat key-value caching, while this system separates logical databases by TTL, data format, and caching concern.
  • The system uses deterministic LRU eviction rather than MemGPT’s autonomous LLM-driven memory management for moving conversation data between hot and cold storage.
  • Huffman coding is selected for small conversation archives because it avoids native dependencies and approaches zlib while outperforming lz4 at those sizes.

III. ARCHITECTURE

The architecture assigns session context and semantic query handling to layered caches that preserve recent conversation state and bypass redundant LLM work for similar queries.

  • The Session Context Window keeps the 20 most recent messages per session in Redis, maintaining a bounded rolling context.
  • Evicted messages are serialized into a Huffman-compressed disk archive, keeping Redis memory usage at O(k) per session regardless of conversation length.
  • When Redis lacks context, the system re-hydrates the latest k messages from disk and can fall back to disk-only reads if Redis is unavailable.
  • A semantic cache hit short-circuits the full search–synthesis pipeline, while misses load session context, check embeddings, and continue execution.
  • The Semantic Query Cache embeds each query, compares it with session- and URL-scoped cached pairs using cosine similarity, and returns a response when similarity reaches τ=0.90.

D. Layer 3: URL Embedding Cache (Redis DB 1)

The architecture treats URL embeddings as a distinct caching concern within a coordinated three-layer Redis design, separating global reuse from session-scoped state and semantic responses.

  • The URL Embedding Cache stores pre-computed 384-dimensional URL vectors globally as raw float32 bytes, allowing embedding work to be reused across sessions.
  • A single coordinator exposes four operations and routes each request to the appropriate session, semantic, or URL-embedding layer.
  • Separate layers support selective flushing, independent monitoring, and namespace isolation, while preserving distinct failure boundaries.
  • The architecture emerged after rejecting a monolithic Redis namespace whose mixed concerns would require application-level TTL management.
  • The design assigns long TTLs to session context, short TTLs to semantic responses, and an intermediate 24-hour TTL to URL embeddings.
  • The three layers occupy separate Redis logical databases, with DB 1 dedicated to long-lived global URL embeddings and distinct scope, TTL, and data format.

B. Huffman Coding vs. gzip/zlib/lz4

The system replaces truncation with Huffman-compressed archival so conversation history survives Redis eviction and supports later retrieval and resumption. A background daemon migrates idle sessions from Redis to disk while reclaiming memory.

  • B. Huffman Coding vs. gzip/zlib/lz4: Conversation archives typically range from 1–100 KB, making dictionary and frame overhead significant for gzip and lz4.Huffman coding uses no dictionary and adds only a symbol table proportional to the alphabet size.
  • B. Huffman Coding vs. gzip/zlib/lz4: Huffman-compressed disk overflow preserves full conversation history instead of discarding messages beyond the rolling window.This supports semantic retrieval, audit/replay, and session resumption after eviction.
  • B. Huffman Coding vs. gzip/zlib/lz4: Idle sessions are migrated to disk before Redis memory is freed, preserving data that ordinary TTL expiry would discard.The daemon addresses memory consumed by inactive sessions while retaining their contents.
  • B. Huffman Coding vs. gzip/zlib/lz4: The session lifecycle overflows the oldest messages when the Redis window exceeds k entries and re-hydrates archived history when users return.The LRU daemon migrates entire idle sessions, while returning users trigger disk-based re-hydration.

V. IMPLEMENTATION

The implementation divides the caching system into independently configurable modules and uses canonical Huffman encoding for compact, reconstructible archives. Encoding builds canonical codes from byte frequencies, while decoding reconstructs the original byte sequence from the stored header and bitstream.

  • V. IMPLEMENTATION: The caching system comprises eight independently configurable modules coordinated through one unified per-session entry point.The modules cover configuration, Redis pooling, Huffman coding, archival, hybrid caching, semantic caching, session context, and coordination.
  • V. IMPLEMENTATION: Canonical ordering stores symbol-to-length mappings so the decoder can reconstruct the same codes without storing the full tree.This design makes the archive’s codec representation sufficient for deterministic decoding.
  • V. IMPLEMENTATION: Each .huff archive combines a fixed 24-byte application header with a variable-length Huffman codec header and compressed payload.The application metadata can be read without decompressing the conversation payload.
  • V. IMPLEMENTATION: Canonical Huffman encoding counts byte frequencies, builds a min-heap, assigns bit-lengths, canonicalizes codes, and packs variable-length bits.The encoded stream prepends a header containing a magic value, data length, symbol table, and padding count.
  • V. IMPLEMENTATION: Decoding parses the header, rebuilds canonical codes and a lookup table, then reads bits until the original byte length is restored.Padding is excluded from the decoded bitstream.

D. Hybrid Conversation Cache

The hybrid conversation cache combines a Redis hot window with Huffman-compressed disk archival. Messages overflow atomically to disk when the window is full, and an empty Redis window is transparently re-hydrated from the archive.

  • D. Hybrid Conversation Cache: The hybrid cache connects hot Redis storage and cold disk storage through overflow and re-hydration paths.Its data flow is illustrated as messages moving between the storage tiers.
  • D. Hybrid Conversation Cache: Each archive stores session metadata in a fixed 24-byte application header and conversation JSON in a compressed bitstream.The layout also includes a Huffman codec header containing the symbol table required for decompression.
  • D. Hybrid Conversation Cache: When the Redis window exceeds its configured size, the oldest messages are appended to the disk archive and their Redis keys are deleted atomically.The overflow runs as a pipelined transaction.
  • D. Hybrid Conversation Cache: If Redis contains no context, the system loads the archive and repopulates Redis with the most recent k messages.This makes disk-backed recovery transparent to the application.

E. Semantic Cache

The system combines per-session semantic response caching with local infrastructure intended to reduce repeated search and inference work. Its cost estimate includes fixed infrastructure and variable provider inference, but request-level semantic-cache savings were not measured.

  • E. Semantic Cache: The semantic cache stores up to 50 query-response entries per session-URL pair, including a 384-dimensional embedding, full response, source URLs, and timestamp.Entries are configurable and stored as one JSON document per session-URL pair.
  • E. Semantic Cache: Cosine similarity over cached embeddings returns the best match exceeding the configured threshold for an incoming query.The normalization uses epsilon 10^-8 to avoid division by zero.
  • E. Semantic Cache: Raw 32-bit embedding bytes use 1,536 bytes for a 384-dimensional vector versus approximately 3,800 bytes as JSON, a 2.5× space saving.The representation is intended to matter when caching thousands of URLs.
  • E. Semantic Cache: The measured deployment used one 8-vCPU, 32 GB Intel Cascade Lake server with three containerized replicas and 30 Hypercorn workers.Redis 7.4 ran in a separate container capped at 2 GB, without a local GPU.
  • E. Semantic Cache: The measurement-period estimate combines approximately $96/month infrastructure with approximately $0.014 variable inference cost, yielding approximately $0.015 per standard query.The estimate is based on provider rates and token usage during the measurement period.
  • E. Semantic Cache: Request-level semantic-cache hits and avoided provider tokens were not measured, so the Redis keyspace hit rate cannot estimate cached inference savings.The reported Redis rate counts internal Redis operations rather than user queries bypassing inference.

B. Latency Profile

Redis reads are two orders of magnitude faster than disk reads, supporting a hot-window design while compressed archival effectiveness improves with payload size.

  • B. Latency Profile: Redis reads are two orders of magnitude faster than disk reads, confirming the value of keeping a hot window in memory.Fig. 8 reports the latency comparison, while the archive design uses disk for colder conversation data.
  • B. Latency Profile: 45% compression ratio is approached for payloads larger than 1 KB, as larger archives provide more statistical regularity for Huffman coding.Synthetic benchmarks confirm that compression ratio improves with payload size.
  • B. Latency Profile: Below 200 B, Huffman overhead reduces effectiveness, but the compression ratio remains below 80%.The symbol-table overhead is proportionally more significant for very small payloads.

1) Comparison with Standard Compressors:

Huffman compresses less effectively than zlib but is selected for small archives because its compression gap is limited at typical sizes and it has zero native dependencies.

  • 1) Comparison with Standard Compressors:: Zlib achieves better compression ratios at all tested sizes, while Huffman consistently outperforms lz4.At 2 KB, zlib-1 achieves 63.2% versus Huffman’s 68.8%, a 5.6 percentage-point difference.
  • 1) Comparison with Standard Compressors:: Huffman is chosen because the compression gap is small below 5 KB, the codec has zero native dependencies, and compression runs off the critical path.The trade-off prioritizes deployment simplicity and acceptable archival behavior over maximum compression.
  • 1) Comparison with Standard Compressors:: 89.3% is the aggregate Redis keyspace hit rate across all three databases, but it is not a direct query-level cache-hit measure.The rate includes TTL refreshes, list reads, existence checks, and other internal operations.

1) Per-Layer Contribution:

The three caching layers address session state, semantic query repetition, and repeated URL embedding, with each layer contributing differently to Redis activity and avoided computation.

  • 1) Per-Layer Contribution:: 75–80% of total keyspace hits are estimated to come from the Session Context Window’s repeated Redis reads for active sessions.This estimate is based on 2–4 reads per user message, 8-turn average sessions, and 16 active sessions.
  • 1) Per-Layer Contribution:: 15–20% of within-session queries are semantic near-duplicates, and each successful hit saves 3–8 seconds by avoiding search and LLM synthesis.The semantic cache uses cosine similarity ≥0.90 within a 5-minute TTL.
  • 1) Per-Layer Contribution:: Each URL embedding-cache hit saves approximately 200 ms, and its 24-hour TTL embeds each URL at most once per day regardless of session count.Popular URLs appear across 10–30% of sessions, giving this layer the highest per-hit savings despite the lowest hit volume.
  • 1) Per-Layer Contribution:: The architecture combines session context, semantic deduplication, and embedding reuse to support long-running sessions with a bounded hot window and disk-backed cold storage.The evaluation reports sub-millisecond cache reads, compressed archival, and session lifecycle management within a minimal Redis footprint.
  • 1) Per-Layer Contribution:: The evaluation uses one historical production snapshot and hardware configuration, so hit rates and latency may vary with query distributions or concurrency patterns.The semantic cache also uses O(n) brute-force similarity with n ≤50, which would require a vector index at significantly larger scale.
Loading 2609.05463v1…