Source-linked AI summary
Spyre-Accelerated Retrieval-Augmented Generation on IBM LinuxONE: A Cloud-Native Architecture for Secure, High-Throughput Enterprise AI Inference
Sandeep Bokkasam, Pankaj D
TL;DR
Regulated enterprises need generative AI architectures that keep sensitive knowledge bases within a defined physical perimeter while addressing latency, security, and compliance concerns. This paper presents a six-subsystem RAG design running entirely on IBM LinuxONE, and reports that it competes with cloud-GPU and on-premises alternatives for 3B–34B enterprise workloads while offering hardware-enforced encryption and confidential computing.
Problem
RAG deployments often assume that regulated knowledge bases can communicate with application servers and GPU-hosted LLMs over a network, creating a deployment challenge when data cannot leave a physical perimeter.
Method
The paper designs a six-subsystem RAG pipeline on IBM LinuxONE, using Telum II for intent-based routing, Spyre for LLM inference, and OpenShift for containerized operation.
Results
For 3B–34B enterprise RAG workloads, LinuxONE with Spyre holds its own on inference throughput and outperforms cloud offloading on latency and security, while on-premises GPUs remain its closest latency competitor.
Takeaways & Limitations
The architecture offers regulated enterprises a path toward production generative AI by keeping data and AI processing within LinuxONE while extending confidential-computing guarantees to AI workloads.
Takeaways & Limitations
The architecture is limited to models up to 34 billion parameters, has strongest support for IBM Granite, requires ahead-of-time compilation, and lacks extensive published Spyre benchmarks.
Abstract
from arXiv · showhide
Running large language models inside enterprise environments has always bumped up against a practical wall: the data lives in one place, the AI horsepower sits somewhere else, and moving sensitive records between the two creates real headaches around latency, security, and regulatory exposure. IBM's Spyre accelerator PCIe inference card built for LinuxONE and the broader IBM Z family changes that equation. In this paper we lay out a six-subsystem RAG architecture that runs entirely on IBM LinuxONE, using Spyre for generative inference, the Telum II on-chip accelerator for lightweight classification tasks, and Red Hat OpenShift for container orchestration. Every piece of the pipeline from query intake through vector retrieval, prompt assembly, LLM inference, compliance filtering, and response delivery stays within a single LinuxONE system, so sensitive data never has to leave the hardware perimeter. We walk through the design choices behind each subsystem, dig into the Spyre compilation and serving stack, explain how LinuxONE's Secure Execution technology extends confidential-computing guarantees to AI workloads, and benchmark the architecture against cloud-GPU and on-premises alternatives. Early analysis points to end-to-end RAG latencies under two seconds and up to a 20x reduction compared to off-platform inference, all while keeping the strong encryption and auditability posture that regulated industries actually need.
1IBM ISDL, INDIA
The paper focuses on IBM LinuxONE, Spyre, retrieval-augmented generation, confidential computing, and cloud-native enterprise AI.
- The paper examines enterprise AI inference using IBM LinuxONE, IBM Spyre Accelerator, and Retrieval-Augmented Generation.
1 Introduction
The paper addresses the latency, compliance, and data-movement risks of sending regulated enterprise data to cloud-hosted AI models. It proposes an end-to-end LinuxONE RAG pipeline with concrete latency, residency, auditability, and operational goals.
- Cloud-hosted GPU inference requires repeated data movement that adds latency, compliance exposure, and audit obligations for regulated organizations.
- LinuxONE combines Linux and Kubernetes compatibility with pervasive encryption, Secure Execution, and vertical scalability.
- The proposed pipeline runs query-to-answer RAG inference on LinuxONE with Spyre inside OpenShift.
- Its goals include sub-twosecond responses for typical 200-token answers, zero data egress, EU AI Act-compatible audit trails, and Kubernetes-based operations.
- The paper covers RAG and hardware background, six subsystems, the Spyre stack, security, benchmarking, deployment advice, and limitations.
2 Background and Related Work
The paper presents RAG as retrieval-grounded generation and targets deployment settings where regulated knowledge cannot leave a physical perimeter. LinuxONE supplies on-platform processing, Telum II supports lightweight inference, and Spyre supports larger language models.
- RAG retrieves trusted knowledge-base passages and supplies them with the user’s question so the model can generate an evidence-grounded response.
- Hybrid retrieval combines BM25 and dense vectors through rank fusion, followed by cross-encoder re-ranking.
- Conventional RAG deployments often assume network access to cloud-hosted LLMs, an assumption that conflicts with regulated data perimeters.
- LinuxONE runs Linux applications and containers while providing pervasive hardware encryption and a Telum II on-chip accelerator for microsecond-latency quantized models.
- Spyre is a PCIe Gen5 inference accelerator with high-bandwidth memory and tensor-processing cores; up to six cards can serve separate models or cooperate through tensor parallelism.
- IBM optimizes Granite models for Spyre and provides ahead-of-time compilation plus vLLM support for continuous batching and PagedAttention.
3 System Architecture
The architecture decomposes an OpenShift-hosted RAG service into independently operable subsystems spanning input handling, routing, retrieval, inference, compliance, and feedback. Telum II handles lightweight decisions while Spyre performs generative inference within the LinuxONE system.
- Architecture overview: Six containerized microservice subsystems can be scaled, updated, and monitored independently under production SLAs.
- Input and preprocessing: The API gateway authenticates requests, applies rate limits, and performs basic routing using OAuth 2.0 or OIDC.
- Input and preprocessing: Input validation detects prompt-injection patterns, enforces length limits, normalizes Unicode, masks selected PII, and retrieves conversation history.
- Input and preprocessing: The preprocessing subsystem adds maybe a millisecond or two and runs on standard Telum II cores.
- Intent detection and routing: Intent routing sends transactional requests to appropriate services while reserving generative inference for queries requiring retrieval and generation.
- Intent detection and routing: A quantized DistilBERT-class classifier on Telum II makes routing decisions in tens of microseconds, using a tunable default confidence threshold θ=0.85.
- Intent detection and routing: The routing layer treats Spyre as a scarce resource and allocates it deliberately instead of sending every request through the most expensive stage.
- Knowledge retrieval: Retrieval uses source routing, parallel BM25 and dense search with Reciprocal Rank Fusion, cross-encoder re-ranking, and filtered context assembly.
4 Inside the Spyre Stack
Spyre combines ahead-of-time model compilation with vLLM-based serving to provide deterministic, optimized inference on LinuxONE. Its estimated latency advantage is attributed primarily to eliminating external endpoint overhead rather than faster raw token generation.
- Hardware and integration: Spyre cards use PCIe Gen5, dedicated high-bandwidth memory, tensor-processing engines, and Kubernetes integration through a device plugin.The hardware targets transformer operations including matrix multiplication, softmax, layer normalization, and attention-score computation.
- Compilation pipeline: Ahead-of-time compilation maps PyTorch or ONNX models onto Spyre execution units while applying quantization, operator fusion, and memory-layout optimization.The compilation process produces an optimized binary tailored to the card’s tensor-processing architecture.
- Compilation pipeline: Tens of minutes of compilation for Granite 8B produce a deterministic, reproducible artifact that can be checksummed for production governance.
- Serving runtime: vLLM handles OpenAI-compatible requests, PagedAttention-based KV-cache management, and continuous batching through the Spyre backend.The performance envelope is presented as an estimate based on architectural analysis and IBM’s published guidance.
- Performance implications: 20× lower latency is attributed mainly to eliminating network serialization, TLS handshakes, API-gateway queuing, and network round-trip latency.The comparison is framed against external inference endpoints, not as faster raw token generation than a high-end GPU.
5 Security and Confidential Computing
LinuxONE makes security a primary architectural property by keeping the RAG pipeline on one system and applying pervasive hardware-enforced encryption. Secure Execution extends these guarantees to model weights, prompts, and retrieved context in memory, including against privileged administrators.
- Zero data egress: The complete RAG pipeline runs on a single LinuxONE system, so sensitive data never crosses a physical network boundary.The deployment includes the API gateway, vector store, databases, embedding models, LLM, compliance filters, and logging infrastructure.
- Pervasive encryption: LinuxONE encrypts data at rest, in transit, and in memory using dedicated cryptographic engines with no measurable performance penalty.The protection covers network traffic within the system as well as prompt, context, and response data paths.
- Secure Execution: Secure Execution creates hardware-isolated enclaves whose encrypted memory keys are inaccessible to the hypervisor and system administrators.
- Secure Execution: Inside a Secure Execution enclave, model weights, prompts, and retrieved context remain protected from privileged inspection throughout inference.The guarantee is relevant when models are proprietary or fine-tuned on sensitive data.
- Secure Execution: Secure Execution is especially relevant to multi-tenant and managed-service deployments because its guarantee does not depend on trusting the infrastructure administrator.
- Regulatory mapping: The architecture maps LinuxONE capabilities to specific regulatory requirements.
6 How It Compares
The comparison presents LinuxONE with Spyre as a strong fit for enterprise RAG workloads in the 3B–34B range, especially when latency, security, and compliance matter. Cloud GPUs retain an advantage for much larger models, while on-premises GPUs remain competitive on latency but lack comparable hardware-enforced protections.
- Evaluation scope: The benchmark compares four deployment approaches across security, latency, scalability, operational overhead, and compliance burden.
- Comparative findings: Cloud GPUs provide greater raw model-size flexibility, making them more suitable when workloads require 70B or 400B models.
- Comparative findings: For 3B–34B enterprise RAG workloads, LinuxONE holds its own on inference throughput and outperforms cloud offloading on latency and security.
- Comparative findings: On-premises GPU clusters are the closest latency competitor but lack hardware-enforced encryption and confidential computing while adding separate infrastructure-management complexity.
- Compliance implications: LinuxONE’s hardware- and platform-managed security posture reduces the compliance-engineering effort required to deploy generative AI in regulated environments.
7 Practical Deployment Guidance
The deployment guidance recommends matching Granite model size and Spyre capacity to task complexity and demand, while treating retrieval quality and observability as operational priorities.
- Model Selection: Granite 3B suits high-volume, well-scoped tasks where speed matters more than nuance.It runs on one Spyre card with headroom for concurrent requests.
- Model Selection: Granite 8B is the recommended sweet spot for general-purpose enterprise RAG deployments.It supports multi-turn document Q&A, report drafting, and customer correspondence while fitting on a single card.
- Model Selection: Granite 13B–34B extends the model range for more demanding workloads.
- Knowledge Base Quality: 256–512-token chunks with 10–15% overlap and structure-aware semantic chunking support retrieval quality.Re-indexing should follow source-document change frequency, and embedding-model versions should remain pinned.
- Capacity Planning: 60–70% average Spyre utilization is recommended to absorb peak loads without degrading response times.vLLM continuous batching improves throughput, but physical capacity limits remain.
- Capacity Planning: Spyre cards cannot auto-scale on a five-minute notice like cloud GPUs, so capacity planning is required.
- Observability: P95 latency, retrieval hit rate, grounding score, and Spyre queue depth are key operational health metrics.The paper recommends deploying OpenTelemetry instrumentation from day one.
8 Limitations and Future Directions
The architecture is bounded by current Spyre model capacity, uneven model support, compilation overhead, and sparse benchmark evidence, while future work targets larger effective capacity and broader on-platform adaptation.
- Limitations: 34 billion parameters is the current ceiling, excluding frontier models in the 70B–400B range.
- Limitations: Spyre support is strongest for IBM Granite, while other popular open-source models lack feature parity.
- Limitations: Ahead-of-time compilation adds friction compared with the load-and-serve workflow of GPU environments.
- Limitations: Sparse published Spyre benchmarks make precise capacity planning harder for early adopters.
- Future Directions: Future directions include Mixture-of-Experts, federated RAG, on-platform RLHF fine-tuning, and agentic AI patterns.
9 Conclusion
The paper presents an end-to-end RAG architecture on IBM LinuxONE that combines Spyre, Telum II, and OpenShift without external GPUs or data egress. It argues that LinuxONE provides on-platform AI compute, default encryption, and confidential-computing guarantees for regulated enterprise deployments.
- Architecture: The six-subsystem design runs entirely on IBM LinuxONE with no external GPUs, cloud round-trips, or data egress.
- Architecture: The pipeline combines query preprocessing, Telum II intent routing, hybrid retrieval, Spyre-accelerated inference, compliance guardrails, and observability.
- Platform Capabilities: Spyre supplies enough AI compute to run production-grade generative models on-platform.
- Platform Capabilities: LinuxONE encrypts data by default and extends confidential-computing guarantees to AI workloads through Secure Execution.
- Enterprise Implication: For regulated enterprises, keeping data in place provides a path toward production generative-AI deployment.