Source-linked AI summary
OGX: An Open-Source, Vendor-Neutral Generative AI Application Server
Francisco Javier Arceo, Sébastien Han, Matthew Farrellee, Charlie Doern, Yuan Tang, Derek Higgins, Varsha Prasad Narsing, Gordon Sim, Sumanth Kamenani, Ben Browning, Raghotham Murthy
TL;DR
Full-stack AI applications remain tightly coupled to proprietary providers, making migration and self-hosting difficult. OGX provides a vendor-neutral, self-hosted server with standard APIs and pluggable providers; its validated retrieval isolation achieved 0% cross-tenant leakage with approximately 19ms added latency.
Problem
Full-stack AI applications remain difficult to migrate or self-host because retrieval, tools, state, and safety remain coupled to proprietary providers and fragmented infrastructure.
Method
OGX implements standard OpenAI, Anthropic, and Google APIs through pluggable provider interfaces, decoupling SDK, model, and deployment choices from application code.
Results
0% cross-tenant leakage was demonstrated for ABAC-gated retrieval, with approximately 19ms added to the search path.
Takeaways & Limitations
OGX supports self-hosted, interoperable AI application deployments whose underlying infrastructure can be switched through configuration rather than client-code changes.
Abstract
from arXiv · showhide
OGX (Open GenAI Stack) is an open-source AI application server and Python library that implements the APIs of major frontier labs (OpenAI, Anthropic, Google) with pluggable backend providers. Developers building agentic AI applications--such as retrieval-augmented generation pipelines, multi-turn agents, and tool-calling workflows--can develop against a single API surface and deploy with any combination of inference engine, vector database, and safety backend, without changing application code. OGX's primary focus is the Responses API for server-side agentic orchestration, conforming to the Open Responses specification. The server also supports the Anthropic Messages API and Google GenAI Interactions API, decoupling SDK choice from model and deployment decisions. With over 20 inference providers, 13 vector store backends, and a companion Kubernetes Operator for production deployment, OGX serves as the self-hosted, model-agnostic backend for AI-powered developer tools including Claude Code, Codex CLI, OpenCode, and OpenHands. The project has over 8,400 GitHub stars, 242 contributors, and 4,000 commits across nearly two years of public development.
1 Introduction
AI applications remain tightly coupled to proprietary providers, especially when they require retrieval, tool calling, conversation state, and safety. OGX provides a self-hosted server with compatible APIs that separates SDK, model, and deployment choices through configuration rather than code changes.
- Full-stack AI applications remain difficult to migrate because retrieval, tool calling, conversation state, and safety guardrails are coupled to proprietary providers.
- AI developer tools require integrated backends for model serving, tool execution, conversation state, and retrieval, forcing self-hosting organizations to assemble multiple systems and glue code.
- OGX provides a complete self-hosted application server implementing OpenAI APIs, primarily Responses, with Anthropic Messages and Google GenAI Interactions compatibility layers.
- OGX decouples SDK, model, and deployment decisions, allowing developers to use standard endpoints and swap infrastructure through configuration without changing application code.
2 State of the Field
The AI application ecosystem is divided into specialized layers, while OGX provides a self-hosted, vendor-neutral server integrating inference, retrieval, tools, conversation management, and safety. Its Open Responses conformance enables interoperability across compatible clients.
- API gateways: API gateways unify access across inference providers but function as pass-through proxies that translate request formats without managing application state or execution.They do not manage vector stores, execute tool calls, or maintain conversation history.
- Client-side frameworks: Client-side frameworks offer rich abstractions for agents and RAG pipelines but distribute security-critical orchestration logic across application code.They are complementary to OGX: frameworks compose agent logic, while OGX provides the server-side execution target.
- Proprietary platforms: Proprietary platforms provide integrated experiences but couple applications to a vendor’s infrastructure and pricing.Examples include OpenAI’s Responses API and Databricks Mosaic AI.
- OGX’s position: OGX provides a self-hosted, vendor-neutral server with full-stack APIs and pluggable providers at every layer.It covers inference, retrieval, tool execution, conversation management, and safety, while conformance to the Open Responses specification supports interoperability with compatible clients.
3 Software Design
OGX uses protocol-defined, pluggable providers and configurable distributions to separate application logic from infrastructure choices. A single server supports multiple client protocols, with the Responses API providing server-side agentic orchestration and centralized security controls.
- Pluggable Provider Architecture: OGX defines each capability through a Protocol interface, with concrete providers implementing inference, vector storage, safety, tools, and file processing backends.A routing layer dispatches requests to providers such as remote inference engines, inline or remote vector stores, moderation services, built-in tools, and MCP servers.
- Deployment Configuration: Distributions package provider selections and configuration into deployable units, allowing production backends to replace prototype backends without changing application code.The starter distribution supports quick setup, while custom distributions target production environments.
- Deployment Modes: OGX supports server and library modes with identical provider routing and API semantics, covering production deployment, multi-language access, notebooks, scripts, and rapid prototyping.The recommended progression is library mode first, followed by server mode when scaling, production isolation, or multi-language access is needed.
- Protocol Compatibility: A single OGX server serves OpenAI-compatible, Anthropic Messages, and Google GenAI Interactions protocols, separating SDK choice from model selection and deployment target.The Responses API is the primary OpenAI-compatible interface and conforms to the Open Responses specification.
- Agentic Orchestration and Security: The Responses API executes inference-to-tool-to-inference loops server-side, centralizing tool authorization, tenant-scoped conversation state, safety checks, and context compaction.ABAC enforces tenant isolation across retrieval, tool execution, state management, and API routing layers.
4 Example Usage
OGX demonstrates portability through unchanged client code across inference providers and vector-store backends. Its examples show standard OpenAI SDK calls for server-side RAG, with provider changes confined to server configuration, alongside deployments across diverse enterprise backends.
- Portability: The same client code works across configured inference providers and vector-store backends.OGX’s examples specifically demonstrate portability independent of server backend selection.
- RAG agent: Standard OpenAI SDK calls are sufficient to build a RAG agent while OGX handles chunking, embedding, storage, retrieval, and context injection.The server exposes these RAG operations transparently through its application interface.
- Deployment switching: Only the server’s distribution configuration changes when switching from local Ollama to a production vLLM cluster; client code remains identical.This example isolates deployment changes from application-level code changes.
- Enterprise backends: OGX portability is demonstrated with IBM watsonx.ai plus Milvus, Oracle Cloud Infrastructure, and Red Hat OpenShift enterprise backends.The examples include enterprise RAG, standardized API access through Llama Stack, and an intelligent operations agent integrating agentic RAG, web search, and MCP tools.
5 Kubernetes Operator
The OGX Kubernetes Operator provides declarative, production-grade OGX deployment on Kubernetes and OpenShift through an OGXServer custom resource. It automates lifecycle management, configuration updates, image changes, and multiple tenant-isolation topologies.
- Deployment lifecycle: The Go-based operator uses the OGXServer CRD to automate OGX server deployment lifecycle on Kubernetes and OpenShift.It handles creation, scaling, updates, and teardown of OGX server pods.
- Resource configuration: An OGXServer CR specifies distribution, replica count, persistent storage, environment overrides, external routing, and network policies.Network policies are enabled by default per custom resource.
- Configuration management: ConfigMap-driven overrides restart matching OGXServer resources for fleet-wide image updates, while watched configuration changes restart pods automatically.Administrators can update the operator ConfigMap image-overrides key, and users can supply config.yaml content through ConfigMaps.
- Deployment topologies: The operator supports shared, per-tenant, and hybrid deployment topologies, trading cost efficiency against namespace, logical, or data-path isolation.Shared instances use ABAC, per-tenant instances use Kubernetes RBAC, and hybrid deployments share inference while separating tenant data paths.
6 Research Impact and Adoption
OGX is deployed in production across multiple enterprise sectors and supports open, vendor-neutral agentic AI APIs and developer tools. Its multitenant security model was peer-reviewed and validated with zero cross-tenant leakage at approximately 19 ms retrieval overhead.
- Production adoption: OGX is deployed in production across telecommunications, semiconductor manufacturing, financial services, insurance, and consulting enterprises.Deployments use pluggable inference providers, tenant-isolated vector stores, server-side agentic orchestration, and Kubernetes-based deployment.
- Production adoption: Published use cases include IBM watsonx.ai with Milvus, Oracle Cloud Infrastructure with OCI AI Blueprints, and Red Hat OpenShift with MCP-based operational agents.
- Security validation: 0% cross-tenant leakage rate was achieved by ABAC-gated retrieval, with approximately 19ms added to the search path.The defense operates at the retrieval layer and remains resilient to prompt injection regardless of model behavior.
- Community adoption: Over 8,400 GitHub stars, 242 unique contributors, and over 4,000 commits mark adoption across 68 releases in nearly two years of public development.The project also has 1,300 forks, weekly community contributor calls, an active Discord server, and integrations from Red Hat, IBM, Oracle, and Infinispan.
- Standards and tooling: OGX conforms to the Open Responses specification and serves as a reference implementation for open, vendor-neutral agentic AI APIs.
- Standards and tooling: OGX supports Claude Code, Codex CLI, OpenCode, and OpenHands by providing a self-hosted, model-agnostic backend that enables organizations to use any model on their own infrastructure.