A reference architecture for running retrieval-augmented generation and GPU-backed LLM inference on Oracle Container Engine for Kubernetes — built entirely with open-source tools.
Why this architecture
Teams evaluating Oracle Cloud Infrastructure for AI workloads usually arrive with the same question: how do the pieces fit together to serve a large language model in production — not as a hosted API call, but as infrastructure you own, schedule, and control?
To answer that concretely, we built a complete, working reference application on OCI: a retrieval-augmented generation (RAG) system with an agent layer, serving an open-weight model on a GPU, running entirely on Oracle Container Engine for Kubernetes (OKE) and provisioned end to end with Terraform. This post walks through the architecture and, more importantly, why each OCI primitive fits where it does — so you can adapt the pattern to your own workloads. The application itself is straightforward to describe: a user asks a question, the system retrieves relevant context from a vector database, grounds the model’s answer in that context with cited sources, and returns a response. An agent mode extends this with tool-calling. But the interesting part for architects isn’t the app — it’s the platform underneath it.
The architecture at a glance
The request path runs through a familiar, layered design:
Ingress → Frontend → Backend → (Vector DB + GPU inference)
- An OCI Load Balancer exposes the application publicly.
- A React frontend (served by nginx) handles the UI.
- A FastAPI backend orchestrates retrieval and generation.
- Qdrant provides vector storage for semantic search.
- vLLM serves the language model over an OpenAI-compatible API, running on a GPU node.
Every component runs as a workload on OKE. The only publicly exposed surface is the load balancer; worker nodes sit on a private subnet with no public IP addresses. All of it — the network, the cluster, the node pools, the registry, and the IAM policies — is defined in Terraform and deploys as a single stack.
Why OKE for the orchestration layer
The choice of OKE as the foundation comes down to three properties that matter for AI workloads specifically.
First, heterogeneous node pools. AI applications are rarely uniform. The frontend and backend are CPU workloads; inference needs a GPU. OKE lets you define separate node pools with different shapes in the same cluster — a cost-efficient CPU pool for the general application tier, and a dedicated GPU pool that exists only when inference is running. This separation is the core of keeping GPU spend under control.
Second, private-by-default networking. Placing worker nodes on a private subnet, with egress through a NAT gateway and OCI service access through a service gateway, is the right security posture for production. OKE integrates cleanly with OCI’s virtual cloud network primitives, so the frontend load balancer is the only public entry point while everything else stays internal.
Third, native integration with the rest of OCI. Block Volumes for persistent vector storage, the OCI Load Balancer for ingress, IAM for workload identity, and the OCI Registry (OCIR) for container images all connect to OKE through first-class integrations rather than bolt-ons. That coherence is what makes the whole thing describable as a single Terraform stack.
Serving the model: GPU inference as a first-class workload
The part most relevant to architects evaluating OCI for AI is how model serving works — because this is where the “GPU on Kubernetes” pattern earns its keep.
The language model runs in vLLM, an inference server that exposes an OpenAI-compatible endpoint. vLLM runs as a standard Kubernetes deployment, but with GPU-specific scheduling:
- A dedicated GPU node pool using an NVIDIA A10 shape, provisioned separately from the CPU pool.
- The NVIDIA device plugin, which advertises nvidia.com/gpu as a schedulable resource so Kubernetes knows the node has a GPU to allocate.
- A resource request in the vLLM pod for one GPU, plus a node selector and toleration so that only inference workloads land on the GPU node — nothing else competes for the expensive hardware.
This is the same pattern used to manage accelerators at much larger scale: GPUs become a schedulable, first-class resource in an orchestrated pool, rather than a fixed machine you SSH into. The backend reaches vLLM over an ordinary Kubernetes service, which means the application code doesn’t need to know or care where the GPU lives. A design detail worth highlighting: the backend was written to be inference-backend-agnostic. A single environment variable switches it between a local development model and the in-cluster GPU-served model. The same container image runs in both environments. That portability is a direct benefit of vLLM’s OpenAI-compatible API and is worth designing for from the start.
Infrastructure as code, end to end
The entire environment is defined in Terraform and packaged as an OCI Resource Manager stack. This delivers two things that matter for adoption:
Reproducibility. The network topology, IAM dynamic groups and policies, OCIR repositories, the OKE cluster, both node pools, and the GPU configuration all live in version controlled Terraform. There is no manual console setup to document or drift from.One-click deployment. Packaging the stack for OCI Resource Manager, with an input schema for the key variables, means the architecture can be deployed into any tenancy through a guided form — including a toggle to enable or disable the GPU node pool. Enabling the GPU is a single variable; tearing it down to stop accelerator billing is the same variable flipped off. Cost control becomes a first-class, declarative decision rather than a manual cleanup task.
Practical guidance for teams adopting this pattern
A few implementation notes will save time for anyone building something similar on OKE.
Right-size the GPU node’s boot volume. GPU container images and model weights are large. Provision the GPU node’s boot volume with generous headroom and ensure the node’s file system expands to use the full volume on first boot (a small cloud-init step handles this). Planning storage up front avoids capacity surprises during model loading.
Use fully qualified image names. OKE worker nodes use the CRI-O container runtime, which expects fully qualified image references (for example, docker.io/org/image:tag). Adopting this convention across all manifests keeps image pulls predictable.
Align your registry authentication. When pulling private images from OCIR, ensure the image pull secret’s registry hostname matches the hostname used in your image tags. Consistency here keeps authentication clean across the cluster.
Design for pod start ordering. Kubernetes does not guarantee the order in which pods become ready. Build a short retry-with-backoff into any service that depends on another at startup — for example, the backend waiting for the vector database. This is standard cloud-native resilience and makes deployments robust.
Separate CPU and GPU workloads deliberately. Keep the GPU node pool dedicated to inference through taints and tolerations. This protects your most expensive resource and makes GPU utilization easy to reason about — which matters when utilization is directly tied to cost.
Where this goes next
A running application is Day 1. The natural next phase is Day 2 operations: fine-tuning the model for a domain, building an evaluation harness so model improvements are measured rather than assumed, and adding full-stack observability — cluster and application metrics through Prometheus and Grafana, GPU telemetry through the NVIDIA DCGM exporter, and cost tracking that ties GPU-hours to utilization. On accelerated infrastructure, cost and performance are two views of the same metric and treating them together is what makes AI workloads sustainable in production.
Closing thoughts
The takeaway for architects evaluating OCI for AI is that the primitives compose cleanly. OKE for orchestration, dedicated GPU node pools for inference, vLLM for OpenAI-compatible serving, Block Volumes for stateful components, the OCI Load Balancer for ingress, OCIR for images, and Terraform plus Resource Manager to tie it all together — this fit into a single, reproducible, and cost-controllable architecture. The result is GPU-served AI that behaves like any other well-managed cloud-native workload: scheduled, observable, and deployable from code.
The complete reference implementation, including the Terraform stack and deployment guide, is available as an open-source project. [Link to repo/docs, https://dkhopade.github.io/learn-ai-from-claude/]