By Sanjay Basu, PhD, Senior Director, Gen AI and GPU, Oracle Cloud Infrastructure

How OCI’s X12 Bare Metal Shape Unlocks Agentic AI on Intel® Xeon 6® CPUs

Every enterprise AI conversation I have starts with the same question. Do we really need GPUs for everything? The honest answer is no. A significant portion of production AI inference workloads, particularly those driven by agentic architectures, consist of short-context reasoning steps, structured extraction tasks, classification calls, and retrieval-augmented generation queries. These workloads do not require the memory bandwidth or parallelism of a discrete GPU. They require fast CPU cores, large system memory, and the right software stack.

That is exactly what the OCI X12 Standard Ax bare metal shape is built for. Powered by a single-socket Intel Xeon 6 6900P Series processor with 120 cores and 1,152 GB of DDR5 memory, this shape is the first OCI compute generation purpose-engineered around agentic AI workloads where CPUs carry the inference load and GPUs handle what CPUs cannot.

What follows is a combination of technical context, production architecture patterns, and results from workloads I ran on the X12 shape. The benchmark numbers are real. The architecture patterns are field-tested.

What Makes the X12 Standard Ax Different

The X12 Standard Ax represents a generational shift from the X9 Standard shapes most OCI customers know today. Where the X9 uses dual 32-core 3^rd gen Intel Xeon (formerly codenamed Ice Lake) Platinum 8358 processors, the X12 consolidates to a single-socket 120-core Intel Xeon 6 6987P-C.

The headline comparison across generations

The single-socket design eliminates cross-socket NUMA latency. For inference workloads where token generation is a tight loop over weight matrices, that matters more than the raw core count difference suggests.

Intel AMX – Matrix Acceleration Without a GPU

Intel AMX is the hardware acceleration on each CPU core that speeds-up matrix multiplication operations at the heart of AI inference that accelerates the matrix multiplication operations at the heart of neural network inference. Unlike a discrete GPU accelerator, AMX executes directly on the CPU die, sharing the same memory address space as the rest of the workload. There is no PCIe transfer, no context switch between host and device, no GPU memory capacity ceiling separate from system DRAM.

For agentic inference specifically, AMX acceleration means the CPU handles the tiled matrix operations of transformer attention and feed-forward layers with dedicated hardware rather than scalar SIMD code paths. OpenVINO and vLLM with IPEX both activate AMX automatically on Xeon 6 CPUs when running quantized models, so the software stack change is minimal.

Benchmark Results from X12 Workloads

The following tables reflect results from running workloads on X12 (120-core BM.Standard4.Ax.120) bare metal instances compared to X9 (64-core BM.Standard3.64) bare metal instances. All AI inference benchmarks used the meta-llama/Meta-Llama-3-8B-Instruct model via llama.cpp with AMX-aware quantization. General compute benchmarks used standard industry benchmark suites. These numbers align with the Intel engineering team characterization data released through the OCI/Intel joint training program.

Note on benchmark status – performance data was measured at ACT 3.2 GHz frequency. A pending LOM patch enables SCT 3.9 GHz, which is expected to improve performance further across all workloads.

AI Inference – SLM Throughput Benchmark (X9 vs X12 Bare Metal)

Model tested are meta-llama/Meta-Llama-3-8B-Instruct on X9 64-core vs X12 120-core bare metal

The 5x to 6x throughput improvement over X9 on the 8B parameter model reflects the combined effect of 88 additional cores, Intel AMX matrix acceleration, and 42% higher memory bandwidth. For practical SLM serving, this translates to substantially higher concurrency per bare metal node, which directly compresses cost-per-token.

General Compute Benchmark Results (X12 VM Performance vs X9 Baseline)

The SPECfp gain at 48% is particularly relevant for inference. Floating-point throughput tracks closely with the matrix operations in quantized model computation. The MySQL and MongoDB results matter for the agentic architecture pattern, where agent steps frequently read from and write to structured data stores.

Pricing Efficiency at the Benchmark Numbers

At $1.43 per hour for a 120-core bare metal node delivering 5x to 6x higher throughput than the X9 shape at $2.56 per hour, the per-token cost improvement is in the range of 10x to 12x on the same model. That is not a rounding error. It changes the architecture decision calculus entirely.

The Agentic Architecture – SLMs Locally, LLMs by API

The most cost-effective agentic AI architecture is not one that routes every request to a 70B+ parameter model running on an H100. It is a heterogeneous system where small language models handle the majority of inference steps and larger models handle only those tasks that genuinely require their capability.

Research from NVIDIA (Belcak and Heinrich, 2025) makes this quantitative. Serving a 7B-parameter SLM is 10x to 30x cheaper in latency, energy consumption, and compute than serving a 70B to 175B-parameter model. The X12 Standard Ax shape operationalizes this principle in OCI.

Figure 1. Agentic AI Inference Architecture on OCI X12 with Intel Xeon 6

Run quantized SLMs (Llama 3.2 3B, Phi-4 Mini, Qwen 3.5, Mistral Small) locally on the 120-core Xeon 6 with AMX acceleration. Use Oracle’s LiteLLM-based intelligent model router to escalate requests that exceed the SLM’s context window, require multi-step reasoning over long documents, or have scored low confidence on a preceding classification step. Route those escalated calls to OCI Generative AI Service (Llama 3.3 70B, Cohere Command R+).

This is the Total Cost of Intelligence pattern applied to compute infrastructure. Use the right resource for each inference call, and the aggregate cost drops 40 to 80% compared to routing everything to a large model. The router itself, built on LiteLLM with LangFuse observability, runs on the same X12 node and adds negligible latency overhead at the concurrency levels typical of enterprise agentic deployments.

Software Stack on OCI X12

The inference stack on an X12 bare metal instance is production-ready with no exotic dependencies. Install OpenVINO or vLLM with the Intel Extension for PyTorch (IPEX), which activates AMX automatically on Granite Rapids. Both frameworks support INT4, INT8, and FP16 quantized model formats. Llama.cpp also works natively and is what I used for the throughput benchmarks above.

For serving, OpenVINO Model Server (OVMS) or vLLM with the OpenVINO backend expose an OpenAI-compatible API endpoint. LangChain, LangGraph, or CrewAI agents connect to this endpoint identically to how they connect to any remote model provider. The local nature of the endpoint means the call is a loopback, not a network hop, which is measurable at sub-millisecond round trips for the transport layer.

Observability through LangFuse or OpenTelemetry captures per-invocation latency, token throughput, and cost attribution, so you can validate the economics in real time and tune your routing thresholds.

Industry Use Cases

Healthcare – Clinical Documentation and Patient Monitoring Agents

Hospitals generate enormous volumes of unstructured clinical data. Physician notes, discharge summaries, radiology reports, nursing assessments. Agentic AI systems built on the X12 shape can process this data locally without requiring GPU capacity that competes with imaging or genomics workloads.

The data residency advantage is critical here. Protected health information (PHI) under HIPAA never needs to leave the OCI tenancy. A 3B to 8B parameter SLM quantized to INT8 running on the X12 with AMX delivers latency low enough for ambient documentation workflows, where a physician cannot wait more than a few seconds for a structured note draft to appear.

Financial Services – Compliance Screening and Document Analysis Agents

Financial institutions process millions of documents annually for regulatory compliance. KYC reviews, AML transaction screening, credit risk assessments, and sanctions screening. An agentic compliance system on the X12 shape runs a locally-hosted SLM that extracts structured fields from scanned documents, cross-references entity databases, and flags anomalies for human review.

Data sovereignty is non-negotiable in financial services. Running inference locally on OCI bare metal within a dedicated tenancy and region satisfies the data localization requirements that many regulators mandate.

Energy and Utilities – Predictive Maintenance Agents

Energy companies operate thousands of distributed assets that generate continuous telemetry data. Agentic AI systems on the X12 shape can ingest sensor streams, classify anomalies in real time, and generate maintenance recommendations without the complexity of a GPU-based inference cluster.

The X12’s single 100G CNIC and high memory bandwidth are well-suited for the streaming ingest pattern these workloads demand. Because inference runs on CPU, there is no contention with GPU clusters handling larger training or simulation workloads.

Manufacturing – Quality Inspection and Production Optimization Agents

Discrete manufacturers increasingly deploy vision-based quality inspection systems on production lines. The X12 shape supports inference for computer vision models alongside agent orchestration and NLP inference simultaneously, using the 120 cores to parallelize workloads that would otherwise require separate GPU instances.

Why OCI for CPU-Based AI Inference

OCI’s off-box virtualization architecture means the X12 bare metal shape delivers all 120 cores and 1,152 GB of memory to the workload with zero hypervisor tax. The control plane runs outside the host, so you are not sharing cores with a hypervisor or a cloud provider’s telemetry stack.

OCI’s flexible VM shapes (VM.Standard4.Ax.Flex) let you right-size for development and testing before scaling to bare metal for production. The same AMX-accelerated inference stack runs identically across both. And when your workload genuinely needs GPU acceleration for larger models, OCI’s GPU portfolio (H100, H200, B200, GB200 NVL72) is available in the same regions, connected through the same VCN fabric, with the same networking credentials.

The Oracle Acceleron SmartNIC on the X12 offloads networking, storage, and security functions from the host CPU onto dedicated hardware, preserving more of the 120 cores for inference rather than infrastructure bookkeeping.

Getting Started

• Provision a BM.Standard4.Ax.120 or VM.Standard4.Ax.Flex shape in your OCI tenancy
• Install the OpenVINO toolkit or vLLM with IPEX for AMX-accelerated inference
• Download a quantized SLM (Llama 3.2 3B Q8, Phi-4 Mini, Qwen 3.5) and serve it via OpenVINO Model Server or vLLM
• Connect your agentic framework (LangChain, LangGraph, CrewAI) to the local inference endpoint
• Deploy the LiteLLM intelligent router to gate escalation to OCI Generative AI Service or external LLM APIs
• Monitor with LangFuse or OpenTelemetry to validate cost savings versus GPU-only inference

The future of enterprise AI inference is not about choosing between CPUs and GPUs. It is about using both intelligently. The X12 Standard Ax shape gives OCI customers a powerful new option for the CPU side of that architecture. 120 cores of Intel Xeon 6 with AMX acceleration, 1,152 GB of DDR5 memory, and a price point that makes SLM-based agentic inference economically compelling at production scale. The benchmark numbers above are not theoretical. They are what I measured.