By Federico Kamelhar, Senior Principal Architect, Agentic AI — Oracle

NVIDIA NeMo Relay, NVIDIA’s open-source library that runs alongside the existing agent harness to observe and control AI agents, now treats Oracle Cloud Infrastructure (OCI) Generative AI as a first-class provider. Every chat call to the managed service — Meta Llama, Cohere Command, xAI Grok, Google Gemini, OpenAI GPT, or an imported NVIDIA Nemotron 3 model — is recognized, decoded, and normalized by the runtime, across all three of the service’s wire formats, with the same support in Rust, Python, and Node.js.

Teams running AI agents on Oracle Cloud can get, from one open-source runtime, a production surface they would otherwise build themselves: an audit trail of every model and tool call, PII redaction before anything reaches the observability pipeline and other available guardrails, and per-run cost tracking — configured outside the agent, with zero changes to agent logic. The same runtime and the same policies cover the managed OCI Generative AI service and self-hosted NVIDIA Nemotron on OCI Kubernetes Engine (OKE), so nothing about governance capabilities changes when the deployment model does. It plugs into LangChain with a single line of middleware. And it complements, rather than replaces, tools such as LangSmith: NeMo Relay records and controls the real execution path in-process, then exports the trajectory to whichever observability backend you already run.

That matters because an agent’s final answer may not be the ground truth. An agent can produce a confident, well-written answer built on steps it never took — later in this post, a trace catches one doing exactly that. What you audit, debug, and bill against is the trajectory: every model call, every tool call, in order. NeMo Relay records that trajectory alongside the agent — an in-process runtime, not a proxy — so your code keeps its own HTTP clients and OCI credentials, and prompts never leave your tenancy boundary.

Figure 1. One runtime, every Oracle path: agents flow through NeMo Relay, which speaks OCI Generative AI natively (oci_genai) and self-hosted OKE endpoints via their OpenAI-compatible API (openai_chat).

Figure 1. One runtime, every Oracle path: agents flow through NeMo Relay, which speaks OCI Generative AI natively (oci_genai) and self-hosted OKE endpoints via their OpenAI-compatible API (openai_chat).

What changed

NeMo Relay now ships an OCI Generative AI codec — the piece that teaches the runtime to read and write the service’s wire formats. It covers all three documented formats (GENERIC for Meta, Google, xAI, OpenAI, and imported models such as Nemotron; COHERE and COHEREV2 for the Cohere families), native FUNCTION tool calling, and streaming. OCI payloads are detected automatically by shape — nothing to register — and the runtime’s governance configurations accept codec: "oci_genai" in configuration, so one policy covers OCI alongside every other provider you use.

Example: a deployment advisor on LangChain

Here’s a question every team adopting NVIDIA Nemotron 3 on Oracle Cloud asks: use the managed service, or self-host on our own GPUs? The honest answer depends on live facts, so we built a small LangChain agent that checks them: one tool searches the managed model catalog, another reads the tenancy’s GPU service limits. The model runs on OCI Generative AI via langchain-oci — a catalog model to start with, your imported Nemotron 3 Super endpoint once it’s deployed — and NeMo Relay observes the whole run through one middleware entry:

from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_oci.chat_models import ChatOCIGenAI

import nemo_relay
from nemo_relay.integrations.langchain import NemoRelayMiddleware

@tool
def list_genai_models(keyword: str) -> str:
    """Search the managed OCI Generative AI model catalog by keyword."""
    ...  # OCI CLI/SDK call against the live tenancy

@tool
def gpu_capacity(shape_family: str) -> str:
    """Read the tenancy's GPU service limits per availability domain."""
    ...

agent = create_agent(
    # model: a catalog model, or your imported NVIDIA Nemotron 3 Super endpoint
    model=ChatOCIGenAI(model_id=MODEL,
                       service_endpoint=ENDPOINT,
                       compartment_id=COMPARTMENT),
    tools=[list_genai_models, gpu_capacity],
    middleware=[NemoRelayMiddleware()],   # every model and tool call, recorded
    system_prompt="Gather real data with the tools before recommending.",
)

exporter = nemo_relay.AtifExporter("advisor-session", "nemotron-advisor", "1.0.0")
exporter.register("atif_advisor")

with nemo_relay.scope.scope("nemotron-advisor", nemo_relay.ScopeType.Agent):
    result = agent.invoke({"messages": [{"role": "user", "content": QUESTION}]})

We ran it against a live tenancy, asking how to deploy Nemotron there. The agent checked the catalog, checked A10 capacity, and answered from what it found:

  [tool] list_genai_models(keyword='nemotron')
  [tool] list_genai_models(keyword='llama')
  [tool] gpu_capacity(shape_family='a10')

=== FINAL ANSWER ===
Based on the search results, there are no Nemotron models available in the
managed OCI Generative AI catalog, but there are several Llama models
available. The tenancy has 32 A10 GPU cores available in one of the
availability domains.

Therefore, the recommendation is to self-host NVIDIA Nemotron on OKE using
the tenancy's own GPU capacity.

Every fact is live tenancy data — Nemotron appears in the managed catalog only after you bring it in through Model Import. Meanwhile, with no instrumentation in the agent code, the exported trajectory recorded all four model turns and three tool calls, each step annotated with the model and its token usage:

"final_metrics": {
  "total_prompt_tokens": 3230,
  "total_completion_tokens": 166,
  "total_steps": 5
}

The same runtime also works without any framework: hand nemo_relay.llm.execute the exact ChatDetails payload you already sent to OCI, and the codec recognizes it by shape. Everything below applies either way.

Figure 2. The deployment advisor: a LangChain agent on OCI Generative AI, with tools that read the live tenancy — and NeMo Relay recording every step.

Figure 2. The deployment advisor: a LangChain agent on OCI Generative AI, with tools that read the live tenancy — and NeMo Relay recording every step.

What teams get

For production teams, the runtime centralizes the controls that would otherwise be rebuilt in every agent project:

  • A complete audit trail, with zero instrumentation. Every model call and tool call becomes a recorded step — request, response, model, tool arguments, token usage — grouped per run and isolated across concurrent agents. When something goes wrong in production, the trajectory is the record you debug against.
  • PII redaction that protects your telemetry, not your prompts. Redaction rewrites what is recorded, while the model receives the original. In the runs behind this post, a two-line policy recorded the requester’s email as j*******@example.com everywhere in the trace, with the original nowhere in it.
  • Guardrails around every call. Relay’s conditional-execution guardrails can block a request before it reaches the model, and its sanitize guardrails can rewrite what gets recorded. In the runs behind this post, a guardrail rejected a request for a production password before OCI Generative AI was ever called — the same guardrail can govern every other provider you route through.
  • Cost visibility per step, per run, per provider. Normalized token usage flows through a pricing catalog into the trajectory. The runs in this post landed at fractions of a cent, itemized — including one trace that spanned the managed service and a self-hosted Nemotron, with cost attributed across both.
  • Your observability stack, not a new one. Trajectories export as ATIF, OpenTelemetry spans, or OpenInference output.
  • Nothing leaves your tenancy. The runtime is an in-process library, not a gateway: your HTTP clients, your OCI signers, your network path.
Figure 3. One recorded advisor run, end to end: every model turn and tool call in a single trajectory — closing with the totals, the cost, and the masked requester email.

Figure 3. One recorded advisor run, end to end: every model turn and tool call in a single trajectory — closing with the totals, the cost, and the masked requester email.

Figure 4. Inside one managed call: the model receives the original payload, while the observability pipeline receives a redacted, normalized, priced copy.

Figure 4. Inside one managed call: the model receives the original payload, while the observability pipeline receives a redacted, normalized, priced copy.

CapabilityCoverage
Chat, buffered and streamingAll three OCI wire formats: GENERIC, COHERE, COHEREV2
Provider detectionAutomatic, by payload shape — no registration
Tool callingNative FUNCTION definitions, tool calls, and results, normalized across formats
GovernancePII redaction via codec: "oci_genai" in configuration; conditional-execution and sanitize guardrails around every call
Cost trackingPricing catalog applied to normalized token usage
ExportATIF trajectories, OpenTelemetry spans, OpenInference output
FrameworksLangChain and LangGraph integrations, or the plain API — no framework required
BindingsRust core, Python, Node.js

Running Nemotron on Oracle Cloud, observed

The model this is built for is NVIDIA Nemotron 3 — the current generation of NVIDIA’s open reasoning models, designed for exactly the agentic workloads described above. Throughout this post, the deployment target is Nemotron 3 Super. Oracle already supports importing Nemotron 3 Super into OCI Generative AI as a managed endpoint, and it can equally run self-hosted on OKE. NeMo Relay speaks the protocol of every supported path:

Deployment pathWhat it isNeMo Relay sees it via
OCI Generative AI — Model ImportOpen-weights Nemotron on a Dedicated AI Cluster, behind a managed endpointoci_genai
OCI Data Science — Model DeployThe NVIDIA NIM container from NGC on a managed endpointopenai_chat
Self-hosted on OKENVIDIA NIM or the open vLLM stack on OCI Kubernetes Engineopenai_chat

We verified both ends against live infrastructure. On the managed side, the integration ran end to end against a dedicated AI cluster serving an imported NVIDIA Nemotron 3 model. On the self-hosted side, we followed the advisor’s recommendation — a cookbook deployment on an OKE GPU node — and built a GPU operations copilot on it, with the self-hosted Nemotron as its model and tools reading live cluster telemetry. Only the endpoint changed: the runtime auto-detected the OpenAI-compatible format, and one trajectory captured the Nemotron analysis, a live-streamed summary, and a second opinion from the managed service, with cost attributed across both providers.

That deployment is also where the audit trail paid for itself. An early version of the copilot fabricated plausible GPU readings — numbers that existed nowhere in the cluster. The answer read fine; the trace showed one real tool call where the answer cited three. For a customer, that’s the difference between trusting an agent’s word and being able to check it.

What customers can build

  • Agents over your own data, audited — planning, tool execution, and memory inside your OCI tenancy and compartment, with a per-run trajectory you can hand to security, compliance, or an evaluation pipeline.
  • One governance policy for a mixed estate — the same redaction and guardrail configuration whether a team calls the managed service, an imported Nemotron 3 endpoint, or NIM and vLLM on OKE.
  • Cost-accountable AI — per-step, per-model, per-provider cost on every run, ready for showback across teams and environments.
  • A safe path from prototype to production — start on managed catalog models, move to your own Nemotron deployment when the economics say so, and keep the agent code, traces, and policies unchanged.

Conclusion

With this release, the pieces for running NVIDIA Nemotron 3 on Oracle Cloud line up end to end: both deployment paths are supported and documented, LangChain and langchain-oci cover the agent, and NeMo Relay covers what production actually demands — the audit trail, redaction, guardrails, and cost accounting underneath, identical whether you call the managed service or your own GPUs.

Next steps

Federico Kamelhar is a Senior Principal Architect on the Agentic AI team at Oracle, where he builds production agent systems on Oracle Cloud Infrastructure.