LiteLLM now routes every request to the right OCI Generative AI model with a trained classifier

By Federico Kamelhar, Distinguished Engineer, Agentic AI — Oracle

Why this matters

Most AI applications send every request to the same model. A password-reset question does not need the model a planning task needs; code belongs with a coding model, an invoice photo with a vision model, a Japanese request with a multilingual one. One model for everything either overpays for the easy requests or shortchanges the hard ones. Choosing per request has been hard to build: someone has to measure which model is good enough for what, wire the choice into a gateway, and keep credentials away from the agents that send the requests.

OCI Generative AI customers can now make that choice per request, with open-source software and configuration. The LiteLLM AI Gateway, which natively supports OCI Generative AI, puts twenty OCI Generative AI models from OpenAI, Google, Meta, xAI and Cohere behind one OpenAI-compatible endpoint. An application asks that endpoint for a model group, never for a vendor model. A small classifier, trained on answers a judge graded, plugs into LiteLLM’s routing and decides which group each request gets: the cheapest one likely to answer it well. LiteLLM’s own configuration adds the business rules: a premium lane by tag, virtual keys with budgets, fallbacks, a context-window check. A public reference implementation is available now. It deploys into an OCI tenancy with five scripts, or with one request to OpenAI Codex, and Codex itself can be its client.

Figure 1. One gateway between every application and every model. Applications ask LiteLLM for a model group with a virtual key; LiteLLM decides the route, applies the governance, signs the call and forwards it to OCI Generative AI. Applications never name a vendor model and never hold an OCI credential.

Four terms carry the rest of this post. A route is a kind of request, such as SIMPLE or CODE, together with the models good enough for it. A model group is LiteLLM’s name for a route’s models, such as oci-code; applications ask for a group and LiteLLM picks the model. The trained router is the classifier that chooses the route for a request. A virtual key is a LiteLLM key with a budget and the list of groups it may use; it is what an application holds instead of an OCI credential.

What’s new

The reference implementation, oci-litellm-routing on GitHub, gives an OCI tenancy:

• Eight routes over twenty OCI Generative AI models. Fast, balanced, strong, reasoning, code, vision, long-context and multilingual, each a LiteLLM model group with its prices from OCI’s public price list and a backup group to fall back to.
• A router trained on outcomes. A classifier that predicts, for each route, the chance of a good answer, and picks the cheapest route over a quality bar. It plugs into LiteLLM’s complexity router as a custom classifier, so the application asks for one model group, oci-auto-trained, and the decision comes back in the response headers.
• GPT-5.3-codex on OCI, through LiteLLM. OCI serves OpenAI’s coding model on its OpenAI-compatible Responses API, which LiteLLM’s OCI provider does not speak; a small custom provider closes the gap, tool calls included.
• Codex, and any OpenAI client, as a routed client. One block in Codex’s config.toml makes LiteLLM its model provider; from then on a question goes to a fast model and a code fix to GPT-5.3-codex.
• Governance from LiteLLM. Virtual keys with budgets and rate limits, a premium lane by tag, fallbacks, retries, latency-based load balancing, the context-window check, and spend per key.
• An agent that holds no credential. A langchain-oci agent in an NVIDIA OpenShell sandbox whose only way out is the gateway.
• The whole thing in one tenancy, built by Codex. An OKE cluster, one VM and an OCI API Gateway, from five idempotent scripts, or from a plain-language request to Codex, which runs the scripts and the checks from a specification that ships with the repository.

Example: an agent whose every call is routed

The gateway is configuration. A route is a model group with its deployments and prices, and the router the application asks for is one more entry in gateway/config.yaml:

x-oci: &oci {oci_region: os.environ/OCI_REGION, oci_compartment_id: os.environ/OCI_COMPARTMENT_ID}

model_list:
  - model_name: oci-fast                      # SIMPLE: quick facts, formatting, short rewrites
    litellm_params: {<<: *oci, model: oci/openai.gpt-6-luna}
    model_info: {input_cost_per_token: 1.0e-07, output_cost_per_token: 5.0e-07, max_input_tokens: 400000}
  - model_name: oci-code                      # CODE: GPT-5.3-codex, on OCI's Responses API
    litellm_params: {<<: *oci, model: oci-responses/openai.gpt-5.3-codex}
  # ... twenty models in eight routes, each route with a backup group
  - model_name: oci-auto-trained              # what the application asks for
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        classifier_type: custom
        classifier_plugin: trained_classifier.plugin
        classifier_plugin_timeout_ms: 4000
        fallback_tier: MEDIUM                 # if the classifier is slow or down
        tier_definitions:
          - {name: SIMPLE}
          - {name: CODE, description: "Writing, reviewing, explaining or debugging source code or SQL"}
          # ... MEDIUM, COMPLEX, REASONING, VISION, LONG_CONTEXT, MULTILINGUAL
        tiers:
          SIMPLE: oci-fast
          MEDIUM: oci-balanced
          COMPLEX: oci-strong
          REASONING: oci-reasoning
          CODE: oci-code
          VISION: oci-vision
          LONG_CONTEXT: oci-long
          MULTILINGUAL: oci-multilingual

router_settings:
  routing_strategy: latency-based-routing
  enable_pre_call_checks: true                # the context-window check
  enable_tag_filtering: true                  # the premium lane
  fallbacks: [{oci-fast: [oci-fast-backup]}, {oci-code: [oci-code-backup]}]   # ... one per route

An agent then asks for the group, never a model. The reference agent is a langchain-oci agent with tools, and its model is an OpenAI-compatible client pointed at the gateway. Every model call the agent makes, the first turn, each turn after a tool result, is routed on its own, and the decision rides back on each reply (app/agent.py, app/tools.py):

from langchain_core.tools import tool
from langchain_oci import create_oci_agent
from langchain_openai import ChatOpenAI

@tool
def get_account(account_id: str) -> str:
    """Look up a customer account by id, e.g. NW-4471: name, plan, monthly fee in USD."""
    return str(ACCOUNTS[account_id.strip().upper()])

model = ChatOpenAI(model="oci-auto-trained", base_url=f"{GATEWAY}/v1", api_key=LITELLM_KEY,
                   include_response_headers=True)        # the routing decision, on every reply
agent = create_oci_agent(model=model, tools=[get_account, lookup_policy, calculate], system_prompt=SYSTEM)

question = ("I was charged twice for September on account NW-4471. Look up our refund policy and draft "
            "a short, polite reply to the customer explaining what happens next.")
result = agent.invoke({"messages": [{"role": "user", "content": question}]})

for m in result["messages"]:                             # one line per model call: route, model, cost
    h = getattr(m, "response_metadata", {}).get("headers")
    if h:
        print(h["x-litellm-complexity-router-tier"], h["x-litellm-model-name"],
              h["x-litellm-response-cost"])

Figure 2 follows the first of those calls. The agent sends the question and its tools to the gateway with its virtual key (step 1). LiteLLM checks the key (2), applies the rules for images, length and language (3), and asks the trained router for the route (4 and 5): SIMPLE, because the fast models passed requests like this one 82% of the time when the router was trained. LiteLLM picks the fastest healthy model in that group (6), signs the call as its own OCI identity and sends it to OCI (7 and 8), records the cost against the key (9), and returns the reply with the decision in its headers (10). The model asked for a tool, so the agent runs it (11) and calls the gateway again (12), and that call is routed on its own. In the recorded runs this dispute took two or three calls, all on the fast route, for less than a thousandth of a dollar in all (evidence/e2e-cloud-2026-10-13.txt).

Figure 2. One request, step by step. Every model call an agent makes takes the same twelve steps inside the gateway, and the decision comes back on each reply in the x-litellm-* headers.

Plan mode goes one step further. A planner call splits the job into tasks, and each task goes through the gateway as its own request, so each gets its own route. An invoice photo with a request to reply and to fix a script became four tasks on four routes in the recorded run: VISION for the photo, SIMPLE for the summary, COMPLEX for the reply, CODE for the script (evidence/e2e-cloud-2026-10-13.txt). The control room that ships with the implementation shows each decision live, from the gateway’s own response:

Figure 3. One request’s decision in the control room: the route and the model, why in one sentence, and each route’s chance of a good answer against the 75% quality bar.

Example: Codex, routed

Codex speaks the Responses API and LiteLLM serves it, so one block makes LiteLLM Codex’s model provider and the router its model:

model_provider = "litellm"
model = "oci-auto-trained"

[model_providers.litellm]
name = "LiteLLM on OCI"
base_url = "https://<gateway-host>/v1"
env_key = "LITELLM_API_KEY"      # a LiteLLM virtual key with its own budget
wire_api = "responses"

Codex holds a LiteLLM key with a budget and no OCI or model-vendor credential. The repository’s check drives real Codex sessions and judges them from the outside, by the gateway’s log and by the result (evidence/codex/codex-clean-home-2026-10-20.md):

A Codex turn carries its whole context, about 9,000 tokens with this setting alone and about 50,000 with plugins and MCP servers, so a code turn costs about twenty times a question. Without routing every question is billed like one; the bigger the Codex setup, the more routing can save.

How a request finds its model

Three layers decide, in order, inside the gateway and before any model is called. They are steps 3, 4 to 5 and 6 of Figure 2:

1. Facts in the request, by rules. An image goes to VISION, more than about 30,000 tokens to LONG_CONTEXT, a language other than English to MULTILINGUAL.
2. What the request needs, by the trained router. A code check sends programming requests to CODE. Otherwise the router predicts each route’s chance of a good answer and takes the cheapest route over the quality bar: 75% by default, and a request can ask for its own.
3. Business rules, by LiteLLM. Tags, virtual keys and budgets, latency-based load balancing, fallbacks, retries and cooldowns.

Figure 4. Three decisions, in order, before any model is called, and three requests decided at three different layers. Every response carries the route, the model and the cost in its x-litellm-* headers.

Trained on outcomes, not opinions

The router learns from graded answers, not from anyone’s opinion of how hard a prompt is. Two models wrote 591 requests across 26 task families, from one-line facts to planning under constraints, including code and seven languages. Every route answered every request through the gateway, three times, and one judge graded the anonymised answers for correctness, completeness and instruction-following: 3,590 real OCI answers, about $18 a labelling.

Labelled three times, only 66% of the requests got the same cheapest passing route every time. That is why the router predicts each route’s chance of a good answer instead of one label, and why a request can set its own bar. The model is small: a logistic model per route on frozen multilingual-e5-small embeddings and seven difficulty signals, about 220 KB, trained in 0.18 seconds, classifying in a median of 79 ms on one Ampere core (evidence/oke-run.json). A customer trains it on their own requests the same way, with the scripts in classifier/.

Figure 5. Label by outcome, train in seconds, plug into LiteLLM.

On 73 held-out requests never used in training, every answer a real OCI call graded by the same judge (evidence/classifier-eval.json):

The trained router matches LiteLLM’s LLM classifier for 9% less, stays 5 points under always using the reasoning models for 19% less, and is the only one of the three that sends code and other languages to their own models. It can be wrong on a single request: on two demo prompts it picks the strong route where the reasoning route passed more often, and the checks report those as known misses instead of hiding them (evidence/strong-vs-reasoning.json). On 150 fresh requests written by another model and graded by two judges from different vendors, the order held (evidence/robustness.json).

Capability table

The isolation layer: the agent holds no credential

The reference agent runs in an OpenShell sandbox. What it finds in LITELLM_API_KEY is a placeholder; the OpenShell proxy swaps in the agent’s LiteLLM virtual key on requests to the gateway, and nowhere else. The sandbox’s whole network surface is one host and two paths, and its key is LiteLLM’s: five model groups, $5 every 30 days and a rate limit. In the recorded run, the sandbox refused the agent’s attempts to reach OCI directly, a model vendor, and the gateway’s admin API, three of three (evidence/codex/codex-cloud-ops-2026-10-15.log). The gateway signs every OCI call as its OKE node, so no OCI key or token is stored in the cloud.

Figure 6. The agent holds no credential; LiteLLM decides, signs and accounts. The agent sees a placeholder, OpenShell holds the agent’s virtual key, LiteLLM holds its master key and the spend, and OCI knows the node’s identity.

The build layer: let Codex do it

Write ~/.config/litellmdemo/deploy.json (your OCI CLI profile and regions; the repository has an example) and start Codex in a clone:

git clone https://github.com/fede-kamel/oci-litellm-routing && cd oci-litellm-routing
codex -s workspace-write -c sandbox_workspace_write.network_access=true \
  --add-dir ~/.config/litellmdemo --add-dir ~/.kube --add-dir ~/.cache/uv --add-dir ~/.npm \
  "$(cat codex/PROMPT.md)"

Codex runs the preflight, shows the plan with what it will create and what that costs, waits for your yes, runs the five build scripts, and runs the checks. The OCI API key stays where the OCI CLI reads it; no secret passes through Codex. Before anyone followed the specification, two fresh Codex sessions reviewed it as a stranger would and found 48 defects, all fixed (evidence/codex/). A first build in an empty tenancy with the final scripts has not been run yet; the repository says so.

Figure 7. You ask; Codex runs the scripts and the checks, and reports. The repository carries the rules, the skills and the specification; the scripts change the cloud; the checks say when it is done.

What customers can build

• One endpoint for every approved model. Models, prices, budgets and fallbacks in one configuration the platform team owns; applications ask for a model group and never hold an OCI credential.
• Cost-aware agents. Each call can go to the cheapest model good enough for it, with the decision and the cost in every response, ready for showback by key, team or tag.
• A router trained on your own outcomes. Label with your requests and your judge, retrain in seconds, and keep the evaluation that says whether it still pays.
• Agents that stay in their lane. A sandboxed agent whose only way out is the gateway, with a budget and a rate limit it cannot raise.
• Platforms that Codex can run. Rules, skills, a specification with checkpoints, and checks that decide when it is done.

Conclusion

Model choice per request is now a gateway decision on OCI Generative AI: LiteLLM makes it, a classifier trained on graded outcomes informs it, and every response says what was decided and why. The reference implementation shows the whole path, from a budgeted key in an OpenAI client, or in Codex, to the right one of twenty OCI models, and it can be built in a tenancy by the same coding agent that uses it.

Next steps

• The code and the evidence: oci-litellm-routing, with the gateway configuration, the router and its training scripts, the agent and sandbox, the control room, the cloud scripts, the Codex harness, and the transcript behind every number in this post.
• LiteLLM on OCI: the provider guide and the announcement, LiteLLM natively supports OCI Generative AI.
• The pieces around it: OCI Generative AI, NVIDIA OpenShell, langchain-oci, and OpenAI Codex.

Check model IDs, prices and availability in your region before you run the examples. The scenarios in the reference implementation use a fictional company, Northwind Cloud.

Federico Kamelhar is a Distinguished Engineer for Agentic AI at Oracle, building production agent systems on Oracle Cloud Infrastructure. Federico contributed the OCI Generative AI support in LiteLLM and in NVIDIA OpenShell.