We are excited to announce that release 26.2 of the Private AI Services Container includes the Private Large Language Models Service. This means that open-weight Large Language Models (LLM) can now be hosted in the Private AI Services Container.

What challenges does the Private Large Language Model Service Address?

Confidentiality / privacy

Commercial Large Language Models can be useful for many things, but usually you need to provide them with your questions, code and/or documents as context. This means that your questions, code, or documents are shared with the provider who hosts the commercial LLM.

Depending on the provider, service configuration, and contractual terms, users may need to consider how prompts, documents, code, and other submitted content are processed and retained.

If your questions, code or documents should remain private and confidential to you or your company, then using a commercial LLM hosted by a third party may not be the right fit for your use case. 

Cost and Token Uncertainty

Commercial LLMs are paid services. This means that you are using some form of subscription or paying for token-based usage. If your subscription is affordable and fixed, or your token usage is low, then this can be OK. However, if your subscription fees can change without notice, or your use cases consume a high number of tokens, you may require deployment options that give you more control over usage planning and costs. 

How does the Private Large Language Model Service Help

Confidentiality / Privacy

When deployed within a customer’s environment, the service is designed to help organizations keep prompts, code, and documents within infrastructure under their control. The Private AI Services Container runs in the customer’s data center, optionally on an air-gapped system, or in their own private cloud. Customers can also deploy the Private Large Language Model Service within their own tenancy on public clouds. These deployment options are designed to help customers maintain control of their data within their chosen environment. 

Cost and Token Certainty 

The Private Large Language Model Service uses open-weight LLMs that are customer-controlled and does not impose token-based usage charges. Customers provide and operate the infrastructure on which the service runs. Customers do not need to pay for the Oracle Private AI Services Container, nor do they have token limits. Organizations are not charged on a per-token basis for the usage of the Private Large Language Model Service. Customers do, however, need to provide their own hardware to run the Private AI Services Container on. The Private Large Language Model Service can use CPUs (Linux x86-64) or GPUs can provide faster inferencing than CPUs for many workloads. 

How does the Private Large Language Model Service work

The Private Large Language Model Service implements a popular REST interface (the OpenAI API) which means that existing clients which use OpenAI compatible APIs, should be able to work with the Private Large Language Model Service.  The key is to change the endpoint URL to point to the Private AI Services Container and to use the (non-paid) API_KEYs of the Private AI Services Container. 

Internally, the Private Large Language Model Service uses the llama.cpp and vLLM runtimes for LLM inferencing. Customers can optionally use the default mistralai/Ministral-3-3B-Reasoning-2512 LLM which ships with the large container, or they can download open-weight LLMs which work with the llama.cpp or vLLM runtimes. This list of open-weight LLMs are known to work with the Private Large Language Model Service. 

This means that many clients can be configured to work with the Private Large Language Model Service:

  • Chat clients like Oracle Private Agent Factory, OpenWebUI, AnythingLLM and OpenAI API Clients 
  • RAG clients like Oracle AI Vector Search, Oracle Private Agent Factory, LangChain and LlamaIndex
  • Coding Agents like OpenCode
  • Autonomous Agents like Hermes Agent

Security for the Private Large Language Model Service

The Private AI Services Container was designed with security in mind: 

  • The container runs in the customer’s infrastructure
  • Deployments can be configured without requiring external AI services
  • Can run in an air gapped environment
  • TLS 1.3 protects data in transit 
  • API keys authenticate requests
  • The PKCS#12 keystore protects certificates and credentials

Get Started

The bottom line: The Oracle Private AI Services Container enables external vector embedding generation supported by API-key authentication, TLS 1.3 encrypted communications, and keystore-based credential management, as well as vector index creation acceleration. It now adds LLM inferencing, Re-ranking and Text Classification without placing additional inferencing workloads on your database. 

You can download the Oracle Private AI Services Container directly from the Oracle Container Registry and get all of the setup details in the Private AI Services Container User’s Guide. 

Try it today! 

Related Blogs