Introducing Private Endpoints in OCI Data Science: Enhanced Security for Sensitive Inference Workloads

February 11, 2025 | 6 minute read
Wendy Yip
Senior Product Manager, OCI Data Science
Nishanth Prakash
Senior Member of Technical Staff
Nabeel Al Saber
Director AI/ ML Platform
Text Size 100%:

The Oracle Cloud Infrastructure (OCI) Data Science model deployment allows customers to deploy machine learning (ML) models as HTTP endpoints on OCI. You can then seamlessly integrate these endpoints into applications to serve predictions with new input data. Previously, all inference endpoints were either public-facing or accessible through the Oracle internal network. While secured with Oracle Identity and Access Management (IAM) authorization, these endpoints might not meet the stringent data protection requirements of highly regulated industries like finance and healthcare.

To address this need, OCI Data Science is introducing private endpoints for model deployment. These endpoints help ensure that all traffic remains within your virtual cloud network (VCN), eliminating exposure to the public internet. This advancement is particularly significant for organizations handling sensitive data, as it provides an extra layer of security, reduces latency, and simplifies network architecture.

Key Benefits of Using Private Endpoints

Private endpoints for OCI Data Science model deployment deliver the following crucial benefits, particularly for organizations operating in regulated or security-sensitive industries:

  • Enhanced security: By eliminating exposure to the public internet, private endpoints drastically reduce the risk of data breaches and unauthorized access, helping to ensure data integrity during inference calls.

  • Regulatory compliance: They support stringent compliance requirements by keeping data traffic confined within designated network boundaries, a necessity for industries like finance, healthcare, and government.

Common Customer Use Cases

Hybrid Cloud Architectures

In hybrid cloud setups, organizations seamlessly combine on-premises infrastructure with OCI's scalable and flexible resources. However, securing data traffic between on-premises systems and the cloud is essential. OCI private endpoints help ensure that sensitive inference traffic remains private, traversing only within the customer’s secure VCN. This setup not only enhances data security but also supports compliance mandates while enabling businesses to use OCI’s advanced capabilities.

Architecture diagram for a hybrid cloud deployment
 Fig 1: Architecture diagram for a hybrid cloud deployment

Multitenant Environments within OCI

In multitenant configurations, where multiple departments or customers share infrastructure, maintaining isolation and data security is paramount. OCI private endpoints provide dedicated, private connections for each tenant’s resources, supporting traffic between the tenant’s VCN and model deployment services remains segregated and secure. This architecture helps ensure confidentiality and prevents the risk of cross-tenant data exposure, offering peace of mind to organizations managing sensitive workloads.

Architecture diagram for the deployment of a multitenant environment in OCI
Fig 2: Architecture diagram for the deployment of a multitenant environment in OCI

Getting Started with Private Endpoints

To begin using private endpoints for model deployment, use the following steps:

Modify the usage limit for private endpoints: By default, the usage limit for Data Science private endpoints is set to 0 in your OCI tenancy. You must request a limit increase through the Oracle Cloud Console to enable this feature.

  1. Check regional availability: Currently, support for model deployment private endpoints is available in the commercial public OC1 region. To use this feature in other OCI realms, you can submit a service request.
  2. Create a private endpoint: Private endpoints for the Data Science service can be created directly in the Oracle Cloud Console. Follow the detailed, step-by-step instructions provided in the OCI documentation to set it up.
  3. When creating a private endpoint, you can specify its purpose as either model deployment or notebook usage. However, you can’t share a private endpoint between these two functions. A single private endpoint can support multiple model deployments, allowing for scalable and efficient use of resources.

Creating a Private Endpoint

To create a private endpoint, use the following steps:

  1. Log into the Oracle Cloud Console.
  2. Under Data Science, select Private endpoints.
  3. Select Create private endpoint.
  4. (Optional) In the Create private endpoint panel, enter a name to identify the private endpoint.
  5. (Optional) Enter a description.
  6. Select the VCN created to provide private access.
  7. Select the subnet that has the private endpoint you want to use. 
  8. (Optional) Enter a subdomain for the private endpoint up to 60 characters.
  9. For Resource Type, select MODEL_DEPLOYMENT from the menu.
  10. Select Create to create the private endpoint.
Creating a private endpoint in the Oracle Cloud Console
Fig 3: Create a private endpoint in the Oracle Cloud Console

Using a Private Endpoint for Model Deployment

To use a private endpoint for deploying a model, use the following steps.

  1. In the Console, navigate to the Model Deployments page.
  2. Select the compartment in which you want to create a model deployment.
  3. Select the Create Model Deployment button to navigate to the page where we input the model deployment configuration to create a model deployment.
  4. Configure all necessary fields needed for creating a model deployment.
  5. In Networking Resources, under Endpoint Type, select the Private endpoint box.
  6. Select the private endpoint from the menu. Select the private endpoint according to your need. 
    Creating a model deployment
    Fig 4: Create a model deployment with private endpoint
     
  7. Select Create. Wait for model deployment to become Active. 
The created private endpint
Fig 5: Select private endpoint for model deployment

 

Integration with AI Quick Actions

You can create and configure a private endpoint for model deployments created inside Data Science AI Quick Actions. To enable private endpoint usage with AI Quick Actions, create a notebook session with custom networking. Ensure that the VCN and subnet of the notebook session are the same as those of the private endpoint. For detailed guidance, follow the step-by-step instructions outlined earlier for creating a private endpoint.

After the large language model (LLM) is deployed using the AI Quick Actions Console (AQUA), you can input text in the prompt and modify the model parameters to generate a response.

Using model deployment with private endpoint in AI Quick Actions
Fig 6: Use model deployment with private endpoint in AI Quick Actions

Secure Model Invocation

To securely invoke your model deployment, use the private endpoint’s fully qualified domain name (FQDN) URL. By utilizing the FQDN, communication between your application and the deployed model remains confined to a private network, enhancing both security and latency. For more information please refer to the following document.

Invoke model deployment with private endpoint
Fig 7:  Invoke model deployoment with the private endpoint's fully qualified domain name (FQDN) URL

Conclusion

Private endpoints empower businesses to deliver secure, low-latency, and compliant model deployments in Oracle Cloud Infrastructure, helping ensure seamless integration into highly regulated and sensitive environments.

For more information, see the following resources:

 

Wendy Yip

Senior Product Manager, OCI Data Science

Nishanth Prakash

Senior Member of Technical Staff

Nishanth has been working as a senior software engineer primarily working on Data Science and Machine learning. He brings in over 9+ years of work experience. Nishanth has a Masters degree in Computer Science from University of Arizona. 

Nabeel Al Saber

Director AI/ ML Platform

Nabeel is a Director at OCI Data Science, he leads the software engineering efforts for Model Deployment. He brings with over 15 years of industry experience. He has a doctoral degree from Purdue University in Computer Engineering and a Masters degree from New Jersey Institute of Technology. 


Previous Post

Agentic AI: A Game Changer for the Construction Industry

Barry Mostert | 4 min read

Next Post


AIStore on OCI with NVIDIA: Scalable, High-Performance AI Storage

Cristian Vlad | 2 min read
Oracle Chatbot
Disconnected