The Oracle Cloud Infrastructure (OCI) Data Science model deployment allows customers to deploy machine learning (ML) models as HTTP endpoints on OCI. You can then seamlessly integrate these endpoints into applications to serve predictions with new input data. Previously, all inference endpoints were either public-facing or accessible through the Oracle internal network. While secured with Oracle Identity and Access Management (IAM) authorization, these endpoints might not meet the stringent data protection requirements of highly regulated industries like finance and healthcare.
To address this need, OCI Data Science is introducing private endpoints for model deployment. These endpoints help ensure that all traffic remains within your virtual cloud network (VCN), eliminating exposure to the public internet. This advancement is particularly significant for organizations handling sensitive data, as it provides an extra layer of security, reduces latency, and simplifies network architecture.
Private endpoints for OCI Data Science model deployment deliver the following crucial benefits, particularly for organizations operating in regulated or security-sensitive industries:
Enhanced security: By eliminating exposure to the public internet, private endpoints drastically reduce the risk of data breaches and unauthorized access, helping to ensure data integrity during inference calls.
Regulatory compliance: They support stringent compliance requirements by keeping data traffic confined within designated network boundaries, a necessity for industries like finance, healthcare, and government.
In hybrid cloud setups, organizations seamlessly combine on-premises infrastructure with OCI's scalable and flexible resources. However, securing data traffic between on-premises systems and the cloud is essential. OCI private endpoints help ensure that sensitive inference traffic remains private, traversing only within the customer’s secure VCN. This setup not only enhances data security but also supports compliance mandates while enabling businesses to use OCI’s advanced capabilities.
In multitenant configurations, where multiple departments or customers share infrastructure, maintaining isolation and data security is paramount. OCI private endpoints provide dedicated, private connections for each tenant’s resources, supporting traffic between the tenant’s VCN and model deployment services remains segregated and secure. This architecture helps ensure confidentiality and prevents the risk of cross-tenant data exposure, offering peace of mind to organizations managing sensitive workloads.
To begin using private endpoints for model deployment, use the following steps:
Modify the usage limit for private endpoints: By default, the usage limit for Data Science private endpoints is set to 0 in your OCI tenancy. You must request a limit increase through the Oracle Cloud Console to enable this feature.
To create a private endpoint, use the following steps:
To use a private endpoint for deploying a model, use the following steps.
You can create and configure a private endpoint for model deployments created inside Data Science AI Quick Actions. To enable private endpoint usage with AI Quick Actions, create a notebook session with custom networking. Ensure that the VCN and subnet of the notebook session are the same as those of the private endpoint. For detailed guidance, follow the step-by-step instructions outlined earlier for creating a private endpoint.
After the large language model (LLM) is deployed using the AI Quick Actions Console (AQUA), you can input text in the prompt and modify the model parameters to generate a response.
To securely invoke your model deployment, use the private endpoint’s fully qualified domain name (FQDN) URL. By utilizing the FQDN, communication between your application and the deployed model remains confined to a private network, enhancing both security and latency. For more information please refer to the following document.
Private endpoints empower businesses to deliver secure, low-latency, and compliant model deployments in Oracle Cloud Infrastructure, helping ensure seamless integration into highly regulated and sensitive environments.
For more information, see the following resources:
Nishanth has been working as a senior software engineer primarily working on Data Science and Machine learning. He brings in over 9+ years of work experience. Nishanth has a Masters degree in Computer Science from University of Arizona.
Nabeel is a Director at OCI Data Science, he leads the software engineering efforts for Model Deployment. He brings with over 15 years of industry experience. He has a doctoral degree from Purdue University in Computer Engineering and a Masters degree from New Jersey Institute of Technology.
Next Post