HeatWave is the only fully-managed MySQL database service that combines transactions, analytics, machine learning, and GenAI services, without ETL duplication. HeatWave also includes HeatWave Lakehouse, allowing users to query data stored in object storage, MySQL databases, or a combination of both. Users can deploy HeatWave MySQL–powered apps on a choice of public clouds: Oracle Cloud Infrastructure (OCI), Amazon Web Services (AWS), and Microsoft Azure.
When using HeatWave, the service needs to access different resources within Oracle Cloud Infrastructure (OCI). HeatWave uses Resource Principals (link) to authenticate to and access these OCI resources like object storage and pretrained foundational models in OCI Generative AI Service.
In this blog we will describe how to configure Dynamic Groups and Policies so that HeatWave can access and read content from as well as write content back to object storage. To configure this access, there are three topics we will cover: (i) Identity Domains, (ii) Dynamic Groups, and (iii) Policies.
Note: before starting, note the OCID of your compartment and copy it in a text file. You will need it later.
Step 1—Identity Domains
If you already have an identity domain set up, or if you are using the Default Domain, then this step is optional, and you can skip to Step 2.
Within OCI Identity and Access Management (IAM), identity domains provide self-contained Identity-as-a-Service solutions that address numerous use-cases. Each identity domain can represent a unique user population with its own configurations and security settings, enabling you to create different access policies for various use-cases. For example, you might create an identity domain to manage employee access across cloud and on-premises applications. To get started and set up an identity domain, use the default identity domain in the root compartment in the tenancy or create another identity domain using the console or APIs. Then, add users or groups and configure access.
For more information, please see (https://docs.oracle.com/en-us/iaas/Content/identity-domains/identity-domains.htm)
You can access identity domains from the hamburger menu (Navigation menu) on the top left of your OCI page at any time (Fig 1.2).
From the home page of the Identity Domain section, you can see the domains available by clicking the ‘Domains’ link on the left (Fig 1.3).
At the root level you should see a default domain already available.
You might have created another domain at this level, or within the scope of a compartment, for example, the compartment where your HeatWave instances will run.
In the compartment named ILA in the tenancy where we are working, you can see that three identity domains have already been created (Fig 1.5). In the Identity Domain, you will create (or import or sync) users, configure security (SSO, MFA, etc…).
In the screenshot below, we have created an identity domain and selected the type as ‘Premium’ (Fig 1.6). You can select the domain type that most closely matches your requirements.
Note: This step is optional. You do not need to create an identity domain. You may select the default domain available.
Be sure to create (or import) users that will need to log in to OCI with this identity domain.
Step 2—Dynamic Groups
From within an identity domain, we will now create a Dynamic Group.
From the documentation (link), ‘Dynamic groups allow you to group HeatWave Service DB systems as principal actors, similar to user groups. You can then create policies to permit DB systems in the dynamic groups to make API calls against Oracle Cloud Infrastructure services, such as Certificates or Object Storage. Membership in the group is determined by a set of criteria you define, called matching rules.’
In this case, our identity domain is ‘lakehouse-test-domain’. No dynamic groups exist, yet (Fig 2.1).
Let us create one.
We have named the dynamic group ‘lakehouse-group’ and set a rule that allows this dynamic group access to all MySQL DB systems (and HeatWave systems, by extension) within a specific compartment by providing that compartment’s OCID. You can also define a dynamic group to be restricted to specific DB systems by providing those DB systems’ OCIDs (Fig 2.2).
ALL {resource.type='mysqldbsystem', resource.compartment.id='ocid1.compartment.oc1..aaaaeew....ewq'}
This is how our dynamic group looks (Fig 2.3):
Step 3—Policies
Simply put, policies (documentation link) define what your groups or dynamic groups can and cannot do. In our case, we want to define a policy that allows the dynamic group created above to access user buckets and their contents. Additionally, if we want to use HeatWave to export query results to object storage, we will have to define our policy to allow the dynamic group to write to object storage as well.
From the Identity Domain home page, select ‘Policies’.
Then select the compartment you want to create the policy in and click ‘Create Policy’.
This shall open a page where you can create a policy using a ‘Policy Builder’.
Each policy will have one or more policy statements.
To have HeatWave Lakehouse read data from object storage, at a minimum you need to specify that the dynamic group you created has access to read buckets and objects in object storage in the specified compartment.
For more information on the syntax for policy statements, see the documentation (link).
Select ‘Storage Management’ from the Policy use cases dropdown and then select ‘Let users download objects from Object Storage buckets’. If you want to be able to write data back to object storage, create another rule and select ‘Let users write objects to Object Storage buckets’ (Fig 3.2).
You then need to select the identity domain (lakehouse-test, in this case), the dynamic group (lakehouse-group, in this case), and the compartment (ILA, in this case).
The policy statements generated will be shown (Fig 3.3).
Allow dynamic-group 'lakehouse-test'/'lakehouse-group' to read buckets in compartment ILA
Allow dynamic-group 'lakehouse-test'/'lakehouse-group' to read objects in compartment ILA
Once you click Create, the policy will be created and all its policy statements displayed (Fig 3.4).
Note: you can also switch to a manual editor by clicking on the ‘Show manual editor’ radio button. When switching between the manual editor and policy builder, verify the policy statements. If you are creating a policy that grants access to a compartment that is not the compartment you are creating the policy in, you may get an error. In this case, you may have to switch to the manual editor and define your policy.
Congratulations! With these policies created, you are now ready to connect to your HeatWave system and read data from object storage. Try HeatWave Lakehouse on OCI today!
More Information:
You can find more information about HeatWave at:
General Information
Blog posts
- Exporting query results to object storage with HeatWave (link)
- MySQL HeatWave Lakehouse on AWS (link)
- Building ML models using data in object storage with HeatWave AutoML (link)
- Performance demos at Oracle CloudWorld: MySQL HeatWave Lakehouse vs. Snowflake, Redshift, BigQuery, and Databricks (link)
Videos and demos
- Videos (link)
- Oracle CloudWorld Keynote with HeatWave Lakehouse (link)
- HeatWave Lakehouse on AWS (link)
- Data Warehouse and Lakehouse Analytics with HeatWave Lakehouse (link)