Query Databricks Native Iceberg Tables from Oracle Autonomous AI Database

This post was written in collaboration with the Oracle Autonomous AI Database development team, with special thanks to Ming Yang for his guidance and technical contributions.

In the previous post, we showed how Oracle Autonomous AI Database can query a Databricks Delta table through UniForm. That pattern is useful when Databricks remains Delta-first, while Oracle reads Iceberg-compatible metadata generated for the Delta table.

In this post, we take the next step: the table is created as native Apache Iceberg in Databricks from the beginning.

This pattern is for teams that want Apache Iceberg to be the open table format across engines. Databricks creates, governs, and maintains the Iceberg table in Unity Catalog. Oracle Autonomous AI Database reads the same table through the Databricks Unity Catalog Iceberg REST interface, without copying data or creating a separate export pipeline.

Databricks documents Unity Catalog access for Apache Iceberg clients through the Iceberg REST catalog, and managed tables can support Apache Iceberg as a storage format. Oracle Autonomous AI Database supports querying external data through credentials and external tables using DBMS_CLOUD.

Use case

A data engineering team owns a curated customers table in Databricks. Unlike the previous UniForm example, this table is not a Delta table. It is created as an Apache Iceberg table from day one.

Databricks remains the system where the table is created, loaded, governed, and maintained. An Oracle application team wants to query the same customer data from Oracle Autonomous AI Database using regular SQL.

The pattern is simple:

1. Create a native Iceberg table in Databricks.
2. Grant a Databricks service principal access to the table.
3. Store Databricks and object storage credentials in Oracle Autonomous Database.
4. Create an Oracle external table that points to the Databricks Iceberg table.

Figure 1. Oracle Autonomous Database queries a Databricks native Iceberg table through Unity Catalog Iceberg REST.

What we build

In Databricks, we create a native Iceberg table called customers_iceberg.

In Oracle Autonomous Database, we create an external table called CUSTOMERS_ICEBERG that maps to the Databricks table data_lake_pm.default.customers_iceberg.

Oracle retrieves the table metadata through the Databricks Iceberg REST endpoint and reads the underlying Iceberg table files from cloud object storage.

Databricks setup

Step 1. Create the native Iceberg table

Run the following in Databricks SQL.

CREATE TABLE data_lake_pm.default.customers_iceberg (
  customer_id BIGINT,
  first_name  STRING,
  last_name   STRING,
  email       STRING,
  city        STRING,
  state       STRING,
  country     STRING,
  created_at  TIMESTAMP
)
USING ICEBERG;

INSERT INTO data_lake_pm.default.customers_iceberg VALUES
  (1, 'John', 'Smith', 'john.smith@example.com', 'Seattle', 'WA', 'US', current_timestamp()),
  (2, 'Mary', 'Johnson', 'mary.johnson@example.com', 'San Francisco', 'CA', 'US', current_timestamp()),
  (3, 'Alex', 'Brown', 'alex.brown@example.com', 'Austin', 'TX', 'US', current_timestamp());

The important part is USING ICEBERG. In this example, the table is not a Delta table with Iceberg compatibility metadata. It is created as a native Iceberg table.

Step 2. Grant access to the service principal

Oracle will connect to Databricks using a Databricks service principal. Grant that service principal access to the catalog, schema, and table.

GRANT USE CATALOG ON CATALOG data_lake_pm
TO `databricks-service-principal-application-id`;

GRANT USE SCHEMA ON SCHEMA data_lake_pm.default
TO `databricks-service-principal-application-id`;

GRANT SELECT ON TABLE data_lake_pm.default.customers_iceberg
TO `databricks-service-principal-application-id`;

GRANT EXTERNAL USE SCHEMA ON SCHEMA data_lake_pm.default
TO `databricks-service-principal-application-id`;

Databricks external access requires external data access to be enabled on the metastore, and external clients need the relevant Unity Catalog privileges, including EXTERNAL USE SCHEMA. (Databricks Docs)

Oracle Autonomous Database setup

Oracle Autonomous Database needs two credentials:

1. UNITY_OAUTH, used to authenticate to Databricks Unity Catalog Iceberg REST.
2. AZURE_BLOB_CRED, used to access the cloud object storage files behind the Iceberg table.

Step 1. Create credentials

Run the following in Oracle Autonomous Database.

BEGIN
  DBMS_CLOUD.CREATE_CREDENTIAL(
    credential_name => 'UNITY_OAUTH',
    username        => 'databricks-service-principal-application-id',
    password        => 'databricks-oauth-client-secret'
  );

  DBMS_CLOUD.CREATE_CREDENTIAL(
    credential_name => 'AZURE_BLOB_CRED',
    username        => 'azure-storage-account-name',
    password        => 'azure-storage-account-key-or-sas-token'
  );
END;
/

Step 2. Create the Oracle external table

This external table maps Oracle table CUSTOMERS_ICEBERG to the Databricks native Iceberg table data_lake_pm.default.customers_iceberg.

BEGIN
  DBMS_CLOUD.CREATE_EXTERNAL_TABLE(
    table_name      => 'CUSTOMERS_ICEBERG',
    credential_name => 'AZURE_BLOB_CRED',
    format          => '
{
  "access_protocol": {
    "protocol_type": "iceberg",
    "protocol_config": {
      "iceberg_catalog_type": "unity",
      "rest_catalog_endpoint": "https://databricks-workspace-host/api/2.1/unity-catalog/iceberg-rest",
      "rest_catalog_prefix": "catalogs/data_lake_pm/",
      "rest_authentication": {
        "rest_auth_cred": "UNITY_OAUTH",
        "rest_auth_endpoint": "https://databricks-workspace-host/oidc/v1/token"
      },
      "table_path": ["default", "customers_iceberg"]
    }
  }
}'
  );
END;
/

Databricks documents the Unity Catalog Iceberg REST endpoint as /api/2.1/unity-catalog/iceberg-rest. (Databricks Docs)

Step 3. Query the Databricks Iceberg table from Oracle

SELECT customer_id, first_name, last_name, city, state, country
FROM customers_iceberg
ORDER BY customer_id;

Expected result:

CUSTOMER_ID  FIRST_NAME  LAST_NAME  CITY           STATE  COUNTRY
1            John        Smith      Seattle        WA     US
2            Mary        Johnson    San Francisco  CA     US
3            Alex        Brown      Austin         TX     US

Oracle users can now query the Databricks native Iceberg table with regular SQL.

Why this matters

The previous UniForm pattern is ideal when a Databricks team already owns Delta tables and wants to expose them to Iceberg clients without rewriting data.

This native Iceberg pattern is different. It is for customers who want Iceberg itself to be the primary table format.

With this approach:

Conclusion

This post extends the previous Delta UniForm example.

With UniForm, Databricks keeps the table as Delta and publishes Iceberg-compatible metadata for external engines.

With native Iceberg, Databricks creates and manages the table as Apache Iceberg from the beginning. Oracle Autonomous Database then reads that same governed Iceberg table through Unity Catalog Iceberg REST.

For customers standardizing on Apache Iceberg as the open table format across engines, this provides a clean interoperability path: Databricks manages the Iceberg data product, Unity Catalog governs access, and Oracle Autonomous Database queries the table without copying or converting the data.

References

Databricks documentation: Access Databricks tables from Apache Iceberg clients. (Databricks Docs)

Databricks documentation: Unity Catalog managed tables and Apache Iceberg support. (Databricks Docs)

Databricks documentation: Enable external data access to Unity Catalog. (Databricks Docs)

Oracle documentation: Query external data with Autonomous Database. (Oracle Docs)

Oracle documentation: DBMS_CLOUD subprograms. (Oracle Docs)