VCN Peering for ADBs with Private Endpoints and Cross-Region Autonomous Data Guard

Following up my post announcing and introducing you to Cross-Region Autonomous Data Guard (X-ADG), I'd like to provide additional detail about the networking setup required if your database with X-ADG enabled is behind a private endpoint (PE). While this is a more advanced, administrative topic, kudos on using private endpoints as it is the most secure setup for your Autonomous Database (ADB). 

Here, I aim to simplify and explain the Virtual Cloud Network (VCN) peering configuration required when connecting to a remote private database lying in a different region than your client application (such as when you failover / switchover your database to a remote region standby, while your mid-tier application still lies in its primary region). Private database hostnames (mapped in the ADB wallet) are not resolvable across regions, since the remote database’s hostname refers to a private IP address in that remote region VCN. To do this, we will need to:

1. Establish remote VCN peering between your two region's VCNs, to be able to send private traffic between them, using a Remote Peering Connection (RPC) within a Dynamic Routing Gateway (DRG)

2. Update your VCN's DNS Resolver to forward one region's ADB hostname across to the other, where a listener in the remote region will translate the hostname into the remote ADB's private IP address

Note: If you expect to always switchover / failover your complete stack (mid-tier, application etc.) to the same remote region, when you do so your ADB instance, you would not need to peer your VCNs.

Additionally, the setup below peers together two regions' VCNs and is intended for connections to your Autonomous Database (ADB) from applications running in Oracle Cloud. To connect to one of the ADB instances from an on-premise application, you will need to set up transit routing.

Below, I provide steps to create such a peered networking setup. I assume you have an autonomous database instance with a remote standby enabled, as well as a public compute instance (VM) or application you will be using to connect to your database instances. I provide example steps to create and configure Virtual Cloud Networks (VCNs), subnets, security lists and Network Security Groups (NSGs) needed to enable private endpoints on Autonomous Databases, before peering your two VCNs. You will perform most of the steps in both the Primary region and the Remote region. You may skip some of the initial steps if you already have private endpoints set up on your databases, but I urge you to read through them to make sure you aren’t missing any of the requirements (such as non-overlapping CIDR blocks).

The example regions I use below are:

Region 1 (Primary/Source Region): Mumbai

Region 2 (Remote/Standby Region): Hyderabad

1. Create 2 VCNs, one in the source and one in the remote region, with non-overlapping CIDR blocks.
   • I used the CIDR block 10.0.0.0/16 in Mumbai
   • and 10.2.0.0/16 in Hyderabad
     Note: Leave the default settings of using DNS hostnames checked. If this is unchecked, users would be expected to manually replace in IP address in their database wallet, instead of resolvable hostnames.
     
      
   
   
    
2. Create a private subnet in each VCN. These are where our primary and standby ADB instances private endpoints will lie. You may keep the default options for route tables, DHCP and security lists.
   • I used the subset of the total CIDR block 10.0.2.0/24 in Mumbai (leaving part of the block 10.0.1.0/24 available for a public subnet that will contain your internet-facing VM or application)
   • I used the full CIDR block 10.2.0.0/16 in Hyderabad. You may leave part of the CIDR block for a public subnet here too, if required
     
      
   
    
3. Within each of your two VCNs, proceed to update your default security lists
   
   In the primary region's (Mumbai) VCN security list, I create an:
   
     - ingress rule with source type CIDR, to allow connections in from that IP range.

           - I keep the default rule for secure SSH protocol traffic into my VCN from any IP address with the CIDR 0.0.0.0/0, so as to connect to a VM or application running in a public subnet.
           - I have allowed incoming TCP connections from my remote (HYD) VCN, with CIDR block 10.2.0.0/16 for destination port 1522, which is the port for ADB-S with PE.
           - I have allowed incoming UDP connections from remote (HYD) VCN, with CIDR block 10.2.0.0/16 for destination port 53, which is the port the OCI DNS resolver uses (more on this in Step 10)
   
     - egress rule with source type CIDR to allow connections out to the listed IP ranges. In this example, in the Mumbai VCN, I keep the default rule of all traffic with CIDR 0.0.0.0/0

   
   In the remote region (Hyderabad) VCN security list, I create an :
   
     - ingress rule with source type CIDR, to allow connections in from that IP range.
           - I keep the default rule for secure SSH protocol traffic into my VCN from any IP address with the CIDR 0.0.0.0/0, so as to connect to a VM or application running in a public subnet.
           - I have allowed incoming TCP connections from my primary (Mumbai) VCN, with CIDR block 10.0.0.0/16 for destination port 1522, which is the port for ADB-S with PE.
           - I have allowed incoming UDP connections from my primary (Mumbai) VCN, with CIDR block 10.0.0.0/16 for destination port 53, which is the port the OCI DNS resolver uses (more on this in Step 10)
   
     - egress rule with source type CIDR to allow connections out to the listed IP ranges. In this example, in the Mumbai VCN, I keep the default rule of all traffic with CIDR 0.0.0.0/0

    
   You may add additional rules, such as to expose your VCN to certain IP addresses only or allow ORDS/HTTPS access by opening up TCP port 443, as your business requires.
   
   
   
   
    
4. Within each of the two region's VCNs, Create a Network Security Group (NSG), which are firewalls rules required for ADB instances with private endpoints. As in the previous step, you may add additional ingress and egress security rules for your private databases. In this example, I do not need any specific rules for ADB-S to block traffic.
   
   
   
    


5. Within each of the two regions, create a Dynamic Routing Gateway (DRG) (ie. your virtual router). From within each region’s DRG, create a VCN attachment to the respective region’s VCN. (Note: If you create this attachment from within the VCN, it is instead named a "DRG attachment")
   
   When creating this VCN attachment, under Advanced, you will notice by default this attachment will add routes appropriately to the DRG route table.
   
   As mentioned above, for transit routing to on-premise applications, you will also need to add the necessary routes to your VCN route table.
   
   
   
    
6. In each of the two region’s DRGs you created, click on Remote Peering Connections Attachments, and create a Remote Peering Connection (RPC), which will be used to peer your two VCNs together. Creating the RPC will also automatically create the RPC attachment to the DRG, as well as the DRG route table for the RPC attachment.
   
    
    
7. This step will only need to be performed in one of the regions. From your primary region’s RPC hit "Establish Connection". This is where we are remote peering your primary region’s VCN to the remote region’s VCN. Select your remote region from the drop-down and enter the OCID of the remote region’s RPC into the dialog (which can be copied from the console of the remote RPC).
   
   Once successful, your two VCNs are peered! If you peek into the RPC of the remote region, you should see the connection established there too.
   
   
   
    
8. Now, creating the RPC attachment should also automatically create Auto-Generated DRG routing table entries with the appropriate routes to the RPC in your DRG. To check whether your DRG routing table has been properly updated for the RPC attachment, go to your DRG → DRG Route Tables → Autogenerated Drg Route Table for RPC, VC, and IPSec attachments → Click "Get All Routes". 
   
   If properly set up, you will see the current region's CIDR block next hop as your local VCN, as well as the remote region's subnet CIDR block's next hop as the RPC (as in the below screenshot). This tells the DRG where to route traffic intended for that particular CIDR block.
   
   If you do not find this in your setup, you will need to generate these rules. On your DRG's console, click on "Autogenerated Import Route Distribution for VCN Routes". Select the available statement and click "Manage Statements". Here, we want to add "Another Statement" to your existing ones, and select the route generator option to generate routes from the attachment type "Remote Peering Connection". Once you hit save, the route rules should now update automatically to include the appropriate hops to your RPC in the remote region. You may check this by following the steps in the paragraph above.
   
   Note, this RPC route table should be setup the same way in each region. Follow this same procedure in your remote region too.
   
   
   
    
9. With remote peering set up, if you haven’t already, enable private endpoints in both your Primary ADB instance and your remote Standby instance. Select the private subnets with NSGs you created above, in each region (so the databases are not exposed to the public internet at all).
   
   
   
    
10. Note down the domain hostname of each of the two databases (primary and remote), listed as the Private Endpoint URL on the database console.
    
    In my example, my Mumbai private ADB's hostname is:
                                         x2ic8wriy.adb.ap-mumbai-1.oraclecloud.com
    and my Hyderabad (remote) private ADB's hostname is: 
                                         x2ic8wqa.adb.ap-hyderabad-1.oraclecloud.com
    
    
    
     
11. In each of your two region's VCN console, go to the DNS Resolver. Create both a Listening endpoint and a Forwarding endpoint in each VCN's private subnet. You can let the defaults be, and it will pick an unused IP address. 
    
    
    
    
    
     
12. Within the DNS resolver in each of the two regions, we will now create domain forwarding rules to the other region's endpoint. This is what will help resolve the respective region's domain hostname to the private IP address of that region.
     
    • In the primary region, create a rule
      • Use the remote region's hostname as the "Domain" (eg. x2ic8wnr.adb.ap-hyderabad-1.oraclecloud.com)
      • Use the primary region’s Forwarder endpoint
      • Enter the destination IP as the IP address of the listener in the remote region.
        (eg. 10.2.2.152)
    • In the remote region, create a rule
      • Use the primary region's hostname as the "Domain"
        (eg. x2ic8wriy.adb.ap-mumbai-1.oraclecloud.com)
      • Use the remote region Forwarder endpoint
      • Enter the destination IP as the listener of the primary region
        (eg. 10.0.2.79)
    
    
    
     
13. Above, we told the DRG where to route the private CIDR traffic meant for the remote region - to the RPC. We need to also make sure we route the previous-hop appropriately too - Let's check that our VCN’s default route table is routing traffic intended for the private subnet in the remote region to the DRG. Go to your primary region’s VCN Console → "Default Route Table" and check whether the remote region's private subnet CIDR block traffic is routed to the DRG, to send it over to the other side. If not, Add the necessary route rule.
    Make sure you do the same in the same in the other region as well.
    
    
     

We're done! Our tunnel is now ready to connect to the remote region's private database from the primary region. Access your primary region's public VM or application and download your ADB wallet to it.

If all went smoothly, our VCNs are now peered, hostnames are properly forwarded across the VCNs and we should be able to use this downloaded wallet and connect to your database instance both in the Primary region, as well as in the Remote region after performing a switchover or failover. Your remote region's hostname will now be resolved to the remote database’s private IP address, and you will be able to connect to your remote private database from your primary region and vice versa!