OCI Inter-Region Cloud Connectivity Use Case - FastConnect and Remote Peering Connections

March 28, 2023 | 8 minute read
Raffi Shahabazian
Principal Cloud Network Architect
Text Size 100%:

Routing network traffic to meet the requirements of the business is always a topic that captures my interest.  There should be strict requirements to determine the flow of network traffic via a specific path and this is typically coupled with a routing protocol that can be manipulated to achieve the given outcome.

Note:  This blog assumes detailed knowledge of routing protocols (e.g. BGP and static), FastConnect circuits, extensive knowledge of Dynamic Routing Gateways (DRGs and its associated components such as attachments, route tables, and import distribution lists.), and Virtual Cloud Networks (VCNs and its associated networking components).

For example, most of us know that BGP can be manipulated to influence network traffic to prefer one network path over another.  We could provision two FastConnect circuits for a given Data Center and leverage as-path prepending and local preference to influence network traffic using one circuit over the other, which is a common and acceptable practice.  We could enable Equal Cost Multipathing (ECMP) to influence network traffic to use both circuits.  We may have two circuits connecting to two different regions from the same data center and we may want to route towards a given region preferring once circuit over the other.  This list can go on based on the business requirements and the technology available to the customer.

For this blog, we will focus on network connectivity between two OCI regions as opposed network connectivity between a traditional Data Center and OCI.  Let’s assume we have OCI presence in the OCI Ashburn and OCI Frankfurt regions.  In this case, we can leverage the Remote Peering Connection (RPC) feature that is available within the DRG to connect the two regions.  Assuming the RPC and associated components are configured appropriately, network traffic will flow between the two regions via the RPC link:

OCI Inter-Region - Remote Peering Connection
OCI Inter-Region - Remote Peering Connection

 

However, what if we wanted to create a FastConnect circuit between the two regions and dynamically exchange network layer reachability information using BGP to have dedicated bandwidth between the two regions while maintain the RPC as a back-up link?  Is this possible?  The short answer is yes, this is possible.  We can create a FastConnect circuit in each region and select one of the many preferred OCI FastConnect Partners to establish the network connectivity.  The basis of this blog will revolve around the following network design:

OCI Inter-Region - FastConnect and Remote Peering Connection
OCI Inter-Region - FastConnect and Remote Peering Connection

 

There are some things to consider influencing the traffic flow via the FastConnect in lieu of the RPC.  For example, what type of a FastConnect Partner should I select?  What will happen if routes are exchanged between the OCI regions using BGP?  Will the DRGs receive the routes and inject the routes into its route table?  How does the DRG prefer the path via the FastConnect in lieu of the RPC?  These are the questions I will address within this blog.

Oracle partners with a variety of FastConnect Partners on a per region basis.  We will need to select a partner that provides layer 3 connectivity between the regions.  Below is an example of what the FastConnect connectivity:

OCI Inter-Region - FastConnect and Remote Peering Connection
OCI Inter-Region - FastConnect BGP Overview

 

Notice the AS-PATH of the BGP advertisement.  OCI will advertise the OCI Ashburn region CIDRs originating from Oracle’s BGP ASN (31898), followed by the BGP ASN provided by the FastConnect circuit.  This provider in turn will advertise the CIDRs to the OCI Frankfurt region, at which point the advertisements will be dropped since the routes originate from the same BGP ASN as being received.  This is the default behavior for BGP as a part of its loop prevention mechanism. 

For this specific design, we will need to ensure the partner selected offers a solution that can provide the following requirements:

The ability to provide BGP as-override

Note:  The as-override feature can be applied on a Network Virtual Appliance (NVA) from the partner’s marketplace that the FastConnect virtual circuit that terminates on or the partner provides the layer 3 virtual circuit connection natively and performs as-override on behalf of you.

Note:  I will not go into the specific configurations on the NVA or how the partner will perform as-override.  For the NVA, I am assuming the knowledge of BGP and associated configurations are understood. In addition, if the partner is performing as-override on your behalf, I am assuming discussions have taken place and the partner is able to provide the necessary configuration items to meet this design option.

The resulting AS-PATH would be as follows:

OCI Inter-Region - FastConnect with BGP AS-override
OCI Inter-Region - FastConnect with BGP AS-override

 

At this point, we have achieved masking the Oracle BGP ASN with a single BGP ASN and the CIDRs are being advertised towards OCI and they are being learned as well.  However, we still have a route conflict in the OCI DRG route tables, with the RPC being the preferred path between regions:

OCI Inter-Region - Dynamic Routing Gateway Route Table
OCI Inter-Region - Dynamic Routing Gateway Route Table

 

Why is this occurring?  In order to understand the outcome, we must first understand the DRG’s preferred priority of importing dynamically learned CIDRs.

Per the DRG documentation, the following is the preferred order for importing active routes:

  • VCN
  • Virtual Circuit
  • IPSec Tunnel
  • Remote Peering Connection

Based on the DRG documentation, RPC connections are less preferred than FastConnect Virtual Circuits.  However, in our case, the RPC is more preferred.  Why is this occurring?  This is due to the CIDR advertisements being originated from VCNs and being advertised across RPCs. 

RPCs leverage the OCI backbone, therefore, we are not able to configure or influence BGP since RPC connections do not leverage BGP.  When the DRG compares CIDR originations, it sees one being learned from the RPC (no BGP ASNs in the AS-PATH) and the other being learned from the FastConnect Virtual Circuit (1 BGP ASN in the AS-PATH). 

Since the same CIDR is being learned via two paths, the path with the least BGP ASNs will win, in our case the RPC, which is a caveat to remember when designing inter-region cloud connectivity within the global OCI footprint.

So, how do we get around this?  While we could further fine-tune the NVA being used by the FastConnect partner or ask the partner honor more granular configurations if they are already honoring the as-override feature by advertising more specific CIDRs via the FastConnect circuit, we could also simplify the configuration by influencing the DRG route table associated with the VCN attachment with a static route to the remote destination VCN CIDR to have a less specific route via the DRG while dynamically importing the more specific route from the FastConnect virtual circuit.  Typically, the summarized (less specific) static route will encompass the remote VCN CIDR block.  It is worth noting that the default mechanism for advertisements for VCN attachments are the subnet, whether it is across the RPC or the FastConnect virtual circuits. 

If we leverage static routing in the DRG route table for the VCN attachment, we can configure the static route in lieu of importing from the RPC attachment.

Note:  Detailed knowledge and understanding of the DRG and its associated components such as route tables and import distribution lists are required to perform these routing configurations.  A list of documentation references is located towards the end of this blog.

Note:  These configurations need to be performed in both regions connecting to/from each other for symmetrical traffic flows.

Below is an example of the import route distribution list, which associated with the DRG route table, which is used by the VCN attachment.

OCI Inter-Region - Dynamic Routing Gateway Import Distribution List
OCI Inter-Region - Dynamic Routing Gateway Import Distribution List

 

Below is the less specific static route for the remote region VCN CIDR:

OCI Inter-Region - Dynamic Routing Gateway Route Table Static Route
OCI Inter-Region - Dynamic Routing Gateway Route Table Static Route

 

Below is an example of the DRG route table associated with the VCN attachment.  Notice we now have two active routes:  one for 192.168.1.0/22 (less preferred) and one for 192.168.1.0/24, which will automatically be more preferred due to standard rules of network routing:

OCI Inter-Region - Dynamic Routing Gateway Route Table
OCI Inter-Region - Dynamic Routing Gateway Route Table

 

With the more specific routes via the FastConnect virtual circuits being properly installed into the DRG route table, we can now leverage the FastConnect path over the RPC path.  Additionally, we could leverage the RPC connection to fallback in the event the FastConnect virtual circuit was unavailable for any given reason.

 

References:

Dynamic Routing Gateways

Remote Peering Connections

FastConnect

FastConnect Partners

Additional Resources:

DRG Technical Overview and Design Workshop YouTube Video

Raffi Shahabazian

Principal Cloud Network Architect


Previous Post

A routing scenario, transit traffic on OCI - Part Two

Andrei Stoian | 5 min read

Next Post


FAW Connectors - Augmenting Fusion Analytics Data With Google Analytics Data

Matthieu Lombard | 5 min read