A routing scenario, is an asymmetric traffic path allowed in OCI?

Last week I was requested to perform a networking architecture review from the routing and security perspective and to help make it work. It was an interesting use case and I thought it is important to share the case with you.

The networking topology that I will present is supposed to permit an asymmetric traffic path withing the OCI network, but the traffic did not worked as expected. So, a normal question would be if OCI does support asymmetric traffic path? If yes, how we should configure different networking resources for the asymmetric traffic to work? Interesting questions, isn't it? Let's find out.

The networking topology in place is listed below:

As we might observe on the above networking diagram, the traffic is indeed asymmetric since the packet 3 (in green) is not following the same path at packet 2 (in red) and the packet 4 (in green) is not following the same path as packet 1 (in red).

Now, let's look at how the route tables for each subnet are configured. 

1) LBaaS subnet:

2) Firewall VNIC 1 subnet:

3) Firewall VNIC 2 subnet:

4) Backend subnet:

It seems that the correct routing configuration is in place.

Note 1: The NSGs configuration are not listed here but the configuration is in place and the desired traffic is permitted.
Note 2: The firewall routing configuration is in place for the traffic to be routed over the respective VNICs, also, it is working in the routed mode and not as a stateful firewall.

For now, let's try a http connection from the Internet host at 129.146.111.94:

Definitively the connection is not established. What can be the root cause of this issue, since all the routing is in place?

I'm propossing to have a packet capture on the VNIC 1 and VNIC 2 at the same time and see what is happening:

VNIC 1 packet capture:

VNIC 2 packet capture:

.

The answer is included in the two packet captures. Let's look at the source MAC address of the packet received on the VNIC 1 (that is the MAC address of the LBaaS, it ends with 16:0d. After the packet is sent out over VNIC 2 the source MAC address is different and ends with 99:76. That is correct, since VNIC 2 is the output interface the source MAC address will be the MAC address of the egress interface.

Now, at the IP header, the source IP address of 10.0.0.14 is the private IP address assigned to the LBaaS which is the same on VNIC 1 and VNIC 2 packet capture as source IP address.

Remember that by default, a VNIC looks at the source and destination listed in the header of each network packet. If the VNIC isn't the source or destination, then the packet is dropped. So, our packet is dropped because the source IP address is 10.0.0.14 and it is not the IP address assigned to the VNIC 2 interface. The solution is to enable the "Skip source/destination check" option. Let's do that on the VNIC 2:

Now, let's give it another try and see if now we can reach the web application:

The connection is working just fine and we can confirm that OCI can support asymmetric traffic path. This really means we have solved the problem? Not really! Generally speaking, the asymmetric traffic in most of the cases is not desired and hard to track. More than that, the above presented networking architecture is not secure, as we can see no firewall is in place between the traffic received from the Internet and the LBaaS. Then a question might be, how we can transform it in a secure and scalable solution? Do we have the answer? Yes, the answer is in this blog post OCI Network Firewall - Securing the LBaaS traffic