As described in our blog about Implementing Oracle Database for Azure with a hub-and-spoke network, Path MTU Discovery (PMTUD) must be enabled when using Oracle Database service for Azure (OracleDB for Azure) connected to a hub-and-spoke network. However, this setup might not be feasible for the following reasons:

  • In Azure, you can’t allow only specific internet control message protocol (ICMP) types, and you don’t want to allow all ICMP traffic for security reasons.
  • Not all multinode third-party firewalls allow session persistence. So, ICMP traffic can misroute between firewall nodes, and PMTUD packets are dropped.
  • In the Azure Firewall service, you can’t enable session persistence, so PMTUD packets can be dropped for the same reason.

If you’re affected by these limitations, manually lowering maximum transmission unit (MTU) settings is considered a workaround because then PMTUD isn’t required. Manually lowering MTU has no performance impact for your connections from your Azure network compared to using PMTUD. Because the maximum segment size (MSS) on the tunnel between Azure and Oracle Cloud Infrastructure (OCI) is limited, manually lowering MTU has the same result that PMTUD achieves.

This blog describes the details of various options for this workaround.

Experienced behavior

PMTUD packets not reaching their destination results in hangs when trying to connect from a resource in a spoke VNet to the OracleDB for Azure resource. Depending on the tools and protocols used, connection failures take various forms. For example, when using SQL*Plus the connection times out, as illustrated in the following figure:

A screenshot of the output for a timed out SQL*Plus connection.
Figure 1: Example of a timed-out SQL*Plus connection

As another example, a direct SSH connection to the database host also times out.

A screenshot of the output for a timed out direct SSL connection.
 Figure 2: Example of timed-out SSH connection

However, we can confirm that connectivity does exist by successfully initiating a telnet session to port 1521 on the database host:

A screenshot of a successful telnet connection.
Figure 3: Example of a successful telnet connection

Resolution

The standard solution to these issues is to enable PMTUD as explained in our blog about Implementing OracleDB for Azure with a hub-and-spoke network. Manually lowering MTU settings is considered a workaround if this method isn’t feasible because PMTUD isn’t required. To achieve this goal, you can either set a fixed session data unit (SDU) size for SQL*Net or set a fixed MTU size for network routes. You can make these changes either on individual clients or centrally on the database server, so you have four resolution options:

 

SQL*Net

MTU

Database server

Set fixed SDU size for SQL*Net in database network configuration

Set fixed MTU size on the database server

Client connection

Set fixed SDU size for SQL*Net connect strings on clients

Set fixed MTU size on the client

 

In the SQL*Net workaround, we only make changes to the SQL*Net connections, so connections to the Oracle database use this method. In the MTU workaround, we change all network traffic hops on the specific routes, so all connections work, including SSH.

For either workaround, we can either make a central change on the database server, which covers all connections, or make individual changes on the clients from which the connections are made. We recommend the following combination of workarounds:

  • Make SQL*Net changes centrally on the database. Any future connections to the database also benefit without having to make any extra changes.
  • Implement MTU changes on specific clients for administrators who require direct SSH connections to the database host.

For other scenarios, you can choose differently, so let’s review all four options in more detail.

 

Set limited SDU size for SQL*Net

You can set a specific packet size for SQL*Net connections using the SDU parameter. If you set it low enough, you don’t need to negotiate the maximum segment size (MSS) of the connection for SQL*Net traffic.

 

Set limited SDU size for SQL*Net connect strings on clients

When a SQL*Net connection is established from a client, you can set the SDU parameter as part of the connect string. For example, when using EZConnect, you can set the parameter sdu=, as shown in Figure 4:

A screenshot of an EZConnect string with a modified SDU parameter.
Figure 4: Example using EZConnect string with modified SDU parameter.

When using a full tnsnames.ora entry, you can set the SDU parameter, according to the SDU documentation.

 

Set limited SDU size for SQL*Net in database network configuration

You can centrally set the default SDU size in the database network configuration using the DEFAULT_SDU_SIZE parameter set in sqlnet.ora, as shown in Figure 5. Consider the following important details:

  • In Exadata Database service, the databases and grid Infrastructure use different sqlnet.ora files. Ensure that the parameter is set in the sqlnet.ora used by the database. This consideration doesn’t apply to the Base Database service, which has only 1 sqlnet.ora file.
  • If clients connecting from other environments, such as non-Azure, also use this database, these settings also affect those connections. So, this setup might not be optimal if you want a larger SDU size for those connections.
A screenshot of the added parameter for default SDU size in sqlnet.ora.
Figure 5: Adding DEFAULT_SDU_SIZE parameter in sqlnet.ora

Set limited MTU size for network routes

We can ensure that MSS never needs to be negotiated by setting a limited MTU size for a specific network route between the application host and the database host. Then, the packet sizes are always small enough to come through Azure, OCI, and the tunnel between them.

This workaround uses the ip route command to set the MTU size for a specific network route. The OCI documentation Avoiding the Need for PMTUD describes this process.

The ip command doesn’t persist through reboots. To persist route and MTU configurations, configure the appropriate ifconfig interface (ifcfg-$iface) files and restart the network interface. If a network restart isn’t possible at the time, you can modify the MTU and route entries at run time with the ip command.

 

Set limited MTU size on the client

On Linux clients, we can use the ip route command to set a specific MTU size for the network route to the database server. Figure 6 shows the command used on a virtual machine (VM) in a spoke VNet, using the complete database subnet (192.168.2.0/24) as the target. After setting the target, we can SSH to the database server in that subnet.

A screenshot of the ip route command setting the MTU size on a database.
Figure 6: Example of ip route command to set MTU size on a client

Set limited MTU size on the database server

For Oracle Base Database, you can use the ip route command directly on the database host. Figure 7 shows setting the route to the spoke1 VNet (10.40.1.0/24) to use a smaller MTU size. Then, connections between these machines are possible without any further settings.

A screenshot of the ip route command setting the MTU size on a database.
Figure 7: Example of ip route command to set MTU size on a database server

For Exadata Database, this workaround is valid when making this change at the DomU level. However, it isn’t certified and therefore not recommended for Exadata Database.

 

Conclusion

Many enterprise customers seek multicloud split stack solutions, which Oracle Database Service for Azure can deliver. These enterprises often use a hub-and-spoke network topology in their Azure environment. Because of the private tunnel created by OracleDB for Azure and the default blocking of any ICMP traffic in Azure VNets, which aren’t directly peered to each other, this setup can result in hanging connections.

In this situation, the solution is to allow ICMP traffic to flow freely between all resources and network components, as described in our blog Implementing OracleDB for Azure with a hub-and-spoke network. If that solution isn’t feasible, consider manually lowering SDU or MTU settings, as described in this blog, as a workaround because the PMTUD isn’t required.

For more information, see the following resources: