X

News, tips, partners, and perspectives for the Oracle Solaris operating system

Why a logical IP is marked as DEPRECATED?

Guest Author

Reported Firewall Problems

We've had many questions, comments and complaints about IP address "problems" when using highly available services
in a Sun Cluster environment. We found out that most, if not all of these were related to configurations where
firewalls were configured between the service running on the cluster, and the clients connecting to the cluster.

So, what is the problem? The firewall administrators often make the assumption that a packet sent from a client to
the logical IP address of an HA service will generate a response IP packet with
exactly the same logical IP address
as the source address. So, they configure an appropriate firewall rule and wonder why this rule does not work, i.e.,
instead there were IP packets coming back from an HA service that did not match
this rule.

Then they start researching
the network configuration on the cluster node that hosts the HA service and find
out that the logical IP
address used by that service was set to a state called "DEPRECATED". And they think this is the root cause
of their problem - which (we think) is not the case.

How does Address Selection really work?

As address selection can become very complicated in complex network setups, the
following will be true for the typical
simple network setup found at most installations.

Let's look at the address selection for an outgoing packet a bit more closely.
First we must make a distinction between TCP
(RFC 793) and UDP (RFC 768).
TCP is a connection-oriented protocol, i.e. a connection is established between
a client and a service. Using this
connection, source and target addresses are always used appropriately;
in a Sun Cluster environment the source address of a packet sent by the service
to a client will usually be the logical
IP address of that HA service - but only if the client used the logical service
address to send its request to the service.

So, this will not cause any problems with firewalls, because you know exactly which IP addresses will be used as
source addresses for outgoing IP addresses.

Let's look into UDP now. UDP is a connectionless protocol, i.e., there is no established connection between a client
and a server (service). A UDP-based service can choose its source address for outgoing packets by binding itself to
a fixed address, but most services don't do this. Instead, they accept incoming
packets from all
network addresses configured. For those readers who are familiar with network programming, the typical code
segment has the following lines in it:


struct sockaddr_in address;
...
address.sin_addr.s_addr = INADDR_ANY;
...
if (bind (..., (struct sockaddr \*) &address, ...) == 0)

Using this typical piece of code, the UDP service listens on all configured IP addresses, and the outbound source address is
set by the IP layer and the choosing algorithm is complex and cannot be influenced. Details can be read in Infodoc 204569
(access on SunSolve for SPECTRUM contract holders only); but we think
they are not that relevant here, except for this quote:
"IP addresses associated with interfaces marked as DEPRECATED will
not normally be used as source addresses by IP
unless deprecated interfaces are all that is available, in which case they will
be used."

The DEPRECATED flag

So, now DEPRECATED comes into play. A DEPRECATED address will - normally - not be used as a source address!
First, why does Sun Cluster set HA IP addresses, i.e. logical or floating addresses into state DEPRECATED? Because
they are floating addresses - there is no guarantee that they will stay on one node. In failure situations an HA IP address will
float to another node together with its service. Or if the administrator decides
to migrate a service; or when the service is
stopped, the logical IP address might disappear on one node.

Let's have a look at services where IP communication is initiated from a cluster
node. E.g. a cluster node might try
to mount an external NFS share on this node temporarily. Whether this is UDP or
TCP based NFS would not matter in this case!
The IP layer would choose a source address; it could
be the logical IP address of an HA service that happens to run on the same system - if it were not DEPRECATED. Now, imagine
the NFS mount is successful, is using the logical IP address and NFS transfers work fine.
Now, the HA service that owns the HA IP address is switched to another
node in the cluster; its IP address would also switch. What would happen to the
NFS traffic between this node and the external
NFS server? It would fail. Packets coming from the NFS server would reach a different node now; namely that of the HA service
that switched, taking its IP address with it. (And the NFS client on the cluster
node would fail as well.....)

So, that is the reason for setting the DEPRECATED flag on HA IP addresses; remember the quote above: "...marked as DEPRECATED will not normally be used...".
Although not setting the DEPRECATED flag would improve the probability that the address potentially be used by the IP layer
as a source address, there is no guarantee and at the end, this would not help.
But the DEPRECATED flag helps to prevent
major problems on cluster nodes.

The Solution

Back to the original question: how can I make my firewall rules work? There are 4 possibilities - in prioritized order, best
practice first:

  1. change your firewall rules to accept all possible addresses from the nodes where packets could be originating from;
  2. change your service, by binding only to the HA service IP address - which is
    only possible if its configuration lets you do this or if you have access to the source code;
  3. move your HA service into a Solaris 10 container, that only uses the logical IP address; in this configuration the logical IP address will always be used as source address, even though it is in state DEPRECATED;
  4. try to manipulate the decision process of the IP layer - which is a very bad
    idea.

To summarize

Sun Cluster sets the DEPRECATED flag on HA service IP addresses by
design and it is a good thing, as it
prevents strange problems with IP based clients on cluster nodes to happen. Not
setting it, would not solve the problems
reported.


Hartmut Streppel
Principal Field Technologist
Systems Practice

Join the discussion

Comments ( 5 )
  • Mohammed Tanvir Alam Sunday, March 29, 2009

    Can it possible that we can prevent this issue without configuring the SO based IPMP ?

    Any Suggestion ?

    Thanks

    M|Tanvir


  • Hartmut Streppel Monday, March 30, 2009

    I am not sure I understand your question, especially "SO based IPMP". It is my understanding that only points a,b and c in the "Solution" paragraph can solve this issue.

    Hartmut


  • Bernd Eckenfels Tuesday, May 5, 2009

    an libc based wrapper or kernel net namespaces (not sure if this linux feature exists on Solaris outside of zones) could make the default ip selection configurable for a process.

    And BTW: outgoing TCP connections also have this problem. Especially nasty in RMI scenarios.


  • guest Sunday, July 26, 2009

    There is fourth way to force service to choose logical IP address. It is enough to close service in zone, and configure zones as cluster nodes. Do not assign any IP address to the zone. So zone will get one address (logical one) after switching our group to it.

    Disadvantage is that when zone does not have any running group we can only access it from zlogin on physical host.


  • Hartmut Streppel Wednesday, July 29, 2009

    Putting the service into a container is a very good idea. If that container only has the logical IP address visible, this will be taken as source address; even if it is marked DEPRECATED. I'll add that to my article.


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.