Configuring IP Filter Support for Failover Services with Solaris Cluster 3.2


For information on the IP Filter feature, I suggest you look at the Solaris 10 System Administration Guide
This write-up is not in current Solaris Cluster 3.2 current documentation, but we will get documented with our next patch.

IP filter in Solaris 10 (FCS to up to Solaris 10 11/06 (update 3)) exists as a STREAMS module that must reside right below IP in any stream. The network/pfil and network/ipfilter SMF services autopush pfil module (based on /etc/ipf/pfil.ap) and loads in filtering rules (from /etc/ipf/ipf.conf) at boot time. Filtering rules can be based on interface, IP addresses/subnets, protocols, and ports.


NOTE : Scalable services are not supported with ipfilter at this time. These steps are only to configure ipfilter for failover services. For enabling cluster wide filters ensure the configuration procedure is replicated on all nodes

Step 1


Edit /etc/iu.ap so that lines corresponding to public NIC devices have "clhbsndr pfil" as the module list. The nodes should be rebooted for
the change to take effect. Nodes can be rebooted in a rolling fashion.

Note that "pfil" must be the last module in the list.

ex : # scstat -i -h `uname -n`

-- IPMP Groups --



 Node Name Group Status Adapter Status
 IPMP Group: node1 sc_ipmp1 Online ce2 Online
 IPMP Group: node1 sc_ipmp1 Online bge2 Online
 IPMP Group: node1 sc_ipmp0 Onlinece0 Online
 IPMP Group: node1sc_ipmp0 Onlinebge0 Online

    
       
       
           
/etc/iu.ap will have these lines modified as below ( for ce/bge )
ce    -1    0    clhbsndr pfil
bge  -1    0    clhbsndr pfil

 

Step 2

Add filter rules to /etc/ipf/ipf.conf on all nodes as needed. See ipf(4) manpage for more information on IP Filter rules syntax.

ex:    block in quick on bge0 from 129.146.106.0/23 to any
         block in quick on ce0 from 129.146.106.0/23 to any

Step 3

Enable the ipfilter SMF service.

svcadm enable /network/ipfilter:default

Filtering rules

Cluster fails over network addresses from node to node. No special procedure or code is needed at the time of failover.

Please make sure filtering rules that reference IP addresses of LogicalHostname and SharedAddress resources are identical on all cluster nodes.

Rules on a standby node will reference a non-existent IP address. But this rule is still part of IP filter's active rule set and will become effective once the node receives the address after a failover.


In addition, rules must be set up that apply uniformly to all NICs in the same IPMP group. That is, if a rule is interface-specific, the same rule must also exist for other interfaces in the same IPMP group.

Non-Cluster mode

In non-cluster mode, the line in /etc/iu.ap containing clhbsndr would not take effect because clhbsndr is not registered. But subsequent autopush setup by SMF service network/pfil (which fails in \*cluster\* mode) would succeed. This ensures that pfil is also pushed in non-cluster mode.

The user must also update /etc/ipf/pfil.ap in addition to /etc/iu.ap. Updates to pfil.ap is slightly different. Refer to IP Filter documentation for more details.

Stateful filtering

IP filter does not support stateful filtering with IPMP since outgoing packets for the same session can go through multiple interfaces. Sun Cluster requires the use of IPMP groups and hence inherits the same limitation.

Thus, only stateless filtering is supported with clustering.

NAT mode

NAT in routing mode is not supported. That is, a cluster node must not be set up as a router/gateway to forward packets to and from another node. This is because such setup easily creates a single point of failure for the whole cluster without first making the routing mechanism HA.


Use of NAT for translation of local addresses is supported. NAT translation rewrites packets on-the-wire and hence is transparent to cluster software.

Note: NAT rules containing IP addresses that are managed by clustering (e.g. LogicalHostname resources) must be replicated on all cluster nodes.

IPFILTER on Cluster transport

If you have the same type of adapter for private and public network, the edits made to /etc/iu.ap can result in pfil getting pushed on private network streams. But the cluster transport module would remove all unwanted modules at stream creation. Hence pfil would be removed. No special user procedure is required.

John Blair
QA Manager
Solaris Cluster

Comments:

Why aren't you using the new "cluster status" command to get the information instead of "scstat"?

Posted by guest on March 07, 2007 at 03:51 AM PST #

Yes, you could use "cluster status -v" to get the similar information. In SC3.2, the old CLI is still available so I guess you could use either one of them. Thanks, Prasanna Kunisetty Sun Cluster Engineering

Posted by Prasanna Kunisetty on March 12, 2007 at 09:26 AM PDT #

I have seen this working fine on Solaris 10u3 systems. But my attempt today to make it work with 10u4 was unsuccessful. I suspect that 10u4 changed IP Filter behaviour enough to break it.

Posted by Vincent Fox on October 25, 2007 at 08:31 AM PDT #

S10U4 has moved the ipfilter functionality into IP, so there is no more need to setup anything related to the pfil module (including changes to /etc/iu.ap.) Could you give more specifics on why this does not work for you anymore? Any error messages you can provide would help.

Posted by Honsing Cheng on October 25, 2007 at 10:51 AM PDT #

I filed a Sun case on this but it's pretty simple. Same ruleset and hardware I was using under u3. If I have ipfilter turned off then u4 works fine. If I turn it on and boot the 2nd node it hangs with errors like so:

WARNING: Path mta16:qfe0 - capri:qfe0 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path.
WARNING: Path mta16:qfe1 - capri:qfe1 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path.

Posted by Vincent Fox on October 25, 2007 at 11:51 AM PDT #

Before S10u4, SC automatically removes the pfil module from the interconnect so that filtering does not interfere with cluster handshakes and infrastructure operations.

With S10u4, filtering can interfere with cluster formation and operation if a rule exists that blocks interconnect traffic. Bear in mind a rule without an interface tag applies to all interfaces.

So for example, a rule that blocks everything except HTTP packets would interfere with cluster formation in S10u4, but not before.

We will publish guidelines on how to filter on the interconnect soon -- basically a list of ports for internal cluster use. Filtering must allow these to "pass".

But for now, you might consider letting all interconnect traffic pass if the cluster interconnect is on private links and filtering is a concern on the public network only.

Try prepending your rules with this:
(modify interconnect names first)

# pass all traffic on interconnect
pass in quick on e1000g1 all
pass out quick on e1000g1 all
pass in quick on e1000g5 all
pass out quick on e1000g5 all
pass in quick on clprivnet0 all
pass out quick on clprivnet0 all

Posted by Honsing Cheng on October 26, 2007 at 10:02 AM PDT #

Had already tried the cluster transports by name, had not thought of the clprivnet0 interface. Adding that in the top of ipf.conf didn't seem to help. Good idea though.

Posted by Vincent Fox on October 26, 2007 at 02:20 PM PDT #

It would be helpful to see exactly the rule set you use -- with all sensitive info removed of course. Or you could email that to me if you like. I'm sure the Sun Case folks would also get back to you soon.

Posted by Honsing Cheng on October 30, 2007 at 08:20 AM PDT #

I got your rule set and was able to replicate the problem on my s10u4 cluster. I can work around it by not using the interface names, but by using the subnets of the private NICs and clprivnet0 in the rule set.

Using subnets also has the benefit of not needing to update the rules when a different private NIC is added later.

1) Figure out subnets with ifconfig. For example:

# ifconfig e1000g1
e1000g1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 7
inet 172.16.0.129 netmask ffffff80 broadcast 172.16.0.255
ether 0:14:4f:f:41:8f
# ifconfig clprivnet0
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500
index 8
inet 172.16.4.1 netmask fffffe00 broadcast 172.16.5.255
ether 0:0:0:0:0:1

(Note the different netmask length on clprivnet0.)

Taking the "broadcast" value and AND it with the "netmask" value, we have subnet for e1000g1 is 172.16.0.128/25 and for clprivnet0 is 172.16.4.0/23.

2) The rules to pass traffic on the interconnect are therefore:

# Rules to pass traffic on cluster interconnect
pass in quick proto tcp/udp from 172.16.0.128/25 to any
pass out quick proto tcp/udp from 172.16.0.128/25 to any

# Rules to pass traffic on cluster logical subnet
in quick proto tcp/udp from 172.16.4.0/23 to any
pass out quick proto tcp/udp from 172.16.4.0/23 to any

Hopefully this also works for you. We will pursue the interface rules separately and find out why that didn't work.

Posted by Honsing Cheng on November 02, 2007 at 07:01 AM PDT #

That seems to work. However, I see this error on console of 2nd node during boot:

ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast

Also, not crazy about allowing 172.x addresses. Security-wise I wanted to limit to specific named interfaces since this systems is on a public network.

Posted by Vincent Fox on November 05, 2007 at 09:58 AM PST #

I was bitten by the same bug, but may have figured out where things break. Could someone please verify this?

It seems not only S10u4 or later are affected, but also systems with patch 120011-14 or later applied. The problem is that after moving ipfilter from STREAMS to IP, ipfilter has no clue about the interfaces SC 3.2 is configuring. The process is now roughly like this:
- Solaris-known interfaces are configured at system boot.
- ipfilter is activated and tables for all configured interfaces are created.
- SC 3.2 starts and svc:/system/cluster/bootcluster (line 670-682) plumbs and configures the cluster transport interfaces.
- ipfilter doesn't know about the those new interfaces and applies only gobal non-interface specific filter rules. See bug-id 6523130 (http://sunsolve.sun.com/search/document.do?assetkey=1-1-6523130-1)

Two possible solutions:
1.) "/usr/cluster/lib/sc/clconfig -c" (line 670 in svc:/system/cluster/bootcluster) should call "ipf -y" after plumbing and configuring additional interfaces, or
2.) ipfilter should revert back to the old behaviour and know about all interfaces, even the not yet configured ones.

I'm not sure which one is the way to go here, but i'll post this on the ipfilter-ML and open a case as well.

Posted by Frank Fegert on December 31, 2007 at 07:50 PM PST #

FYI that the particular issue Frank Fegert observed is being addressed via the Sun Case mechanism.

Posted by sc blog administrator on January 08, 2008 at 07:19 AM PST #

Anybody found any sensible warkarounds maybe?
I am facing the same problem here with Sol10_update4 and SunCluster 3.2.

I think I am going to change the way services get started and move the ipfilter after bootcluster.

Posted by tomasz on January 29, 2008 at 12:43 AM PST #

With the below commands the ipfilter starts after the SunCluster (bootcluster service). The ipfilter then recognizes all the plumbed interfaces correctly and applies appropriate rules.

svcs -l svc:/milestone/network
svccfg -s svc:/milestone/network delpg network
svcadm refresh svc:/milestone/network

svccfg -s ipfilter listpg
svccfg -s ipfilter addpg bootcluster dependency
svccfg -s ipfilter setprop bootcluster/entities = fmri: 'svc:/system/cluster/bootcluster'
svccfg -s ipfilter setprop bootcluster/grouping = astring: 'require_all'
svccfg -s ipfilter setprop bootcluster/restart_on = astring: 'restart'
svccfg -s ipfilter setprop bootcluster/type = astring: 'service'
svccfg -s ipfilter listprop bootcluster\*

svcadm refresh ipfilter
svcadm clear ipfilter

to remove the newly applied service dependency:
svccfg -s ipfilter delpg bootcluster

Posted by Tomasz on January 30, 2008 at 07:00 PM PST #

Hi,
How to know the details about "S10U4" and what is meaning of it?

Thanks
MTanvir

Posted by Tanvir on April 11, 2009 at 05:12 AM PDT #

S10U4 is shorthand for Solaris 10 update 4 which is the "unofficial" name of Solaris 10 08/07.

If you try a websearch on "Solaris 10 update 4" or "Solaris 10u4" you'll find plenty of info.

Posted by Burt Clouse on April 15, 2010 at 05:01 AM PDT #

hi Burt,
Thanks for your update, I have one question..i want to shutdown 2 node cluster and I am running "scshutdown" command...is it necessary to bring offline all resource group before shutting down the node?

Thanks
MTanvir

Posted by Tanvir on April 16, 2010 at 02:31 AM PDT #

We have recently conducted another test on IP filter on Solaris Cluster 3.2. It is now possible to use the private NIC names in the rule set instead of the private IP addresses, as many of you have concern about doing that before.

For the details, please refer to -
http://www.sun.com/bigadmin/features/ipfilter_config.jsp

Posted by sujata roy on April 16, 2010 at 07:13 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

mkb

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today