X

News, tips, partners, and perspectives for the Oracle Solaris operating system

Configuring IP Filter Support for Failover Services with Solaris Cluster 3.2

Guest Author

For information on the IP Filter feature, I suggest you look at the Solaris 10 System Administration Guide
This write-up is not in current Solaris Cluster 3.2 current documentation, but we will get documented with our next patch.

IP filter in Solaris 10 (FCS to up to Solaris 10 11/06 (update 3)) exists as a STREAMS module that must reside right below IP in any stream. The network/pfil and network/ipfilter SMF services autopush pfil module (based on /etc/ipf/pfil.ap) and loads in filtering rules (from /etc/ipf/ipf.conf) at boot time. Filtering rules can be based on interface, IP addresses/subnets, protocols, and ports.

NOTE : Scalable services are not supported with ipfilter at this time. These steps are only to configure ipfilter for failover services. For enabling cluster wide filters ensure the configuration procedure is replicated on all nodes

Step 1

Edit /etc/iu.ap so that lines corresponding to public NIC devices have "clhbsndr pfil" as the module list. The nodes should be rebooted for
the change to take effect. Nodes can be rebooted in a rolling fashion.

Note that "pfil" must be the last module in the list.

ex : # scstat -i -h `uname -n`

-- IPMP Groups --

 Node Name Group Status Adapter Status
 IPMP Group: node1 sc_ipmp1 Online ce2 Online
 IPMP Group: node1 sc_ipmp1 Online bge2 Online
 IPMP Group: node1 sc_ipmp0 Onlinece0 Online
 IPMP Group: node1sc_ipmp0 Onlinebge0 Online

    
       
       
           
/etc/iu.ap will have these lines modified as below ( for ce/bge )
ce    -1    0    clhbsndr pfil
bge  -1    0    clhbsndr pfil

Step 2

Add filter rules to /etc/ipf/ipf.conf on all nodes as needed. See ipf(4) manpage for more information on IP Filter rules syntax.

ex:    block in quick on bge0 from 129.146.106.0/23 to any
         block in quick on ce0 from 129.146.106.0/23 to any

Step 3

Enable the ipfilter SMF service.

svcadm enable /network/ipfilter:default

Filtering rules

Cluster fails over network addresses from node to node. No special procedure or code is needed at the time of failover.

Please make sure filtering rules that reference IP addresses of LogicalHostname and SharedAddress resources are identical on all cluster nodes.

Rules on a standby node will reference a non-existent IP address. But this rule is still part of IP filter's active rule set and will become effective once the node receives the address after a failover.

In addition, rules must be set up that apply uniformly to all NICs in the same IPMP group. That is, if a rule is interface-specific, the same rule must also exist for other interfaces in the same IPMP group.

Non-Cluster mode

In non-cluster mode, the line in /etc/iu.ap containing clhbsndr would not take effect because clhbsndr is not registered. But subsequent autopush setup by SMF service network/pfil (which fails in \*cluster\* mode) would succeed. This ensures that pfil is also pushed in non-cluster mode.

The user must also update /etc/ipf/pfil.ap in addition to /etc/iu.ap. Updates to pfil.ap is slightly different. Refer to IP Filter documentation for more details.

Stateful filtering

IP filter does not support stateful filtering with IPMP since outgoing packets for the same session can go through multiple interfaces. Sun Cluster requires the use of IPMP groups and hence inherits the same limitation.

Thus, only stateless filtering is supported with clustering.

NAT mode

NAT in routing mode is not supported. That is, a cluster node must not be set up as a router/gateway to forward packets to and from another node. This is because such setup easily creates a single point of failure for the whole cluster without first making the routing mechanism HA.

Use of NAT for translation of local addresses is supported. NAT translation rewrites packets on-the-wire and hence is transparent to cluster software.

Note: NAT rules containing IP addresses that are managed by clustering (e.g. LogicalHostname resources) must be replicated on all cluster nodes.

IPFILTER on Cluster transport

If you have the same type of adapter for private and public network, the edits made to /etc/iu.ap can result in pfil getting pushed on private network streams. But the cluster transport module would remove all unwanted modules at stream creation. Hence pfil would be removed. No special user procedure is required.

John Blair
QA Manager
Solaris Cluster

Join the discussion

Comments ( 14 )
  • Vincent Fox Thursday, October 25, 2007

    I have seen this working fine on Solaris 10u3 systems. But my attempt today to make it work with 10u4 was unsuccessful. I suspect that 10u4 changed IP Filter behaviour enough to break it.


  • Honsing Cheng Thursday, October 25, 2007

    S10U4 has moved the ipfilter functionality into IP, so there is no more need to setup anything related to the pfil module (including changes to /etc/iu.ap.) Could you give more specifics on why this does not work for you anymore? Any error messages you can provide would help.


  • Vincent Fox Thursday, October 25, 2007

    I filed a Sun case on this but it's pretty simple. Same ruleset and hardware I was using under u3. If I have ipfilter turned off then u4 works fine. If I turn it on and boot the 2nd node it hangs with errors like so:

    WARNING: Path mta16:qfe0 - capri:qfe0 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path.

    WARNING: Path mta16:qfe1 - capri:qfe1 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path.


  • Honsing Cheng Friday, October 26, 2007

    Before S10u4, SC automatically removes the pfil module from the interconnect so that filtering does not interfere with cluster handshakes and infrastructure operations.

    With S10u4, filtering can interfere with cluster formation and operation if a rule exists that blocks interconnect traffic. Bear in mind a rule without an interface tag applies to all interfaces.

    So for example, a rule that blocks everything except HTTP packets would interfere with cluster formation in S10u4, but not before.

    We will publish guidelines on how to filter on the interconnect soon -- basically a list of ports for internal cluster use. Filtering must allow these to "pass".

    But for now, you might consider letting all interconnect traffic pass if the cluster interconnect is on private links and filtering is a concern on the public network only.

    Try prepending your rules with this:

    (modify interconnect names first)

    # pass all traffic on interconnect

    pass in quick on e1000g1 all

    pass out quick on e1000g1 all

    pass in quick on e1000g5 all

    pass out quick on e1000g5 all

    pass in quick on clprivnet0 all

    pass out quick on clprivnet0 all


  • Vincent Fox Friday, October 26, 2007

    Had already tried the cluster transports by name, had not thought of the clprivnet0 interface. Adding that in the top of ipf.conf didn't seem to help. Good idea though.


  • Honsing Cheng Tuesday, October 30, 2007

    It would be helpful to see exactly the rule set you use -- with all sensitive info removed of course. Or you could email that to me if you like. I'm sure the Sun Case folks would also get back to you soon.


  • Honsing Cheng Friday, November 2, 2007

    I got your rule set and was able to replicate the problem on my s10u4 cluster. I can work around it by not using the interface names, but by using the subnets of the private NICs and clprivnet0 in the rule set.

    Using subnets also has the benefit of not needing to update the rules when a different private NIC is added later.

    1) Figure out subnets with ifconfig. For example:

    # ifconfig e1000g1

    e1000g1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 7

    inet 172.16.0.129 netmask ffffff80 broadcast 172.16.0.255

    ether 0:14:4f:f:41:8f

    # ifconfig clprivnet0

    clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500

    index 8

    inet 172.16.4.1 netmask fffffe00 broadcast 172.16.5.255

    ether 0:0:0:0:0:1

    (Note the different netmask length on clprivnet0.)

    Taking the "broadcast" value and AND it with the "netmask" value, we have subnet for e1000g1 is 172.16.0.128/25 and for clprivnet0 is 172.16.4.0/23.

    2) The rules to pass traffic on the interconnect are therefore:

    # Rules to pass traffic on cluster interconnect

    pass in quick proto tcp/udp from 172.16.0.128/25 to any

    pass out quick proto tcp/udp from 172.16.0.128/25 to any

    # Rules to pass traffic on cluster logical subnet

    in quick proto tcp/udp from 172.16.4.0/23 to any

    pass out quick proto tcp/udp from 172.16.4.0/23 to any

    Hopefully this also works for you. We will pursue the interface rules separately and find out why that didn't work.


  • Vincent Fox Monday, November 5, 2007

    That seems to work. However, I see this error on console of 2nd node during boot:

    ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast

    Also, not crazy about allowing 172.x addresses. Security-wise I wanted to limit to specific named interfaces since this systems is on a public network.


  • sc blog administrator Tuesday, January 8, 2008

    FYI that the particular issue Frank Fegert observed is being addressed via the Sun Case mechanism.


  • tomasz Tuesday, January 29, 2008

    Anybody found any sensible warkarounds maybe?

    I am facing the same problem here with Sol10_update4 and SunCluster 3.2.

    I think I am going to change the way services get started and move the ipfilter after bootcluster.


  • Tomasz Thursday, January 31, 2008

    With the below commands the ipfilter starts after the SunCluster (bootcluster service). The ipfilter then recognizes all the plumbed interfaces correctly and applies appropriate rules.

    svcs -l svc:/milestone/network

    svccfg -s svc:/milestone/network delpg network

    svcadm refresh svc:/milestone/network

    svccfg -s ipfilter listpg

    svccfg -s ipfilter addpg bootcluster dependency

    svccfg -s ipfilter setprop bootcluster/entities = fmri: 'svc:/system/cluster/bootcluster'

    svccfg -s ipfilter setprop bootcluster/grouping = astring: 'require_all'

    svccfg -s ipfilter setprop bootcluster/restart_on = astring: 'restart'

    svccfg -s ipfilter setprop bootcluster/type = astring: 'service'

    svccfg -s ipfilter listprop bootcluster\*

    svcadm refresh ipfilter

    svcadm clear ipfilter

    to remove the newly applied service dependency:

    svccfg -s ipfilter delpg bootcluster


  • Tanvir Saturday, April 11, 2009

    Hi,

    How to know the details about "S10U4" and what is meaning of it?

    Thanks

    MTanvir


  • Burt Clouse Thursday, April 15, 2010

    S10U4 is shorthand for Solaris 10 update 4 which is the "unofficial" name of Solaris 10 08/07.

    If you try a websearch on "Solaris 10 update 4" or "Solaris 10u4" you'll find plenty of info.


  • Tanvir Friday, April 16, 2010

    hi Burt,

    Thanks for your update, I have one question..i want to shutdown 2 node cluster and I am running "scshutdown" command...is it necessary to bring offline all resource group before shutting down the node?

    Thanks

    MTanvir


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.