Saturday Dec 19, 2009

Quorum server patches and Sun Cluster 3.2

This is an early notify because there are some troubles around with the following Sun Cluster 3.2 quorum server patches:
127404-03 Sun Cluster 3.2: Quorum Server Patch for Solaris 9
127405-04 Sun Cluster 3.2: Quorum Server patch for Solaris 10
127406-04 Sun Cluster 3.2: Quorum Server patch for Solaris 10_x86
All these patches are part of Sun Cluster 3.2 11/09 Update3 release, but also available on My Oracle Support.

These patches delivers new features which requires attention in case of upgrade or patching. The installation of the mentioned patches on a Sun Cluster 3.2 quorum server can lead to a panic of all Sun Cluster 3.2 nodes which use this quorum server. The panic of Sun Cluster 3.2 nodes are as follows:
...
Dec 4 16:43:57 node1 \^Mpanic[cpu18]/thread=300041f0700:
Dec 4 16:43:57 node1 unix: [ID 265925 kern.notice] CMM: Cluster lost operational quorum; aborting.
...

Update 11.Jan.2010:
More details available Alert 1021769.1: Sun Cluster 3.2 Quorum Server Patches Cause All Cluster Nodes to Panic with "Cluster lost operational quorum"

General workaround before patching:
If the Sun Cluster only use the quorum server as a quorum device, temporarily add a second quorum device.
For example:
1) Configure a second temporary quorum device on each cluster node which uses the quorum server (use a disk if available).
# clq add d13 (or use clsetup)
2) Un-configure the quorum server from each cluster node that uses the quorum server.
# clq remove QuorumServer1 (or use clsetup)
3) Verify that the quorum server no longer serves any cluster.
on the Sun Cluster nodes # clq status
on the Sun Cluster quorum server # clqs show +
4) Install the Sun Cluster 3.2 Quorum Server patch
# patchadd 12740x-0x
5) Reboot and start the quorum server if not already started
# init 6
6) From a cluster node, configure the patched quorum server again as a quorum device.
# clq add -t quorum_server -p qshost=129.152.200.5 -p port=9000 QuorumServer1
7) Un-configure the temporary quorum device
# clq remove d13


Workaround if quorum server is already patched:
If patch is already installed (and rebooted) on the Sun Cluster 3.2 quorum server but quorum server is not working correctly then the following messages on the Sun Cluster nodes are visible:
...
Dec 3 17:24:24 node1 cl_runtime: [ID 868277 kern.warning] WARNING: CMM: Erstwhile online quorum device QuorumServer1 (qid 1) is inaccessible now.
Dec 3 17:29:20 node1 cl_runtime: [ID 237999 kern.notice] NOTICE: CMM: Erstwhile inaccessible quorum device QuorumServer1 (qid 1) is online now.
Dec 3 17:29:24 node1 cl_runtime: [ID 868277 kern.warning] WARNING: CMM: Erstwhile online quorum device QuorumServer1 (qid 1) is inaccessible now.
Dec 3 17:32:58 node1 cl_runtime: [ID 237999 kern.notice] NOTICE: CMM: Erstwhile inaccessible quorum device QuorumServer1 (qid 1) is online now.
...
DO NOT TRY to remove the quorum server on the Sun Cluster 3.2 nodes because this can end up in a panic loop.

Do:
1) Clear the configuration on the quorum server
# clqs clear <clustername> <quorumname>
or # clqs clear +

2.) Re-add the quorum server to the Sun Cluster nodes
# cluster set -p installmode=enabled
# clq remove QuorumServer1
# clq add -t quorum_server -p qshost=129.152.200.5 -p port=9000 QuorumServer1
# cluster set -p installmode=disabled


This is reported in Bug 6907934

As stated in my last blog the following note of the Special Install Instructions of Sun Cluster 3.2 core patch -38 and higher is very important.
NOTE 17: Quorum server patch 127406-04 (or greater) needs to be installed on quorum server host first, before installing 126107-37 (or greater) Core Patch on cluster nodes.
This means if using a Sun Cluster 3.2 quorum server then it's necessary to upgrade the quorum server before upgrade the Sun Cluster 3.2 nodes to Sun Cluster 3.2 11/09 update3 which use the quorum server. AND furthermore the same apply in case of patching. If installing the Sun Cluster core patch -38 or higher (-38 is part of Sun Cluster 3.2 11/09 update3)
126106-38 Sun Cluster 3.2: CORE patch for Solaris 10
126107-38 Sun Cluster 3.2: CORE patch for Solaris 10_x86
126105-38 Sun Cluster 3.2: CORE patch for Solaris 9
then the same rule apply. First update the quorum server and then the Sun Cluster nodes. Please refer to details above how to do it...
For upgrade also refer to the document: How to upgrade quorum server software

Keep in mind: Fresh installations with Sun Cluster 3.2 11/09 update3 on the Sun Cluster nodes and on the quorum server are NOT affected!

Monday Mar 09, 2009

memory leaks in "rgmd -z global" process

A memory leak occurs in the "rgmd -z global" process on Sun Cluster 3.2 1/09 Update2. The global zone instance of the rgmd process leaks memory in most situations such as "scstat" or "cluster show" and other basic commands. The problem is severe and the rgmd heap grows to a large size and crashes the Sun Cluster node.

The issue only happen if one of the following Sun Cluster core patches are active.
126106-27 or -29 or -30 Sun Cluster 3.2: CORE patch for Solaris 10
126107-28 or -30 or -31 Sun Cluster 3.2: CORE patch for Solaris 10_x86
Due to the fact that this patches are also part of the Sun Cluster 3.2 1/09 Update2 release the issue occur also on fresh installed Sun Cluster 3.2 1/09 Update2 systems.

The error can look as follows:
Analyze the grow of memory allocation with (or similar tools)
# prstat
3942 root 61M 11M sleep 101 - 0:00:02 0.7% rgmd/41
sometime later the increase of the memory allocation is visible.
3942 root 61M 20M sleep 101 - 0:01:15 0.7% rgmd/41
or
# pmap -x <pid_of_rgmd-z_global> | grep heap
00022000 47648 6992 6984 - rwx-- [ heap ]
sometime later the increase of the memory allocation is visible.
00022000 47648 15360 15352 - rwx-- [ heap ]

When the memory is full the Sun Cluster node panics with the following message:
Feb 25 07:59:23 node1 RGMD[1843]: [ID 381173 daemon.error] RGM: Could not allocate 1024 bytes; node is out of swap space; aborting node.
...
Feb 25 08:10:05 node1 cl_dlpitrans: [ID 624622 kern.notice] Notifying cluster that this node is panicking
Feb 25 08:10:05 node1 unix: [ID 836849 kern.notice]
Feb 25 08:10:05 node1 \^Mpanic[cpu0]/thread=2a100047ca0:
Feb 25 08:10:05 node1 unix: [ID 562397 kern.notice] Failfast: Aborting zone "global" (zone ID 0) because "globalrgmd" died 30 seconds ago.
Feb 25 08:10:06 node1 unix: [ID 100000 kern.notice]
...

Update 20.Mar.2009:
Available now:
Alert 1020253.1 Memory Leak in the "rgmd" Process of Solaris Cluster 3.2 may Cause a failfast Panic

Update 17.Jun.2009:
The -33 revision of the Sun Cluster core patch is the first released version which fix this issue.
126106-33 Sun Cluster 3.2: CORE patch for Solaris 10
126107-33 Sun Cluster 3.2: CORE patch for Solaris 10_x86


Workaround: Use previous version -19 to prevent issue.
126106-19 Sun Cluster 3.2: CORE patch for Solaris 10
126107-19 Sun Cluster 3.2: CORE patch for Solaris 10_x86

The issue is reported in bug 6808508 (description: scalable services coredump during the failover due to network failure). A fix is in progress. This blog will be updated when the fix is available.

Monday Jun 30, 2008

prevent reservation conflict panic if using active/passive storage controller

Reservation conflicts can happen in a Sun Cluster environment if using active/passive storage controllers e.g. SE6540, SE6140, FLX380.

First of all you should always consider to disable auto-failback flag if using MPxIO on shared devices. This can also prevent reservation conflict panics.

Change the auto-failback value in /kernel/drv/scsi_vhci.conf to disable.
e.g of kernel/drv/scsi_vhci.conf
...
# Automatic failback configuration
# possible values are auto-failback="enable" or auto-failback="disable"
auto-failback="disable";
...


Furthermore the reservation conflict panic was seen when one cluster node is down and the shared storage array made some (at least 2 or 3) failovers between the active/passive controllers. The behavior always depends on the design of the storage array controller.

Two workarounds are available at the moment:

1.) In case of Sun Cluster 3.2 force the cluster to do scsi3 reservations even in 2 node cluster configurations. If you have a 3 node (or more nodes), the cluster should do scsi3 reservations anyway.

Be aware of Alert 1019005.1. In case of SE6540/SE6140/FLX380 use firmware 6.60.11.xx (which is part of CAM 6.1) or higher. To avoid trouble update this code before enabling SCSI3 reservations.

To force the Sun Cluster 3.2 to do scsi3 reservations run the command:
# cluster set -p global_fencing=prefer3

Verify the setting using :
# cluster show | grep -i scsi
   Type:                       scsi
   Access Mode:        scsi3


2.) Allow Reservation on Unowned LUNs in SE6540/SE6140. You should prefer the workaround #1 but in case of Sun Cluster 3.1 you can not force scsi3 reservation mechanism for 2 node clusters. So, there is a need to use scsi2 reservations.

The bit "Allow Reservation on Unowned LUNs" determines the controller response to Reservation/Release commands that are received for LUNs that are not owned by the controller. The value needs to be changed from 0x01 to 0x00. Beware this setting will be lost after a NVSRAM update!

Using CAM management software do the following:
# cd /opt/SUNWsefms/bin/

For 6540/FLX380/FLX240/FLX280 run:
# ./service -d -c set -q nvsram region=0xf2 offset=0x19 value=0x00 host=0x02

For 6140 and 6130 run:
# ./service -d -c set -q nvsram region=0xf2 offset=0x19 value=0x00 host=0x00

Reboot both controllers in order to make the change active :
# ./service -d -c reset -t a

Wait at least 5 minutes until the A controller is up again.
# ./service -d -c reset -t b


Why this not happing before? With the changes of patch 125081-14 (sparc) or 125082-14 (x86) Sun deliver new driver for MPxIO. Due to this changes the problem can be triggered.

About

I'm still mostly blogging around Solaris Cluster and support. Independently if for Sun Microsystems or Oracle. :-)

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today