By Stefan Hinker-Oracle on Apr 04, 2016
If you run multiple grid infrastructures (aka RAC Clusters) on SuperCluster, which share the same set of Exadata Storage Servers (aka cells), adding ASM Scoped Security to the setup is a good idea. Even if there are no security reasons like multi-tenancy, just simply preventing accidental use of one cluster's diskgroups by another cluster should be reason enough to implement this simple precaution.
Of course, there is good documentation on this feature, available here. However, as often the case, the devil is in the details, so here's a comprehensive example of how to do this:
- Shutdown the cluster you want to modify. Use "crsctl stop crs" on all cluster nodes.
- Create a key for this cluster
On any storage cell, use "cellcli -e create key". It will give you an ASCII string to use as a key. Copy that string to a temporary place. In this example, I'll use the key '9e9a606a461a1abc6af43626e85af3b7'
- Invent a unique name to use for this cluster. In this example, I'll use "marsc1" to denote the first cluster running on mars.
- Create a name/key pair on all cells using this unique name and the key from above. On all cells, execute this cellcli command:
assign key for 'marsc1'='9e9a606a461a1abc6af43626e85af3b7'
- Here's the most difficult part. We'll need to assign all griddisks that are used by our cluster to this unique name. Cellcli's filters and wildcards don't help much here. Here's how I did it:
- On all cells, create a list of all disks belonging to marsc1. In cellcli, do:
spool /tmp/disks list griddisk where asmdiskgroupname='DATAC1' attributes name list griddisk where asmdiskgroupname='RECOC1' attributes name
- In /tmp/disks on each cell, there will now be a number of lines similar to this:
DATAC1_CD_00_marsceladm04 DATAC1_CD_01_marsceladm04 DATAC1_CD_02_marsceladm04 DATAC1_CD_03_marsceladm04
- Using your favorite file manipulation tools (I used awk and vi), use this file to create a command file that contains one "alter griddisk" command for each griddisk. Mine looked like this afterwards:
alter griddisk DATAC1_CD_00_marsceladm04 availableTo='marsc1' alter griddisk DATAC1_CD_01_marsceladm04 availableTo='marsc1' alter griddisk DATAC1_CD_02_marsceladm04 availableTo='marsc1' alter griddisk DATAC1_CD_03_marsceladm04 availableTo='marsc1'
- Run this command script on each cell. Of course, each cell will have its own script.
# cellcli < script
- Check that it worked using cellcli:
list griddisk attributes name,availableTo
- Finally, enter the unique name and the key in a file called "cellkey.ora". On Solaris, this file is located in /etc/oracle/cell/network-config
My file looks like this:
- Restart crs on all nodes:
crsctl start crs
That should be all. You can easily verify that your other clusters can no longer see these diskgroups or disks from another cluster's asm: asmcmd lsdg --discovery
Now, repeat this for all of your clusters. The end result will be exclusive access to each cluster's disks, with no danger of intentional snooping or unintentional use.
One tool that comes in very handy for doing stuff on all cells at the same time is "cssh" - a one to many commandline included in recent versions of Solaris.