Deploying Oracle RAC DB 11gR2 on LDoms Virtualization

Introduction

This blog provides a "cookbook" procedure to configure and install Oracle Real Application Cluster (RAC) Database 11.2.0.2 on Solaris 10 and Logical Domains on Sun CMT servers (also using ZFS).  If you are not familiar with LDom, review this document called "Beginners Guide to Oracle VM Server for SPARC: Understanding and Deploying Logical Domains".

 Other references used are:

  1. Oracle Technology Network
  2. Oracle Database Online Documentation for 11gR2
  3. Supported Virtualization Technologies for Oracle RAC DB

Objective

In this procedure, I install and configure Oracle RAC DB 11gR2 (11.2.0.2) on Solaris 10 CMT Servers using the Logical Domains as the hosted virtualized environement.  I will set up a two node RAC DB server as follows:

  1. 2 T2000 Servers
  2. 1 SE3320 SCSI Disk Array (shared by T2000's, Oracle ASM manages this storage)

Preparation

  1. Obtain the Cluster "SCAN" address (Single Client Access Name (SCAN)).
    • nslookup opnrac01-cluster-scan   >>>  should round robin through at least 3 IP addresses on the same subnet as the Cluster listener e.g. --> 10.45.74.6, 10.45.74.7, 10.45.74.8
    • In this scenario, GNS is not configured.
  2. Obtain host names and IP addresses for the physical hosts, LDom hosts,  Virtual IP's, and private IP addresses.

 Server Host IP
 LDom Host IP
 Virtual IP (VIP)
 Cluster Private IP
10.35.74.24 10.45.74.14 10.45.74.10  10.10.10.10
10.35.74.25 10.45.74.15 10.45.74.11  10.10.10.11

 T2000 Network Config
 IP Address Assignment
 Usage  Subnet
 e1000g0  Server Host IP
 Control Domain
 LAN
 e1000g1  LDom Host IP
 Guest Domain
 Same as Cluster SCAN, VIP on LAN
 e1000g2  Cluster Private IP
 Guest Domain
 Private
 e1000g3  TBD  TBD

*Note:

  • The above IP addresses are fictitious addresses
  • Configuring the Storage Array is outside the scope of this document
  • Configuring the T2000 server is outside the scope of this document

Procedure

Configure the Control Domain for LDom

Follow the steps from pages 31-41 as defined in "Beginners Guide to Oracle VM Server for SPARC: Understanding and Deploying Logical Domains" to configure the Control Domain for LDom.

Verify that LDom is configured on the T2000 servers.

# ldm -V

Logical Domain Manager (v 1.2)
        Hypervisor control protocol v 1.3
        Using Hypervisor MD v 1.1

System PROM:
        Hypervisor      v. 1.7.3.       @(#)Hypervisor 1.7.3.c 2010/07/09 15:14
        OpenBoot        v. 4.30.4.      @(#)OBP 4.30.4.b 2010/07/09 13:48
  

Prepare e1000g1 and e1000g2 for LDom as defined in the network config planning table.

# /opt/SUNWldm/bin/ldm add-vsw net-dev=e1000g1 primary-vsw1 primary
# /opt/SUNWldm/bin/ldm add-vsw net-dev=e1000g2 primary-vsw2 primary 
  

The Control Domain is ready for new guest domains.  Verify its available services.

# ldm list-services primary
VCC
    NAME             LDOM             PORT-RANGE
    primary-vcc0     primary          5000-5100

VSW
    NAME             LDOM             MAC               NET-DEV   DEVICE     DEFAULT-VLAN-ID PVID VID                  MTU   MODE
    primary-vsw0     primary          00:14:4f:fa:06:ae e1000g0   switch@0   1   1                         1500
    primary-vsw1     primary          00:14:4f:fa:77:19 e1000g1   switch@1   1   1                         1500
    primary-vsw2     primary          00:14:4f:fa:ff:4f e1000g2   switch@2   1   1                         1500

VDS
    NAME             LDOM             VOLUME         OPTIONS          MPGROUP        DEVICE    
primary-vds0     primary              vol1boot                                       /ldoms/opnaus14ldm/ldom.vdisk
                                      vol1shared                                     /dev/rdsk/c2t0d0s4

 
  

Configuring  Guest Domain (LDom)

This is a domain that hosts a RAC node on the first T2000.  Create another similar domain on the other T2000.

ldm create opn1ldm

ldm set-vcpu 8 opn1ldm
ldm set-memory 8G opn1ldm

ldm add-vnet vnet1 primary-vsw1 opn1ldm
ldm add-vnet vnet2 primary-vsw2 opn1ldm

zfs create dpool/ldoms/opn1ldm
#
# Make sure the LDom's virtual disk for root device is more than 20GB for Solaris, RAC and Oracle DB binaries.
#
mkfile -nv 40G /ldoms/opn1ldm/rvol1.vdisk
ldm add-vdsdev /ldoms/opn1ldm/rvol1.vdisk  vol1boot@primary-vds0
ldm add-vdisk vdisk1 vol1boot@primary-vds0 opn1ldm
#
# This shared disk is managed by ASM from both LDom nodes
#
ldm add-vdsdev /dev/rdsk/c2t0d0s4 vol1shared@primary-vds0
ldm add-vdisk sd1 vol1shared@primary-vds0 opn1ldm

ldm set-variable auto-boot\?=true opn1ldm
#
# Because we have 2 devices, we want to make sure that vdisk1 is the boot device. 
# In the JumpStart Template for this domain, make sure the zfs disk is set properly.
# e.g. base_config_profile_zfs_disk="c0d0s0"
#
ldm set-variable boot-device=vdisk1 opn1ldm

ldm bind opn1ldm
ldm start opn1ldm

Once the LDom is configured, query its status.

# ldm ls
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     4G       2.0%  3d 19h 13m
opn1ldm          active     -n----  5000    8     4G        15%  1h 36m

Make a note of the LDom's console port number: 5000.  It is used for accessing the virtual console of that guest domain.  More details of the LDom can be listed with the following command.  Make note of the MAC address for the JumpStart server configuration.

# ldm ls -l opn1ldm
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
opn1ldm          active     -n----  5000    8     4G       6.7%  1h 40m
MAC
    00:14:4f:f9:ee:8c
HOSTID
    0x84f9ee8c
CONTROL
    failure-policy=ignore
DEPENDENCY
    master=
VCPU
    VID    PID    UTIL STRAND
    0      4      4.2%   100%
    1      5      4.0%   100%
    2      6      3.9%   100%
    3      7      7.8%   100%
    4      8      3.7%   100%
    5      9      8.4%   100%
    6      10     4.1%   100%
    7      11     4.0%   100%
MEMORY
    RA               PA               SIZE
    0x8000000        0x108000000      4G
VARIABLES
    boot-device=disk:a sd1:a vdisk1
NETWORK
    NAME             SERVICE                     DEVICE     MAC               MODE   PVID VID                  MTU
    vnet1            primary-vsw1@primary        network@0  00:14:4f:f8:a1:56      1                         1500
    vnet2            primary-vsw2@primary        network@1  00:14:4f:f8:45:4c      1                         1500
DISK
    NAME             VOLUME                      TOUT DEVICE  SERVER         MPGROUP
    vdisk1           vol1boot@primary-vds0            disk@0  primary
    sd1              vol1shared@primary-vds0          disk@1  primary
VCONS
    NAME             SERVICE                     PORT
    opn1ldm          primary-vcc0@primary        5000

Steps x2

This document is based on a two RAC node scenario, LDom creation and configuration need to be done twice -- once on each node.  However, RAC and DB runInstaller or runcluvfy scripts are run only once and they do all the verification and configuration on defined nodes.

Booting the Guest Domain (LDom)

Once the guest domain is created, it needs to have Solaris OS installed.  There are a number ways to accomplish this task:

  1. Install Solaris OS from DVD
  2. Install Solaris OS from a JumpStart Server
  3. Clone a pre-configured Solaris image from another LDom

 In my case, I installed Solaris OS from my JumpStart Server.

You can access the LDom's virtual console using the telnet command with its console port number from the Control Domain.

# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connecting to console "opn1ldm" in group "opn1ldm" ....
Press ~? for control options ..
{0} ok

*Note: Configuring the JumpStart server is beyond the scope of this document.  If you need more information, refer to the JumpStart Enterprise Toolkit (JET) Wiki.

From the Open Boot Prompt (OBP) (aka the "ok" prompt), type in the command to boot from the JumpStart server.

{0} ok boot vnet1 - install -w

Once the Solaris system is up, back it up with the zfs snapshot command from the Control Domain.

# zfs snapshot /ldoms/opn1ldm@BaseSolaris10u9

Hint: This ZFS snapshot can be used to clone or recover the LDom image.

Verify that all network configuration is plumbed and correct.

Check available devices

# dladm show-dev
vsw0            link: up        speed: 1000  Mbps       duplex: full
vsw1            link: up        speed: 1000  Mbps       duplex: full
vsw2            link: up        speed: 100   Mbps       duplex: full
e1000g0         link: up        speed: 1000  Mbps       duplex: full
e1000g1         link: up        speed: 1000  Mbps       duplex: full
e1000g2         link: up        speed: 100   Mbps       duplex: full
e1000g3         link: unknown   speed: 0     Mbps       duplex: half

Check network interfaces

# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vsw0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 10.35.74.24 netmask ffffff00 broadcast 10.35.74.255
        ether 0:14:4f:fa:06:ae 

Check Cluster SCAN by typing in the nslookup command 3 times and see if it goes round robin through three different IP addresses as defined.

# nslookup opnrac01-cluster-scan

 Check network time synchronization with ntp.

# ntpq -p
     remote           refid      st t when poll reach   delay   offset    disp
==============================================================================
*austin8-rtr-3- root.us.oracle.co  6 u 1003 1024  377     2.27    0.708    1.11

Configure the LDom's Solaris for RAC Installation

The base Solaris 10 installation need to be configured with system and kernel parameters that RAC requires.

Disable Automount of /home

The grid and oracle users' home directories will be created locally in /home so automount will be disabled for this mount point.

# svcs autofs
disabled       13:48:39 svc:/system/filesystem/autofs:default

# vi /etc/auto_master
#/home         auto_home       -nobrowse         <<<<  Comment out this line
# svcadm enable system/filesystem/autofs

Create Groups, Users and Resource Control

groupadd oinstall
groupadd dba
groupadd asmadmin
groupadd asmdba
groupadd asmoper
#
# Solaris 10 Project based Resource Control instead of the use of /etc/system
# No reboot required.  User just need to log back in to inherit changes
#
projadd group.dba
projmod -sK "project.max-shm-memory=(privileged,4G,deny)" group.dba
projmod -sK "project.max-sem-ids=(privileged,100,deny)" group.dba
projmod -sK "project.max-shm-ids=(privileged,100,deny)" group.dba
projmod -sK "project.max-sem-nsems=(privileged,256,deny)" group.dba
projmod -sK "process.max-file-descriptor=(privileged,65536,deny)" group.dba
projects -l group.dba
echo "*** Make sure you have commented out /home in /etc/auto_master ****"
useradd -g oinstall -G dba,asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -K project=group.dba -m grid
useradd -g oinstall -G dba,asmdba -d /home/oracle -s /bin/bash -K project=group.dba -m oracle
passwd oracle
# Enter password for oracle user

passwd grid

# Enter password for grid user

mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory/
mkdir -p /u01/11.2.0/grid
chown -R grid:oinstall /u01/11.2.0/grid
chmod -R 775 /u01/11.2.0/grid
mkdir -p /u01/app/oracle
mkdir /u01/app/oracle/cfgtoollogs
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
mkdir -p /u01/app/oracle/product/11.2.0/db_1
chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/db_1
chmod -R 775 /u01/app/oracle/product/11.2.0/db_1/
chown grid:asmadmin /dev/rdsk/c2t0d0s4
chmod 660 /dev/rdsk/c2t0d0s4

Define the netmask for the Cluster Private Network

# vi /etc/netmasks
10.10.10.0      255.255.255.0       # <<< Add this line

Configure the LDom's Network Interfaces

# ifconfig -a
# vi /etc/hosts
### Add the private host entries
#
# Private Network for RAC Heartbeat
#
10.10.10.10     opn1ldm-priv
10.10.10.11     opn2ldm-priv

# echo "opn1ldm-priv" > /etc/hostname.vnet2
# ifconfig vnet2 plumb
# ifconfig vnet2 11.11.11.10 netmask 255.255.255.0 up

Configure SSH Timeout Parameter

# vi /etc/ssh/ssd_config
#### Look for LoginGraceTime and change the value from 600 to 0
LoginGraceTime 0

Configure Oracle Recommended TCP/UDP Parameters

Create the file /etc/init.d/nddconfig with the following entries.

#!/bin/sh
/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000
/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500
/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000
/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500
/usr/sbin/ndd -set /dev/udp udp_xmit_hiwat 65536
/usr/sbin/ndd -set /dev/udp udp_recv_hiwat 65536 

Link the file accordingly so that when the LDom gets rebooted, the ndd settings would persist.

# ln -s /etc/init.d/nddconfig /etc/rc2.d/S31nddconfig

Configure Network Time Protocol (ntp)

Obtain the NTP server for Cluster Node Time Synchronization.  Add this file  /etc/inet/ntp.conf with following entries including the NTP server IP address. 

# Some of the devices have tuning parameters, called "fudge" factors,
# that can be set on the server line.  See the ntpd documentation.
restrict default ignore
restrict 127.0.0.1
broadcastdelay 0.008
authenticate no
fudge 127.127.1.0 stratum 10
fudge 127.127.1.0 # localhost timeserver
driftfile /var/ntp/ntp.drift
# statsdir /var/ntp/ntpstats/
# filegen peerstats file peerstats type day enable
# filegen loopstats file loopstats type day enable
# filegen clockstats file clockstats type day enable
restrict 10.135.247.1 nomodify notrap
server 10.135.247.1
# Oracle RAC requirements
disable pll
slewalways yes
# End Oracle RAC requirements

Then, enable the NTP service through SMF.

# svcadm enable network/ntp

Verify DNS Configuration

$ cat /etc/resolv.conf
nameserver 192.168.54.86
nameserver 192.168.54.89
search us.oracle.com, oraclecorp.com

If runInstaller complains about redundant entries of "domain" and "search, only one of these lines should be there but not both.

Solution:
Remove either the "domain" or "search" lines from resolv.conf on every cluster node .
Usually it is preferable to have search rather than domain
However the decision of which to remove is up to you.
It's important that the resolv.conf match on all nodes, so be sure to remove the same line from the file on all nodes.

Set up SSH Logins

This step is for manual set up of SSH.  Otherwise, the runInstaller.sh script can handle the steps automatically.

# su - grid

grid: ssh-keygen -t rsa   (Press <RETURN> to take default values.  Run this on both nodes)
grid: cd ~/.ssh
grid: cat > authorized_keys
(copy the key from id_rsa.pub LDom1 to LDom2 and vice versa)
(also copy the key of its own on the same machine i.e. the authorized_keys)
grid: ssh date  (to test it out from both nodes.. no more password prompting)

Should do the same for the "oracle" user.

Install RAC

Before installing RAC, it is a good idea to verify using the cluster verify command:

${GRID_BINARIES}/runcluvfy.sh stage -pre crsinst -n opn1ldm,opn2ldm -r 11gR2 -verbose 

Grid Installer's SSH is pointing to invalid directory  /usr/local/bin

The following workaround can be used:

If some command fails due to wrong path set, user can copy the files locally,
edit and pass the location to runInstaller. for example:

1) Copy file from installation tarball to local directory

    $ cp Disk1/stage/properties /tmp/properties


2) Update the location for the SSH public key and ssh-keygen binary in file
    /tmp/properties/ssPaths_sol.properties to correct path

    $ cat /tmp/ssPaths_sol.properties

     ...
    SSH_HOST_KEY_LOC=/etc/ssh
    SSH_KEYGEN_PATH=/usr/bin/ssh-keygen
       ...


3) reinvoke the runInstaller with specified option:

 
# su - grid
$ ./runInstaller -J-Doracle.sysman.prov.PathsPropertiesLoc=/tmp/properties 

Grid Installation Bug

If "PRVF-5149 Error" is encountered, this can be ignored as written in http://download.oracle.com/docs/cd/E11882_01/readmes.112/e17129/toc.htm  (Nov 2010).

Bug 10041397

During Oracle Grid Infrastructure for a cluster installation on Oracle Virtual Machine (Oracle VM), when virtual shared disks are used as Oracle ASM devices, the prerequisite check for Oracle ASM devices gives a warning that these devices are not shared. For example:

PRVF-5149 : WARNING: Storage "/dev/xvde1" is not shared on all nodes

Workaround:  This warning can be ignored.

Install Database

su - oracle

runInstaller ... Done!

Summary

LDom is a good choice for RAC deployment.  Here are some good reasons:

  • Virtualized hosted environment
  • Easy reallocation/deallocation of system resources dynamically
  • Partitioning physical servers with multiple LDoms to host applications in their own secure and isolated operating environment
  • Optimized system utilization and space/utility consumption
  • Improved management and maintenance with LDom + ZFS
  • Agility gained -- i.e. can be migrated from one physical server to another

Solaris Containers are also a good choice and I'll write another blog entry on that topic.


Comments:

Post a Comment:
Comments are closed for this entry.
About

Mostly pertaining to Cloud Computing, Application Infrastructure, Oracle Exastack, Exalogic, Solaris, Java and Sun servers for the enterprise!

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today