SPARC T5-4 LDoms for RAC and WebLogic Clusters

This blog is part of the SPARC T5-4 RAC and WebLogic Cluster series:

I wanted to use two Oracle SPARC T5-4 servers to simultaneously host both Oracle RAC and a WebLogic Server Cluster. I chose to use Oracle VM Server for SPARC to create a cluster like this:


There are plenty of trade offs and decisions that need to be made, for example:


Both Oracle SPARC T5-4 servers were to be configured like this:


Cntl
0.25
 4 
64GB

                    App LDom
                   2.75 CPU's                   
                    44 cores                    
                     704 GB        
     DB LDom
     One CPU    
     16 cores   

      256 GB  


The systems started with everything in the primary domain:

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-c--  UART    512   1023G    0.0%  0.0%  11m

# ldm list-spconfig
factory-default [current]
primary

# ldm list -o core,memory,physio
NAME             
primary          

CORE
    CID    CPUSET
    0      (0, 1, 2, 3, 4, 5, 6, 7)
    1      (8, 9, 10, 11, 12, 13, 14, 15)
    2      (16, 17, 18, 19, 20, 21, 22, 23)
-- SNIP
    62     (496, 497, 498, 499, 500, 501, 502, 503)
    63     (504, 505, 506, 507, 508, 509, 510, 511)

MEMORY
    RA               PA               SIZE            
    0x30000000       0x30000000       255G
    0x80000000000    0x80000000000    256G
    0x100000000000   0x100000000000   256G
    0x180000000000  
0x180000000000   256G # Give this memory block to the DB LDom

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@300                          pci_0           
    pci@340                          pci_1           
    pci@380                          pci_2           
    pci@3c0                          pci_3           
    pci@400                          pci_4           
    pci@440                          pci_5           
    pci@480                          pci_6           
    pci@4c0                          pci_7           
    pci@300/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE1
    pci@300/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE2
    pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0
    pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0   
    pci@340/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE3
    pci@340/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE4
    pci@380/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE9
    pci@380/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE10
    pci@3c0/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE11
    pci@3c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE12
    pci@400/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE5
    pci@400/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE6
    pci@440/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE7
    pci@440/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE8
    pci@480/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE13
    pci@480/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE14
    pci@4c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE15
    pci@4c0/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE16
    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1
    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2   


Added an additional service processor configuration:

# ldm add-spconfig split

# ldm list-spconfig
factory-default
primary
split [current]


And removed many of the resources from the primary domain:

# ldm start-reconf primary
# ldm set-core 4 primary
# ldm set-memory 32G primary
# ldm rm-io pci@340 primary
# ldm rm-io pci@380 primary
# ldm rm-io pci@3c0 primary
# ldm rm-io pci@400 primary
# ldm rm-io pci@440 primary
# ldm rm-io pci@480 primary
# ldm rm-io pci@4c0 primary
# init 6


Needed to add resources to the guest domains:

# ldm add-domain db

# ldm set-core cid=`seq -s"," 48 63` db

# ldm add-memory mblock=0x180000000000:256G db


# ldm add-io pci@480 db

# ldm add-io pci@4c0 db


# ldm add-domain app

# ldm set-core 44 app
# ldm set-memory 704G  app

# ldm add-io pci@340 app
# ldm add-io pci@380 app
# ldm add-io pci@3c0 app
# ldm add-io pci@400 app
# ldm add-io pci@440 app




Needed to set up services:

# ldm add-vds primary-vds0 primary
# ldm add-vcc port-range=5000-5100 primary-vcc0 primary


Needed to add a virtual network port for the WebLogic application domain:

# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         ...
   lo0/v6         static     ok           --         ...
net0              ip         ok           --         ...
   net0/v4        static     ok           --         xxx.xxx.xxx.xxx/24
   net0/v6        addrconf   ok           --         ....
   net0/v6        addrconf   ok           --         ...
net8              ip         ok           --         --
   net8/v4        static     ok           --         ...

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             unknown    0      unknown   ixgbe1
net0              Ethernet             up         1000   full      ixgbe0
net8              Ethernet             up         10     full      usbecm2

# ldm add-vsw net-dev=net0 primary-vsw0 primary
# ldm add-vnet vnet1 primary-vsw0 app


Needed to add a virtual disk to the WebLogic application domain:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t5000CCA02505F874d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca02505f874
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD0/disk
       1. c0t5000CCA02506C468d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca02506c468
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD1/disk

       2. c0t5000CCA025067E5Cd0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca025067e5c
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD2/disk
       3. c0t5000CCA02506C258d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca02506c258
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD3/disk
Specify disk (enter its number): ^C

# ldm add-vdsdev /dev/dsk/c0t5000CCA02506C468d0s2 HDD1@primary-vds0
# ldm add-vdisk HDD1 HDD1@primary-vds0 app



Add some additional spice to the pot:

# ldm set-variable auto-boot\\?=false db
# ldm set-variable auto-boot\\?=false app
# ldm set-var boot-device=HDD1 app


Bind the logical domains:

# ldm bind db
# ldm bind app


At the end of the process, the system is set up like this:

# ldm list -o core,memory,physio
NAME            
primary         

CORE
    CID    CPUSET
    0      (0, 1, 2, 3, 4, 5, 6, 7)
    1      (8, 9, 10, 11, 12, 13, 14, 15)
    2      (16, 17, 18, 19, 20, 21, 22, 23)
    3      (24, 25, 26, 27, 28, 29, 30, 31)

MEMORY
    RA               PA               SIZE           
    0x30000000       0x30000000       32G

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@300                          pci_0          
    pci@300/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE1
    pci@300/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE2
    pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0
    pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0  
------------------------------------------------------------------------------

NAME            
app             

CORE
    CID    CPUSET
    4      (32, 33, 34, 35, 36, 37, 38, 39)
    5      (40, 41, 42, 43, 44, 45, 46, 47)
    6      (48, 49, 50, 51, 52, 53, 54, 55)
    7      (56, 57, 58, 59, 60, 61, 62, 63)
    8      (64, 65, 66, 67, 68, 69, 70, 71)
    9      (72, 73, 74, 75, 76, 77, 78, 79)
    10     (80, 81, 82, 83, 84, 85, 86, 87)
    11     (88, 89, 90, 91, 92, 93, 94, 95)
    12     (96, 97, 98, 99, 100, 101, 102, 103)
    13     (104, 105, 106, 107, 108, 109, 110, 111)
    14     (112, 113, 114, 115, 116, 117, 118, 119)
    15     (120, 121, 122, 123, 124, 125, 126, 127)
    16     (128, 129, 130, 131, 132, 133, 134, 135)
    17     (136, 137, 138, 139, 140, 141, 142, 143)
    18     (144, 145, 146, 147, 148, 149, 150, 151)
    19     (152, 153, 154, 155, 156, 157, 158, 159)
    20     (160, 161, 162, 163, 164, 165, 166, 167)
    21     (168, 169, 170, 171, 172, 173, 174, 175)
    22     (176, 177, 178, 179, 180, 181, 182, 183)
    23     (184, 185, 186, 187, 188, 189, 190, 191)
    24     (192, 193, 194, 195, 196, 197, 198, 199)
    25     (200, 201, 202, 203, 204, 205, 206, 207)
    26     (208, 209, 210, 211, 212, 213, 214, 215)
    27     (216, 217, 218, 219, 220, 221, 222, 223)
    28     (224, 225, 226, 227, 228, 229, 230, 231)
    29     (232, 233, 234, 235, 236, 237, 238, 239)
    30     (240, 241, 242, 243, 244, 245, 246, 247)
    31     (248, 249, 250, 251, 252, 253, 254, 255)
    32     (256, 257, 258, 259, 260, 261, 262, 263)
    33     (264, 265, 266, 267, 268, 269, 270, 271)
    34     (272, 273, 274, 275, 276, 277, 278, 279)
    35     (280, 281, 282, 283, 284, 285, 286, 287)
    36     (288, 289, 290, 291, 292, 293, 294, 295)
    37     (296, 297, 298, 299, 300, 301, 302, 303)
    38     (304, 305, 306, 307, 308, 309, 310, 311)
    39     (312, 313, 314, 315, 316, 317, 318, 319)
    40     (320, 321, 322, 323, 324, 325, 326, 327)
    41     (328, 329, 330, 331, 332, 333, 334, 335)
    42     (336, 337, 338, 339, 340, 341, 342, 343)
    43     (344, 345, 346, 347, 348, 349, 350, 351)
    44     (352, 353, 354, 355, 356, 357, 358, 359)
    45     (360, 361, 362, 363, 364, 365, 366, 367)
    46     (368, 369, 370, 371, 372, 373, 374, 375)
    47     (376, 377, 378, 379, 380, 381, 382, 383)

MEMORY
    RA               PA               SIZE           
    0x30000000       0x830000000      192G
    0x4000000000     0x80000000000    256G
    0x8080000000     0x100000000000   256G

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@340                          pci_1          
    pci@380                          pci_2          
    pci@3c0                          pci_3          
    pci@400                          pci_4          
    pci@440                          pci_5          
    pci@340/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE3
    pci@340/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE4
    pci@380/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE9
    pci@380/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE10
    pci@3c0/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE11
    pci@3c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE12
    pci@400/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE5
    pci@400/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE6
    pci@440/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE7
    pci@440/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE8
------------------------------------------------------------------------------

NAME            
db              

CORE
    CID    CPUSET
    48     (384, 385, 386, 387, 388, 389, 390, 391)
    49     (392, 393, 394, 395, 396, 397, 398, 399)
    50     (400, 401, 402, 403, 404, 405, 406, 407)
    51     (408, 409, 410, 411, 412, 413, 414, 415)
    52     (416, 417, 418, 419, 420, 421, 422, 423)
    53     (424, 425, 426, 427, 428, 429, 430, 431)
    54     (432, 433, 434, 435, 436, 437, 438, 439)
    55     (440, 441, 442, 443, 444, 445, 446, 447)
    56     (448, 449, 450, 451, 452, 453, 454, 455)
    57     (456, 457, 458, 459, 460, 461, 462, 463)
    58     (464, 465, 466, 467, 468, 469, 470, 471)
    59     (472, 473, 474, 475, 476, 477, 478, 479)
    60     (480, 481, 482, 483, 484, 485, 486, 487)
    61     (488, 489, 490, 491, 492, 493, 494, 495)
    62     (496, 497, 498, 499, 500, 501, 502, 503)
    63     (504, 505, 506, 507, 508, 509, 510, 511)

MEMORY
    RA               PA               SIZE           
    0x80000000       0x180000000000   256G

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@480                          pci_6          
    pci@4c0                          pci_7          
    pci@480/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE13
    pci@480/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE14
    pci@4c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE15
    pci@4c0/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE16
    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1
    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2  


Start the domains:

# ldm start app
LDom app started

# ldm start db
LDom db started


Make sure to start the vntsd service that was created, above.

# svcs -a | grep ldo
disabled        8:38:38 svc:/ldoms/vntsd:default
online          8:38:58 svc:/ldoms/agents:default
online          8:39:25 svc:/ldoms/ldmd:default

# svcadm enable vntsd



Now use the MAC address to configure the Solaris 11 Automated Installation.


Database Logical Domain
# telnet localhost 5000
{0} ok devalias
screen                   /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@7/display@0
disk7                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p3
disk6                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p2
disk5                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p1
disk4                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p0
scsi1                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0
net3                     /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0,1
net2                     /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0
virtual-console          /virtual-devices/console@1
name                     aliases
{0} ok boot net2
Boot device: /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0  File and args:
1000 Mbps full duplex Link up
Requesting Internet Address for xx:xx:xx:xx:xx:xx

Requesting Internet Address for xx:xx:xx:xx:xx:xx
WLS Logical Domain
# telnet localhost 5001
{0} ok devalias
hdd1                     /virtual-devices@100/channel-devices@200/disk@0
vnet1                    /virtual-devices@100/channel-devices@200/network@0
net                      /virtual-devices@100/channel-devices@200/network@0
disk                     /virtual-devices@100/channel-devices@200/disk@0
virtual-console          /virtual-devices/console@1
name                     aliases
{0} ok boot net
Boot device: /virtual-devices@100/channel-devices@200/network@0  File and args:
Requesting Internet Address for xx:xx:xx:xx:xx:xx

Requesting Internet Address for xx:xx:xx:xx:xx:xx

Repeat the process for the second SPARC T5-4, install Solaris, RAC and WebLogic Cluster, and you are ready to go. Maybe buying a SuperCluster would have been easier.

Comments:

Hi,

Interesting post. But what about applying the max-cores constraint for the db and maybe the app LDOM unless this will leads to higher pricing for Oracle licence, right?

--
Best regards,
Julien Gabel.

Posted by Julien Gabel on August 20, 2014 at 10:03 AM EDT #

Thanks Julien. As I said, "there are plenty of trade offs and decisions that need to be made." Solaris/SPARC has many virtualization alternatives and features, as discussed in "Oracle Solaris 11 Virtualization Technology" < http://www.oracle.com/technetwork/server-storage/solaris11/technologies/virtualization-306056.html >, allowing users to deploy virtualized environments that are specifically tailored to meet the needs of their site. In my case, I wanted to run a benchmark that can be used to recommend the ratio of application-to-database compute power, so I wanted to hold the ratio constant during each test. Your suggestion may very well result in a better configuration for your deployment.

Posted by Jeff on August 20, 2014 at 11:32 AM EDT #

Very good information, nice blog. Appreciated.

Posted by Prabhat on November 26, 2014 at 07:17 AM EST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Jeff Taylor-Oracle

Search

Archives
« April 2015
SunMonTueWedThuFriSat
   
1
2
4
5
6
8
9
10
11
12
13
14
15
16
17
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today