X

Jeff Taylor's Weblog

  • Sun
    August 18, 2014

SPARC T5-4 LDoms for RAC and WebLogic Clusters

This blog is part of the SPARC T5-4 RAC and WebLogic Cluster series:

I wanted to use two Oracle SPARC T5-4 servers to simultaneously host both Oracle RAC and a WebLogic Server Cluster. I chose to use Oracle VM Server for SPARC to create a cluster like this:



There are plenty of trade offs and decisions that need to be made, for example:



Both Oracle SPARC T5-4 servers were to be configured like this:


Cntl

0.25
 4 
64GB

                    App LDom
                   2.75 CPU's                   
                    44 cores                    
                     704 GB        
     DB LDom
     One CPU    
     16 cores   

      256 GB  



The systems started with everything in the primary domain:

# ldm list

NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME

primary          active     -n-c--  UART    512   1023G    0.0%  0.0%  11m


# ldm list-spconfig

factory-default [current]

primary


# ldm list -o core,memory,physio

NAME             

primary          


CORE

    CID    CPUSET

    0      (0, 1, 2, 3, 4, 5, 6, 7)

    1      (8, 9, 10, 11, 12, 13, 14, 15)

    2      (16, 17, 18, 19, 20, 21, 22, 23)

-- SNIP

    62     (496, 497, 498, 499, 500, 501, 502, 503)

    63     (504, 505, 506, 507, 508, 509, 510, 511)


MEMORY

    RA               PA               SIZE            

    0x30000000       0x30000000       255G

    0x80000000000    0x80000000000    256G

    0x100000000000   0x100000000000   256G

    0x180000000000  
0x180000000000   256G # Give this memory block to the DB LDom


IO

    DEVICE                           PSEUDONYM        OPTIONS

    pci@300                          pci_0           

    pci@340                          pci_1           

    pci@380                          pci_2           

    pci@3c0                          pci_3           

    pci@400                          pci_4           

    pci@440                          pci_5           

    pci@480                          pci_6           

    pci@4c0                          pci_7           

    pci@300/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE1

    pci@300/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE2

    pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0

    pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0   

    pci@340/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE3

    pci@340/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE4

    pci@380/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE9

    pci@380/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE10

    pci@3c0/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE11

    pci@3c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE12

    pci@400/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE5

    pci@400/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE6

    pci@440/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE7

    pci@440/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE8

    pci@480/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE13

    pci@480/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE14

    pci@4c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE15

    pci@4c0/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE16

    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1

    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2   



Added an additional service processor configuration:

# ldm add-spconfig split

# ldm list-spconfig
factory-default
primary
split [current]



And removed many of the resources from the primary domain:

# ldm start-reconf primary

# ldm set-core 4 primary

# ldm set-memory 32G primary

# ldm rm-io pci@340 primary

# ldm rm-io pci@380 primary

# ldm rm-io pci@3c0 primary

# ldm rm-io pci@400 primary

# ldm rm-io pci@440 primary

# ldm rm-io pci@480 primary

# ldm rm-io pci@4c0 primary

# init 6



Needed to add resources to the guest domains:

# ldm add-domain db


# ldm set-core cid=`seq -s"," 48 63` db

# ldm add-memory mblock=0x180000000000:256G db



# ldm add-io pci@480 db


# ldm add-io pci@4c0 db


# ldm add-domain app


# ldm set-core 44 app

# ldm set-memory 704G  app


# ldm add-io pci@340 app

# ldm add-io pci@380 app

# ldm add-io pci@3c0 app

# ldm add-io pci@400 app

# ldm add-io pci@440 app





Needed to set up services:

# ldm add-vds primary-vds0 primary
# ldm add-vcc port-range=5000-5100 primary-vcc0 primary



Needed to add a virtual network port for the WebLogic application domain:

# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         ...
   lo0/v6         static     ok           --         ...
net0              ip         ok           --         ...
   net0/v4        static     ok           --         xxx.xxx.xxx.xxx/24
   net0/v6        addrconf   ok           --         ....
   net0/v6        addrconf   ok           --         ...
net8              ip         ok           --         --
   net8/v4        static     ok           --         ...

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             unknown    0      unknown   ixgbe1
net0              Ethernet             up         1000   full      ixgbe0
net8              Ethernet             up         10     full      usbecm2

# ldm add-vsw net-dev=net0 primary-vsw0 primary
# ldm add-vnet vnet1 primary-vsw0 app



Needed to add a virtual disk to the WebLogic application domain:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t5000CCA02505F874d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca02505f874
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD0/disk
       1. c0t5000CCA02506C468d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca02506c468
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD1/disk

       2. c0t5000CCA025067E5Cd0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca025067e5c
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD2/disk
       3. c0t5000CCA02506C258d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
          /scsi_vhci/disk@g5000cca02506c258
          /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD3/disk
Specify disk (enter its number): ^C

# ldm add-vdsdev /dev/dsk/c0t5000CCA02506C468d0s2 HDD1@primary-vds0
# ldm add-vdisk HDD1 HDD1@primary-vds0 app




Add some additional spice to the pot:

# ldm set-variable auto-boot\\?=false db
# ldm set-variable auto-boot\\?=false app
# ldm set-var boot-device=HDD1 app



Bind the logical domains:

# ldm bind db
# ldm bind app



At the end of the process, the system is set up like this:

# ldm list -o core,memory,physio
NAME            
primary         

CORE
    CID    CPUSET
    0      (0, 1, 2, 3, 4, 5, 6, 7)
    1      (8, 9, 10, 11, 12, 13, 14, 15)
    2      (16, 17, 18, 19, 20, 21, 22, 23)
    3      (24, 25, 26, 27, 28, 29, 30, 31)

MEMORY
    RA               PA               SIZE           
    0x30000000       0x30000000       32G

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@300                          pci_0          
    pci@300/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE1
    pci@300/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE2
    pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0
    pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0  
------------------------------------------------------------------------------

NAME            
app             

CORE
    CID    CPUSET
    4      (32, 33, 34, 35, 36, 37, 38, 39)
    5      (40, 41, 42, 43, 44, 45, 46, 47)
    6      (48, 49, 50, 51, 52, 53, 54, 55)
    7      (56, 57, 58, 59, 60, 61, 62, 63)
    8      (64, 65, 66, 67, 68, 69, 70, 71)
    9      (72, 73, 74, 75, 76, 77, 78, 79)
    10     (80, 81, 82, 83, 84, 85, 86, 87)
    11     (88, 89, 90, 91, 92, 93, 94, 95)
    12     (96, 97, 98, 99, 100, 101, 102, 103)
    13     (104, 105, 106, 107, 108, 109, 110, 111)
    14     (112, 113, 114, 115, 116, 117, 118, 119)
    15     (120, 121, 122, 123, 124, 125, 126, 127)
    16     (128, 129, 130, 131, 132, 133, 134, 135)
    17     (136, 137, 138, 139, 140, 141, 142, 143)
    18     (144, 145, 146, 147, 148, 149, 150, 151)
    19     (152, 153, 154, 155, 156, 157, 158, 159)
    20     (160, 161, 162, 163, 164, 165, 166, 167)
    21     (168, 169, 170, 171, 172, 173, 174, 175)
    22     (176, 177, 178, 179, 180, 181, 182, 183)
    23     (184, 185, 186, 187, 188, 189, 190, 191)
    24     (192, 193, 194, 195, 196, 197, 198, 199)
    25     (200, 201, 202, 203, 204, 205, 206, 207)
    26     (208, 209, 210, 211, 212, 213, 214, 215)
    27     (216, 217, 218, 219, 220, 221, 222, 223)
    28     (224, 225, 226, 227, 228, 229, 230, 231)
    29     (232, 233, 234, 235, 236, 237, 238, 239)
    30     (240, 241, 242, 243, 244, 245, 246, 247)
    31     (248, 249, 250, 251, 252, 253, 254, 255)
    32     (256, 257, 258, 259, 260, 261, 262, 263)
    33     (264, 265, 266, 267, 268, 269, 270, 271)
    34     (272, 273, 274, 275, 276, 277, 278, 279)
    35     (280, 281, 282, 283, 284, 285, 286, 287)
    36     (288, 289, 290, 291, 292, 293, 294, 295)
    37     (296, 297, 298, 299, 300, 301, 302, 303)
    38     (304, 305, 306, 307, 308, 309, 310, 311)
    39     (312, 313, 314, 315, 316, 317, 318, 319)
    40     (320, 321, 322, 323, 324, 325, 326, 327)
    41     (328, 329, 330, 331, 332, 333, 334, 335)
    42     (336, 337, 338, 339, 340, 341, 342, 343)
    43     (344, 345, 346, 347, 348, 349, 350, 351)
    44     (352, 353, 354, 355, 356, 357, 358, 359)
    45     (360, 361, 362, 363, 364, 365, 366, 367)
    46     (368, 369, 370, 371, 372, 373, 374, 375)
    47     (376, 377, 378, 379, 380, 381, 382, 383)

MEMORY
    RA               PA               SIZE           
    0x30000000       0x830000000      192G
    0x4000000000     0x80000000000    256G
    0x8080000000     0x100000000000   256G

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@340                          pci_1          
    pci@380                          pci_2          
    pci@3c0                          pci_3          
    pci@400                          pci_4          
    pci@440                          pci_5          
    pci@340/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE3
    pci@340/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE4
    pci@380/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE9
    pci@380/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE10
    pci@3c0/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE11
    pci@3c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE12
    pci@400/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE5
    pci@400/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE6
    pci@440/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE7
    pci@440/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE8
------------------------------------------------------------------------------

NAME            
db              

CORE
    CID    CPUSET
    48     (384, 385, 386, 387, 388, 389, 390, 391)
    49     (392, 393, 394, 395, 396, 397, 398, 399)
    50     (400, 401, 402, 403, 404, 405, 406, 407)
    51     (408, 409, 410, 411, 412, 413, 414, 415)
    52     (416, 417, 418, 419, 420, 421, 422, 423)
    53     (424, 425, 426, 427, 428, 429, 430, 431)
    54     (432, 433, 434, 435, 436, 437, 438, 439)
    55     (440, 441, 442, 443, 444, 445, 446, 447)
    56     (448, 449, 450, 451, 452, 453, 454, 455)
    57     (456, 457, 458, 459, 460, 461, 462, 463)
    58     (464, 465, 466, 467, 468, 469, 470, 471)
    59     (472, 473, 474, 475, 476, 477, 478, 479)
    60     (480, 481, 482, 483, 484, 485, 486, 487)
    61     (488, 489, 490, 491, 492, 493, 494, 495)
    62     (496, 497, 498, 499, 500, 501, 502, 503)
    63     (504, 505, 506, 507, 508, 509, 510, 511)

MEMORY
    RA               PA               SIZE           
    0x80000000       0x180000000000   256G

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@480                          pci_6          
    pci@4c0                          pci_7          
    pci@480/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE13
    pci@480/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE14
    pci@4c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE15
    pci@4c0/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE16
    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1
    pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2  



Start the domains:

# ldm start app
LDom app started

# ldm start db
LDom db started



Make sure to start the vntsd service that was created, above.

# svcs -a | grep ldo
disabled        8:38:38 svc:/ldoms/vntsd:default
online          8:38:58 svc:/ldoms/agents:default
online          8:39:25 svc:/ldoms/ldmd:default

# svcadm enable vntsd




Now use the MAC address to configure the Solaris 11 Automated Installation.


Database Logical Domain
# telnet localhost
5000

{0} ok devalias

screen                  
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@7/display@0


disk7                   
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p3


disk6                   
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p2


disk5                   
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p1


disk4                   
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p0


scsi1                   
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0


net3                    
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0,1


net2                    
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0


virtual-console          /virtual-devices/console@1


name                     aliases

{0} ok boot net2

Boot device:
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0  File
and args:


1000 Mbps full duplex Link up

Requesting Internet Address for xx:xx:xx:xx:xx:xx

Requesting Internet Address for xx:xx:xx:xx:xx:xx
WLS Logical Domain
# telnet localhost
5001
{0} ok devalias
hdd1                    
/virtual-devices@100/channel-devices@200/disk@0

vnet1                   
/virtual-devices@100/channel-devices@200/network@0

net                     
/virtual-devices@100/channel-devices@200/network@0

disk                    
/virtual-devices@100/channel-devices@200/disk@0

virtual-console         
/virtual-devices/console@1

name                     aliases
{0} ok boot net
Boot device:
/virtual-devices@100/channel-devices@200/network@0  File
and args:

Requesting Internet Address for xx:xx:xx:xx:xx:xx

Requesting Internet Address for xx:xx:xx:xx:xx:xx


Repeat the process for the second SPARC T5-4, install Solaris, RAC
and WebLogic Cluster, and you are ready to go. Maybe buying a
SuperCluster would have been easier.

Join the discussion

Comments ( 3 )
  • Julien Gabel Wednesday, August 20, 2014

    Hi,

    Interesting post. But what about applying the max-cores constraint for the db and maybe the app LDOM unless this will leads to higher pricing for Oracle licence, right?

    --

    Best regards,

    Julien Gabel.


  • Jeff Wednesday, August 20, 2014

    Thanks Julien. As I said, "there are plenty of trade offs and decisions that need to be made." Solaris/SPARC has many virtualization alternatives and features, as discussed in "Oracle Solaris 11 Virtualization Technology" < http://www.oracle.com/technetwork/server-storage/solaris11/technologies/virtualization-306056.html >, allowing users to deploy virtualized environments that are specifically tailored to meet the needs of their site. In my case, I wanted to run a benchmark that can be used to recommend the ratio of application-to-database compute power, so I wanted to hold the ratio constant during each test. Your suggestion may very well result in a better configuration for your deployment.


  • Prabhat Wednesday, November 26, 2014

    Very good information, nice blog. Appreciated.


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.