Thursday Jul 05, 2012

LDoms with Solaris 11

Oracle VM Server for SPARC (LDoms) release 2.2 came out on May 24. You can get the software, see the release notes, reference manual, and admin guide here on the Oracle VM for SPARC page.

Oracle VM Server for SPARC enables you to create multiple virtual systems on a single physical system.Each virtual system is called alogical domain and runs its own instance of Oracle Solaris 10 or Oracle Solaris 11.

The version of the Oracle Solaris OS software that runs on a guest domain is independent of the Oracle Solaris OS version that runs on the primary domain. So, if you run the Oracle Solaris 10 OS in the primary domain, you can still run the Oracle Solaris 11 OS in a guest domain, and if you run the Oracle Solaris 11 OS in the primary domain, you can still run the Oracle Solaris 10 OS in a guest domain

In addition to that starting with the Oracle VM Server for SPARC 2.2 release you can migrate guest domain even if source and target machines have different processor type.

You can migrate guest domain from a system with UltraSPARC  T2+ or SPARC T3 CPU to a system with a SPARC T4 CPU.The guest domain on the source and target system must run Solaris 11 In order
to enable cross CPU migration.In addition to that you need to change the cpu-arch property value on the source system.

For more information about Oracle VM Server for SPARC (LDoms) with Solaris 11 and  Cross CPU Migration refer to the following white paper

Thursday Nov 12, 2009

Performance Study / Best Practices for running MySQL on Xen Based Hypervisors


This blog entry provides technical insight into the benchmark of the MySQL database on Xen Virtualization environment based on the xVM Hypervisor

Introduction to the xVM Hypervisor

 xVM hypervisor can securely execute multiple virtual machines simultaneously, each running its own operating system, on a single physical system. Each virtual machine instance is called a domain. There are two kinds of domains. The control domain is called domain0, or dom0. A guest OS, or unprivileged domain, is called a domainU or domU. Unlike virtualization using zones, each domain runs a full instance of an operating system.

Introduction to the MySQL database

 MySQL database is the world's most popular open source database because of its fast performance, high reliability, ease of use, and dramatic cost savings.

Tests Objective:

The main objective is to bring an understanding on how MySQL behaves within a virtualized environment, using UFS or ZFS file system

Tests Description:

We built a test environment by using a Sun X4450 MySQL 5.4 was installed on OpenSolaris 2009_06 because of the OS built-in integration with the xVM Hypervisor . A separate set of performance tests was run with MySQL data placed on a SAN disk. xVM guest OS is OpenSolaris 2009_06 .

When running under xVM the server resources ( cpu, memory) were divided between the dom0 and domU guest OS.

 dom0 - 2 vcpu and 2GB RAM

domU -  4 vcpu and 6GB RAM

  • We used paravirtualized domU operating system in order to get the best performance.

  • We chose to analyze the performance behavior for InnoDB storage engine due to its high popularity.

  • We chose to analyze the performance behavior for two file systems (ZFS and UFS) in order to check which file system performs better for MySQL .

SysBench was used as loading tool to test base performance for each configuration.

The tool is simple to use, modular, cross-platform and multi-threaded. It also and can give a good feeling regarding the performance for a simple database use.  

Hardware configuration:

Server :SUN X4450 ,with 2X 2.9GHz dual-core CPU,8GB RAM , 2 X 146 GB internal disks.

Storage :StorageTek 6140 - configured RAID 0+1, directly attached to the server.

Software: MySQL 5.4, OpenSolaris 2009_06 .

The SysBench script

sysbench --test=oltp --mysql-table-engine=innodb --oltp-table-size=10000000 --mysql-socket=/tmp/mysql.sock --mysql-user=root prepare sysbench --num-threads=8 --max-time=900 --max-requests=500000 --test=oltp --mysql-user=root --mysql-host=localhost --mysql-port=3306 --mysql-table-engine=innodb --oltp-test-mode=complex --oltp-table-size=80000000 run

We tested it with different number of threads  4 ,8 ,16 ,32  (--num-threads=8 )  

The benchmark layout

After the creation of OpenSolaris 2009_06 in domU we attached the SAN storage

Attached the file system to the guest

xm block-attach para-opensolaris phy:/dev/dsk/c0t600A0B8000267DD400000A8D494DB1A6d0p0 3 w

Verified access to the file system from the guest

root@para-opensolaris:~# format
Searching for disks...done

     0. c7d3 <DEFAULT cyl 4096 alt 0 hd 128 sec 32>
     1. c7t0d0 <DEFAULT cyl 3915 alt 0 hd 255 sec 63>

zpool create -f xvmpool c7d3

root@para-opensolaris:~# zpool list
rpool    29.8G  10.6G  19.2G    35%  ONLINE  -
xvmpool  117.94G  89.5K  117.94G     0%  ONLINE  -

The first result of running this benchmark on UFS

The first result of running this benchmark on ZFS

The results after match ZFS recored size to block size and limiting  ZFS ARC size

zfs create -o recordsize=16k xvmpool/mysql

set zfs:zfs_arc_max = 0x10000000   in /etc/system

The results after disable ZFS cache flush ( We have battery backed cache) 

set zfs:zfs_nocacheflush = 1   in /etc/system


After ZFS tuning we were able to receive the same results as UFS  

Thus, we can benefit from ZFS extra features like snapshot and clone.

For more information about ZFS and OpenSolaris

Sunday Aug 02, 2009

Logical Domains Physical-to-Virtual (P2V) Migration

The Logical domains P2V migration tool automatically converts an existing physical system
to a virtual system that runs in a logical domain on a chip multithreading (CMT) system.
The source system can be any of the following:

    \*  Any sun4u SPARC system that runs at least the Solaris 8 Operating System
    \*  Any sun4v system that runs the Solaris 10 OS, but does not run in a logical domain

In this entry I will demonstrate how to use Logical domains P2V migration tool to migrate Solaris running on V440 server (physical)
Into guest domain running on the T5220 server (virtual)
Architecture layout :

Before you can run the LogicalDomains P2VMigration Tool, ensure that the following are true:

  •     Target system runs at least LogicalDomains 1.1 on the following:

  •      Solaris 10 10/08 OS

  •     Solaris 10 5/08 OS with the appropriate LogicalDomains 1.1 patches

  •     Guest domains run at least the Solaris 10 5/08 OS

  •     Source system runs at least the Solaris 8 OS


In addition to these prerequisites, configure an NFS file system to be shared by both the source
and target systems. This file system should be writable by root.However, if a shared file system
is not available, use a local file system that is large enough to hold a file system dump of the
source system on both the source and target systems Limitations
Version 1.0 of the LogicalDomains P2VMigration Tool has the following limitations:

  •      Only UFS file systems are supported.

  •      Each guest domain can have only a single virtual switch and virtual disk

  •      The flash archiving method silently ignores excluded file systems.

The conversion from a physical system to a virtual system is performed in the following phases:

  • Collection phase   Runs on the physical source system. collect creates a file system image of the source system based on the configuration information that it collects about the source system.

  • Preparation phase. Runs on the control domain of the target system. prepare creates the logical domain on the target system based on the configuration information collected in the collect phase.

  • Conversion phase. Runs on the control domain of the target system. In the convert phase,the created logical domain is converted into a logical domain that runs the Solaris 10 OS by using the standard Solaris upgrade process.

    Collection phase
    On the target machine T5220

    Prepare the NFS server in order to hold the a file system dump of the source system on both the source and target systems.
    In this use case I will use the target machine (T5220) as the NFS server
    # mkdir /p2v
    # share -F nfs -o root=v440 /p2v

    Verify the NFS share
    # share
    -  /p2v  root=v440  ""
    Install the Logical Domains P2V MigrationTool
    Go to the Logical Domains download page at
    Download the P2V software package, SUNWldmp2v
    Use the pkgadd command to install the SUNWldmp2v package
    # pkgadd -d . SUNWldmp2v

    Create the /etc/ldmp2v.conf file we will use it later
    # cat /etc/ldmp2v.conf

    On the source machine V440
    Install the Logical Domains P2V MigrationTool
    # pkgadd -d . SUNWldmp2v
    Mount the NFS share
    # mkdir /p2v
    # mount t5220:/p2v /p2v
    Run the collection command
    # /usr/sbin/ldmp2v collect -d /p2v/v440
    Collecting system configuration ...
    Archiving file systems ...
    DUMP: Date of this level 0 dump: August 2, 2009 4:11:56 PM IDT
    DUMP: Date of last level 0 dump: the epoch
    DUMP: Dumping /dev/rdsk/c1t0d0s0 (mdev440-2:/) to /p2v/v440/ufsdump.0.
    The collection phase took 5 minutes for 4.6 GB dump file
    Preparation phase
    On the target machine T5220
    Run the preparation command
    We will keep the source machine (V440) mac address
    # /usr/sbin/ldmp2v prepare -d /p2v/v440 -o keep-mac v440
     Creating vdisks ...
    Creating file systems ...
    Populating file systems ...
    The preparation phase took 26 minutes

    We can see that for each physical cpu on the V440 server the LDom P2V Tool create 4 vcpu on the guest domain and assigns the same amount memory that the physical system has
    # ldm list -l v440

    v440                   inactive   ------                                   4     8G



        NAME             SERVICE                     DEVICE     MAC               MODE   PVID VID                  MTU
        vnet0            primary-vsw0                           00:03:ba:c4:d2:9d        1

        NAME             VOLUME                      TOUT DEVICE  SERVER         MPGROUP
        disk0            v440-vol0@primary-vds0

    Conversion Phase
    Before starting the conversion phase shout down the source server (V440) in order to avoid ip address conflict
    On the V440 server
    # poweroff
    On the jumpstart server

    You can use the Custom JumpStart feature to perform a completely hands-off conversion.
    This feature requires that you create and configure the appropriate sysidcfg and profile files for the client on the JumpStart server.
    The profile should consist of the following lines: install_type upgrade root_device  c0d0s0

    The sysidcfg file :

    timeserver=localhost timezone=Europe/Amsterdam
    terminal=vt100 security_policy=NONE nfs4_domain=dynamic network_interface=PRIMARY {netmask= default_route=none protocol_ipv6=no}
    On the target server T5220
    # ldmp2v convert -j -n vnet0 -d /p2v/v440 v440Testing original system status ...
    LDom v440 started
    Waiting for Solaris to come up ...
    Using Custom JumpStart
    Connected to 0.
    Escape character is '\^]'.

    Connecting to console "v440" in group "v440" ....
    Press ~? for control options ..
    SunOS Release 5.10 Version Generic_139555-08 64-bit
    Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
    Use is subject to license terms.

    For information about the P2V migration tool, see the ldmp2v(1M) man page.

Thursday Jul 02, 2009

Storage virtualization with COMSTAR and ZFS

COMSTAR is a software framework that enables you to turn any
OpenSolaris host into a SCSI target that can be accessed over the
network by initiator hosts. COMSTAR breaks down the huge task of
handling a SCSI target subsystem into independent functional modules.
These modules are then glued together by the SCSI Target Mode Framework (STMF).

COMSTAR features include:

    \* Extensive LUN Masking and mapping functions
    \* Multipathing across different transport protocols
    \* Multiple parallel transfers per SCSI command
    \* Scalable design
    \* Compatible with generic HBAs

COMSTAR is integrated into the latest OpenSolaris

In this entry I will demonstrate the integration between COMSTAR and ZFS

Architecture layout :

You can install all the appropriate COMSTAR packages

# pkg install storage-server
On a newly installed OpenSolaris system, the STMF service is disabled by default.

You must complete this task to enable the STMF service.
View the existing state of the service
# svcs stmf
 disabled 15:58:17 svc:/system/stmf:default

Enable the stmf service
# svcadm enable stmf
Verify that the service is active.
# svcs stmf
 online 15:59:53 svc:/system/stmf:default

Create a RAID-Z storage pool.
The server has six controllers each with eight disks and I have
built the storage pool to spread I/O evenly and to enable me to build 8
 RAID-Z stripes of equal length.
# zpool create -f  tank \\
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \\
raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \\
raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \\
raidz c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \\
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0

After the pool is created, the zfs utility can be used to create 50GB ZFS volume.
# zfs create -V 50g tank/comstar-vol1

Create a logical unit using the volume.
# sbdadm create-lu /dev/zvol/rdsk/tank/comstar-vol1
Created the following logical unit :

--------------------------- ------------------- ----------------
600144f07bb2ca0000004a4c5eda0001 53687025664 /dev/zvol/rdsk/tank/comstar-vol1

Verify the creation of the logical unit and obtain the Global Unique Identification (GUID) number for the logical unit.
# sbdadm list-lu
Found 1 LU(s)
-------------------------------- ------------------- ----------------
600144f07bb2ca0000004a4c5eda0001 53687025664 /dev/zvol/rdsk/tank/comstar-vol1

This procedure makes a logical unit available to all initiator hosts on a storage network.
Add a view for the logical unit.

# stmfadm add-view GUID_number

Identify the host identifier of the initiator host you want to add to your view.
Follow the instructions for each port provider to identify the initiators associated with each
port provider.

You can see that the port mode is Initiator

# fcinfo hba-port
       HBA Port WWN: 210000e08b91facd
        Port Mode: Initiator
        Port ID: 2
        OS Device Name: /dev/cfg/c16
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402R00-0634175788
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b91facd
        Max NPIV Ports: 63
        NPIV port list:

Before making changes to the HBA ports, first check the existing port
View what is currently bound to the port drivers.
In this example, the current binding is pci1077,2422.
# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic
cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci zfs sd sockfs ip hook
neti sctp arp usba qlc fctl  
fcp cpc random crypto stmf nfs lofs logindmux ptm ufs sppp nsctl ipc ]
> ::devbindings -q qlc
ffffff04e058f560 pci1077,2422, instance #0 (driver name: qlc)
ffffff04e058f2e0 pci1077,2422, instance #1 (driver name: qlc)

Quit mdb.
> $q
Remove the current binding, which in this example is qlc.
In this example, the qlc driver is actively bound to pci1077,2422.
You must remove the existing binding for qlc before you can add that
binding to a new driver.

 Single quotes are required in this syntax.
# update_drv -d -i 'pci1077,2422' qlc
Cannot unload module: qlc
Will be unloaded upon reboot.

This message does not indicate an error.
The configuration files have been updated but the qlc driver remains
bound to the port until reboot.
Establish the new binding to qlt.
Single quotes are required in this syntax.
# update_drv -a -i 'pci1077,2422' qlt
  Warning: Driver (qlt) successfully added to system but failed to

This message does not indicate an error. The qlc driver remains bound
to the port, until reboot.
The qlt driver attaches when the system is rebooted.
Reboot the system to attach the new driver, and then recheck the
# reboot
# mdb -k

Loading modules: [ unix genunix specfs dtrace mac cpu.generic
cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci zfs sd sockfs ip hook
neti sctp arp usba fctl stmf lofs fcip cpc random crypto nfs logindmux
ptm ufs sppp nsctl ipc ]
> ::devbindings -q qlt
ffffff04e058f560 pci1077,2422, instance #0 (driver name: qlt)
ffffff04e058f2e0 pci1077,2422, instance #1 (driver name: qlt)

Quit mdb.
 > $q

You can see that the port mode is Target

# fcinfo hba-port

        HBA Port WWN: 210000e08b91facd
        Port Mode: Target
        Port ID: 2
        OS Device Name: /dev/cfg/c16
        Manufacturer: QLogic Corp.
        Model: 375-3294-01
        Firmware Version: 04.04.01
        FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
        Serial Number: 0402R00-0634175788
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 200000e08b91facd
        Max NPIV Ports: 63
        NPIV port list:

Verify that the target mode framework has access to the HBA ports.
# stmfadm list-target -v

Target: wwn.210100E08BB1FACD
Operational Status: Online
Provider Name : qlt
Alias : qlt1,0
Sessions : 0
Target: wwn.210000E08B91FACD
Operational Status: Online
Provider Name : qlt
Alias : qlt0,0
Sessions : 1
Initiator: wwn.210000E08B89F077
Alias: -
Logged in since: Thu Jul 2 12:02:59 2009

Now for the client setup :

On the client machine verify that you can see the new logical unit
# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0                 CD-ROM       connected    configured   unknown
c1                             scsi-bus     connected    configured   unknown
c1::dsk/c1t0d0                 disk         connected    configured   unknown
c1::dsk/c1t2d0                 disk         connected    configured   unknown
c1::dsk/c1t3d0                 disk         connected    configured   unknown
c2                             fc-private   connected    configured   unknown
c2::210000e08b91facd           disk         connected    configured   unknown
c3                             fc           connected    unconfigured unknown
usb0/1                         unknown      empty        unconfigured ok
usb0/2                         unknown      empty        unconfigured ok

You might need to rescan the SAN BUS in order to discover the new logical unit
# luxadm -e forcelip /dev/cfg/c2
# format
Searching for disks...
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
1. c1t2d0 <FUJITSU-MAV2073RCSUN72G-0301-68.37GB>
2. c1t3d0 <FUJITSU-MAV2073RCSUN72G-0301-68.37GB>
3. c2t210000E08B91FACDd0 <SUN-COMSTAR-1.0 cyl 65533 alt 2 hd 16 sec 100>
Specify disk (enter its number):

You can see the SUN-COMSTAR-1.0 in the disk properties
Now you can build storage pool on top of it
# zpool create comstar-pool c2t210000E08B91FACDd0
Verify the pool creation
comstar-pool 49.8G 114K 49.7G 0% ONLINE -

After the pool is created, the zfs utility can be used to create a ZFS volume.
#  zfs create -V 48g comstar-pool/comstar-vol1

For more information about COMSTAR please check the COMSTAR  project on OpenSolaris

Sunday Jun 14, 2009

3rd annual Java Technology Day June 22nd, 2009

Lots of cool staff  JavaFX,  Virtualization , Cloud computing , Mysql

I will have two sessions in the event :

1. Virtualization from the Desktop to the Enterprise

2. Sun in front of the cloud.

I am looking forward to see you 

Tuesday Apr 21, 2009

IGT Cloud Computing WG Meeting

Yesterday Sun hosted the Israeli Association of Grid Technology (IGT) event "IGT Cloud Computing WG Meeting" at the Sun office in Herzeliya. During the event Nati Shalom, CTO GigaSpaces, Moshe Kaplan, CEO Rocketier , Haim Yadid, Performance Expert, ScalableJ, presented various cloud computing technologies,There were 50 attendees from a wide breadth of technology firms.

For more information regarding using Sun's cloud see .

meeting agenda :

Auto-Scaling Your Existing Web Application Nati Shalom, CTO, Gigaspaces


In this session, will cover how to take a standard JEE web application and scale it out  or down dynamically, without changes to the application code. Seeing as most web applications are over-provisioned to meet  infrequent peak loads, this is a dramatic change, because it enables growing your application as needed, when needed, without paying for unutilized resources.  Nati will discuss the challenges involved in dynamic scaling, such as ensuring the integrity and consistency of the application, how to keep the load-balancer in sync with servers' changing location, and how to maintain affinity and high availability of session information with the load balancer. If time permits, Nati  will show a live demo of a Web 2.0 app scaling dynamically on the Amazon cloud.


How your very large databases can work in the cloud computing world?
Moshe Kaplan, RockeTier, a performance expert and scale out architect
Cloud computing is famous for its flexibility, dynamic nature and ability to infinite growth. However, infinite growth means very large databases with billions of records in it. This leads us to a paradox: "How can weak servers support very large databases which usually require several CPUs and dedicated hardware?"
The Internet industry proved it can be done. These days many of the Internet giants, processing billions of events every day, are based on cloud computing architecture such and sharding. What is Sharding ? What kinds of Sharding can you implement? What are the best practices?

Utilizing the cloud for Performance Validation
Haim Yadid, Performance Expert, ScalableJ

Creating Loaded environment is crucial for software performance validation. Execution of such a simulated environment required usually great deal of hardware which is then left unused during most of the development cycle. In this short session I will suggest utilizing cloud computing for performance validation. I will present a case study where loaded environment used 12 machines on AWS for the duration of the test. This approach gives much more flexibility and reduces TCO dramatically. We will discuss the limitation of this approach and suggest means to address them.


Sunday Apr 19, 2009

Ldom with ZFS

Logical Domains offers a powerful and consistent methodology for creating virtualized server environments across the entire CoolThreads server range:

   \* Create multiple independent virtual machines quickly and easily
     using the hypervisor built into every CoolThreads system.
   \* Leverage advanced Solaris technologies such as ZFS cloning and
     snapshots to speed deployment and dramatically reduce disk
     capacity requirements.

In this entry I will demonstrate the integration between Ldom and ZFS

Architecture layout

Downloading Logical Domains Manager and Solaris Security Toolkit

Download the Software

Download the zip file ( from the Sun Software Download site. You can find the software from this web site:

 Unzip the zip file.
# unzip
Please read the REDME file for any prerequisite
The installation script is part of the SUNWldm package and is in the Install subdirectory.

# cd LDoms_Manager-1_1

Run the install-ldm installation script with no options.
# Install/install-ldm

Select a security profile from this list:

a) Hardened Solaris configuration for LDoms (recommended)
b) Standard Solaris configuration
c) Your custom-defined Solaris security configuration profile

Enter a, b, or c [a]: a

Shut down and reboot your server
# /usr/sbin/shutdown -y -g0 -i6

Use the ldm list command to verify that the Logical Domains Manager is running
# /opt/SUNWldm/bin/ldm list

primary active -n-c-- SP 32 16256M 0.0% 2d 23h 27m

Creating Default Services

You must create the following virtual default services initially to be able to use them later:
vdiskserver – virtual disk server
vswitch – virtual switch service
vconscon – virtual console concentrator service

Create a virtual disk server (vds) to allow importing virtual disks into a logical domain.
# ldm add-vds primary-vds0 primary

Create a virtual console concentrator (vcc) service for use by the virtual network terminal server daemon (vntsd)
# ldm add-vcc port-range=5000-5100 primary-vcc0 primary

Create a virtual switch service
(vsw) to enable networking between virtual network
(vnet) devices in logical domains
# ldm add-vsw net-dev=e1000g0 primary-vsw0 primary

Verify the services have been created by using the list-services subcommand.

# ldm list-services

Set Up the Control Domain

Assign cryptographic resources to the control domain.
# ldm set-mau 1 primary

Assign virtual CPUs to the control domain.
# ldm set-vcpu 4 primary

Assign memory to the control domain.
# ldm set-memory 4G primary

Add a logical domain machine configuration to the system controller (SC).
# ldm add-config initial

Verify that the configuration is ready to be used at the next reboot
# ldm list-config

initial [next poweron]

Reboot the server
# shutdown -y -g0 -i6

Enable the virtual network terminal server daemon, vntsd
# svcadm enable vntsd

Create the zpool

# zpool create ldompool c1t2d0 c1t3d0

# zfs create ldompool/goldimage

# zfs create -V 15g ldompool/goldimage/disk_image

Creating and Starting a Guest Domain

Create a logical domain.
# ldm add-domain goldldom

Add CPUs to the guest domain.
ldm add-vcpu 4 goldldom

Add memory to the guest domain
# ldm add-memory 2G goldldom

Add a virtual network deviceto the guest domain.
# ldm add-vnet vnet1 primary-vsw0 goldldom

Specify the device to be exported by the virtual disk server as a virtual disk to the guest domain
# ldm add-vdsdev /dev/zvol/dsk/ldompool/goldimage/disk_image vol1@primary-vds0

Add a virtual disk to the guest domain.
# ldm add-vdisk vdisk0 vol1@primary-vds0 goldldom

Set auto-boot and boot-device variables for the guest domain
# ldm set-variable auto-boot\\?=false goldldom
# ldm set-var boot-device=vdisk0 goldldom

Bind resources to the guest domain goldldom and then list the domain to verify that it is bound.
# ldm bind-domain goldldom
# ldm list-domain goldldom

primary          active     -n-cv-  SP      4     4G       0.2%  15m
goldldom         bound      ------  5000    4     2G

Start the guest domain
# ldm start-domain goldldom
Connect to the console of a guest domain
# telnet 0 5000
Connected to 0.
Escape character is '\^]'.
Connecting to console "goldldom" in group "goldldom" ....
Press ~? for control options ..

{0} ok

Jump-Start the goldldom

{0} ok boot net - install
We can login to the new guest and verify that the file system is zfs

# zpool list
rpool  14.9G  1.72G  13.2G    11%  ONLINE  -
Restore the goldldom configuration to an "as-manufactured" state with the sys-unconfig command

# sys-unconfig
This program will unconfigure your system.  It will cause it
to revert to a "blank" system - it will not have a name or know
about other systems or networks.
This program will also halt the system.
Do you want to continue (y/n) y

Press ~. in order to return to the primary domain

Stop the guest domain
# ldm stop goldldom
Unbind the guest domain

# ldm unbind  goldldom
Snap shot the disk image
# zfs snapshot ldompool/goldimage/disk_image@sysunconfig

Create new zfs file system for the new guest
# zfs create ldompool/domain1

Clone the goldldom disk image
# zfs clone ldompool/goldimage/disk_image@sysunconfig ldompool/domain1/disk_image

# zfs list
ldompool 17.0G 117G 21K /ldompool
ldompool/domain1 18K 117G 18K /ldompool/domain1
ldompool/domain1/disk_image 0 117G 2.01G -
ldompool/goldimage 17.0G 117G 18K /ldompool/goldimage
ldompool/goldimage/disk_image 17.0G 132G 2.01G -
ldompool/goldimage/disk_image@sysunconfig 0 - 2.01G -

Creating and Starting the second  Domain

# ldm add-domain domain1
# ldm add-vcpu 4 domain1
# ldm add-memory 2G domain1
# ldm add-vnet vnet1 primary-vsw0 domain1
# ldm add-vdsdev /dev/zvol/dsk/ldompool/domain1/disk_image vol2@primary-vds0
# ldm add-vdisk vdisk1 vol2@primary-vds0 domain1
# ldm set-var auto-boot\\?=false domain1
# ldm set-var boot-device=vdisk1 domain1

# ldm bind-domain domain1
# ldm list-domain domain1
domain1 bound ------ 5001 8 2G

Start the domain
# ldm start-domain domain1

Connect to the console
# telnet 0 5001
{0} ok boot net -s

Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Booting to milestone "milestone/single-user:default".
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet0...
Configured interface vnet0
Requesting System Maintenance Mode

# zpool import -f rpool
# zpool export rpool
# reboot

Answer the configuration questions

Login to the new domain and verify that we have zfs file system

# zpool list
rpool 14.9G 1.72G 13.2G 11% ONLINE -

Monday Mar 16, 2009

Brief Technical Overview and Installation of Ganglia on Solaris

Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters.

For this setup we will use the following software packages:

1. Ganglia - the core Ganglia package

2. Zlib - zlib compression libraries

3. Libgcc - low-level runtime library

4. Rrdtool - round Robin Database graphing tool

5. Apache web server with php support

You can get the  packagers ( 1-3)  from sunfreeware (depending on your architecture -  x86 or SPARC)

Unzip and Install the packages

1. gzip -d ganglia-3.0.7-sol10-sparc-local.gz

pkgadd -d ./ganglia-3.0.7-sol10-sparc-local

2. gzip -d zlib-1.2.3-sol10-sparc-local.gz

pkgadd -d ./zlib-1.2.3-sol10-sparc-local

3. gzip -d libgcc-3.4.6-sol10-sparc-local.gz

pkgadd -d ./libgcc-3.4.6-sol10-sparc-local

4. You will need pkgutil from blastwave in order to install rrdtool software packages


gunzip pkgutil-1.2.1,REV=2008.11.28-SunOS5.8-sparc-CSW.pkg.gz

pkgadd -d pkgutil-1.2.1,REV=2008.11.28-SunOS5.8-sparc-CSW.pkg


Now you can install packages with all required dependencies with a single command:

/opt/csw/bin/pkgutil -i rrdtool
5. You will need to download Apache ,PHP and Core libraries from Cool stack

Core libraries used by other packages

bzip2 -d CSKruntime_1.3.1_sparc.pkg.bz2

pkgadd -d ./CSKruntime_1.3.1_sparc.pkg

Apache 2.2.9,  PHP 5.2.6

bzip2 -d CSKamp_1.3.1_sparc.pkg.bz2

pkgadd -d ./CSKamp_1.3.1_sparc.pkg

The following packages are available:

1 CSKapache2 Apache httpd

(sparc) 2.2.9

2 CSKmysql32 MySQL 5.1.25 32bit

(sparc) 5.1.25

3 CSKphp5 PHP 5

(sparc) 5.2.6

Select package(s) you wish to process (or 'all' to process

all packages). (default: all) [?,??,q]:1,3

Select the 1 and 3 option

Enable the web server service

svcadm enable svc:/network/http:apache22-csk

Verify it is working

svcs svc:/network/http:apache22-csk

STATE          STIME    FMRI

online         17:02:13 svc:/network/http:apache22-csk

 Locate the Web server  DocumentRoot

grep DocumentRoot /opt/coolstack/apache2/conf/httpd.conf

DocumentRoot "/opt/coolstack/apache2/htdocs"

Copy the Ganglia directory tree

cp -rp /usr/local/doc/ganglia/web  /opt/coolstack/apache2/htdocs/ganglia

Change the rrdtool path on  /opt/coolstack/apache2/htdocs/ganglia/conf.php

from /usr/bin/rrdtool  to /opt/csw/bin/rrdtool



Start the gmond daemon with the default configuration

/usr/local/sbin/gmond --default_config > /etc/gmond.conf

Edit /etc/gmond.conf  ,change  name = "unspecified"  to name="grid1"  (This is our grid name.)

Verify that it has started : 

ps -ef | grep gmond
nobody 3774 1 0 16::57:41 ? 0:55 /usr/local/gmond

In order to debug any problem, try:

/usr/local/sbin/gmond --debug=9

Build the directory for the rrd images

mkdir -p /var/lib/ganglia/rrds
chown -R nobody  /var/lib/ganglia/rrds
Add the folowing line to /etc/gmetad.conf

data_source "grid1"  localhost

Start the gmetad daemon


Verify it -->

 ps -ef | grep gmetad

nobody  4350     1   0 17:10:30 ?           0:24 /usr/local/sbin/gmetad

To debug any problem

/usr/local/sbin/gmetad --debug=9

Point your browser to: http://server-name/ganglia

Wednesday Mar 04, 2009

Technical overview GlassFish 3.0 on Amazon cloud

The integrated GigaSpaces GlassFish solution with its components is captured in the following diagram :


   SLA Driven deployment environment:

The SLA Driven deployment environment is responsible for hosting all services in the network. It basically does match making between the application requirements and the availability of the resources over the network. It is comprised of the following components:

    • Grid Service Manager - GSM – responsible for managing the application lifecycle and deployment

    • Grid Service Container GSC – a light weight container which is essentially a wrapper on top of the Java process that exposes the JVM to the GSM and provides a means to deploy and undeploy services dynamically.

    • Processing-Unit (PU )– Represents the application deployment unit. A Processing Unit is essentially an extension of the spring application context that packages specific application components in a single package and uses dependency injection to mesh together these components. The Processing Unit is an atomic deployment artifact and its composition is determined by the scaling and failover granularity of a given application. It, therefore,  is the unit-of-scale and failover. There are number of pre-defined Processing Unit types :

      • Web Processing Unit Web Processing Unit is responsible for managing Web Container instances and enables them to run within SLA driven container environment. With a Web Processing Unit, one can deploy the Web Container as group of services and apply SLAs or QoS semantics such as one-per-vm, one-per-machine, etc.  In other words, one can easily use the Processing Unit SLA to determine how web containers would be provisioned on the network. In our specific case most of the GlassFish v3 Prelude integration takes place at this level.

      • Data Grid Processing Unit Data Grid is a processing unit that wraps the GigaSpaces space instances. By wrapping the space instance it adds SLA capabilities avliable with each processing unit. One of the common SLA is to ensure that primary instances will not be running on the same machine as the backup instances etc. It also determines deployment topology (partitioned, replicated), as well as scaling policy, etc. The data grid includes another instance, not shown in the above diagram, called the Mirror Service. The Mirror Service is responsible for making sure that all updates made on the Data Grid will be passed reliably to the underlying database.


  • Load Balancer Agent – The Load Balancer Agent is responsible for listening to web-containers availability and add those instances to the Load Balancer list when a new container is added, or remove it when it has been removed. The Load Balancer Agent is currently configured to work with the Apache Load Balancer but can be easily set up to work with any external Load Balancer.

How it works:

The following section provides a high-level description of how all the above components work together to provide high performance and scaling.


  • Deployment - The deployment of the application is done through the GigaSpaces processing-unit deployment command. Assigning specific SLA as part of the deployment lets the GSM know how we wish to distribute the web instances over the network. For example, one could specify in the SLA that there would be only one instance per machine and define the minimum number of instances that need to be running. If needed, one can add specific system requirements such as JVM version, OS-Type, etc. to the SLA . The deployment command points to to a specific web application archive (WAR). The WAR file needs to include a configuration attribute in its META-INF configuration that will instruct the deployer tool to use GlassFish v3 Prelude as the web container for this specific web application. Upon deployment the GlassFish-processing-unit will be started on the available GSC containers that matches the SLA definitions. The GSC will assign specific port to that container instance. .  When GlassFish starts it will load the war file automatically and start serving http requests to that instance of the web application.

  • Connecting to the Load Balancer - Auto scaling -The load balancer agent is assigned with each instance of the GSM. It listens for the availability of new web containers and ensures that the available containers will join the load balancer by continuously updating the load-balancer configuration whenever such change happens. This happens automatically through the GigaSpaces discovery protocol and does not require any human intervention.

  • Handling failure - Self healing - If one of the web containers fails, the GSM will automatically detect that and start and new web container on one of the available GSC containers if one exists. If there is not enough resources available, the GSM will wait till such a resource will become available. In cloud environments, the GSM will initiate a new machine instance in such an event by calling the proper service on the cloud infrastructure.

  • Session replication - HttpSession can be automatically backed up by the GigaSpaces In Memroy Data Grid (IMDG) . In this case user applications do not need to change their code. When user data is stored in the HttpSession,  that data gets stored into the underlying IMDG. When the http request is completed that data is flushed into the shared data-grid servers.

  • Scaling the database tier - Beyond session state caching, the web application can get a a reference to the GigaSpaces IMDG and use it to store data in-memory in order to reduce contention on the database. GigaSpaces data grid automatically synchronizes updates with the database. To enable maximum performance, the update to the database is done in most cases asynchronously (Write-Behind). A built-in hibernate plug-in handles the mapping between the in-memory data and the relational data model. You can read more on how this model handles failure as well as consistency, aggregated queries here.

Tuesday Mar 03, 2009

GlassFish 3.0 on Amazon cloud

Here is how you can run the demo of GlassFish 3.0 on Amazon cloud. 

Where should I start? The best way to get started is to run a demo applicaiton and see for yourself how this integration works. To make it even simpler we offer the demo on our new cloud offering. This will enable you to expereince how a full production ready environment which include full clustering, dynamic scaling, full high avliability and Session replication works in one click. To run the demo on the cloud follow the follwoing steps:

1. Download the GlassFish web scaling deployment file from here to your local machine.

2. Get the mycloud page and get your free access code – this will enable you to get access to the cloud.



3. Select the stock-demo-2.3.0-gf-sun.xml then hit the Deploy button (you first need to save the attached file in one your local machine) – The system will start provisioning the web application on the cloud. This will include a machine for managing the cloud, a machine for the load-balancer and machines for the web and data-grid containers. After approximately 3 minutes the application will be deployed completely. At this point you should see “running” link on the load-balancer machine. Click on this link to open your web-client application.


4. Test auto-scaling – click multiple times on the “running” link to open more clients. This will enable us to increase the load (request/sec) on the system. As soon as the request/sec will grow beyond a certain threshold you’ll see new machines being provisioned. After two minutes approximately the machine will be running and a new web-container will be auto-deployed into that machine. This new web-container will be linked automatically with the load-balancer and the load-balancer in return will spread the load to this new machine as well. This will reduce the load on each of the servers.

5. Test self-healing – you can now kill one of the machines and see how your web client is behaving. You should see that even though the machine was killed the client was hardly effected and system scaled itself down automatically.

Seeing what’s going on behind the scene:

All this may seam to be like a magic for you. If you want to access the machines and watch the web containers, the data-grid instances and the machines as well as the real time statistics you can click on the Manage button button. This will open-up a management console that is running on the cloud through the web. With this tool you can view all the components of our systems. You can even query the data using SQL browser and view the data as it enters the system. In addition to that you can choose to add more services, relocate services through a simple click of a mouse or drag and drop action.


For more information regarding using Glassfish see


OpenSolaris on Amazon EC2

Yesterday Sun hosted the Israeli Association of Grid Technology (IGT) event "Amazon AWS Hands-on workshop" at the Sun office in Herzeliya. During the event Simone Brunozzi, Amazon Web Services Evangelist, demonstrated Amazon's Ec2 and S3 using the AWS console. There were 40 attendees from a wide breadth of technology firms.

For more information regarding using OpenSolaris on amazon EC2 see

Wednesday Jan 28, 2009

xVM public API

There are two ways of accessing the public API, the simplest is by writing a Java client using JMX, or alternatively for non-Java client programs using Web Services for Management (WS-MAN, JSR-262).


This example demonstrates the usage of the direct access to the read-only copy of the domain model. This example can be run against both Sun xVM Server or Sun xVM Ops Center.

This example performs the following functions:

  • Configures the connection

  • Performs security settings

  • Opens the connection (locally or remotely)

  • Queries the domain model for all the OperatingSystems objects and displays the value of their Hosts name

  • Closes the connection

\* Copyright 2006 Sun Microsystems, Inc. All rights reserved.
\* SUN PROPRIETARY/CONFIDENTIAL. Use is subject to license terms.
import com.sun.hss.type.os.OperatingSystem;
import com.sun.hss.type.server.Server;
import com.sun.hss.type.virtserver.VirtServerContainer;
import com.sun.hss.type.virtserver.VirtServerOperatingSystem;
import com.sun.hss.type.xvmserver.XVMApplianceDetails;
import com.sun.hss.type.xvmserver.XVMServer;
import java.util.HashMap;
import java.util.Map;




public class ServerClient {

static private JMXConnector connector = null;
static private MBeanServerConnection mbsc = null;
static private String hostname = null;

\* Simple SocketFactory that uses a trust manager that ALWAYS
\* accept the certificate provided by the server we try to connect to.
\* This is unsafe and should NOT be used for production code.
private static SSLSocketFactory getSocketFactory() {
X509TrustManager tm = new AnyServerX509TrustManager();
TrustManager[] tms = {tm};

try {
SSLContext sslc = SSLContext.getInstance("TLSv1");
sslc.init(null, tms, null);

SSLSocketFactory factory = sslc.getSocketFactory();

return factory;
} catch (Exception ex) {

return null;

\* Small trustmanager that ALWAYS accepts the certificate provided
\* by the server we try to connect to.
\* This is unsafe and should NOT be used for production code.
public static class AnyServerX509TrustManager implements X509TrustManager {
// Documented in X509TrustManager

public X509Certificate[] getAcceptedIssuers() {
// since client authentication is not supported by this
// trust manager, there's no certicate authority trusted
// for authenticating peers
return new X509Certificate[0];

// Documented in X509TrustManager
public void checkClientTrusted(X509Certificate[] certs, String authType)
throws CertificateException {
// this trust manager is dedicated to server authentication
throw new CertificateException("not supported");

// Documented in X509TrustManager
public void checkServerTrusted(X509Certificate[] certs, String authType)
throws CertificateException {
// any certificate sent by the server is automatically accepted

\* Create a WSMAN connection using the given credentials
\* @param host
\* @param userName
\* @param userPass
\* @return MBeanServerConnection
private static void setupConnection(String host, String user, String pass)
throws Exception {
try {
int port = 443;

String urlPath = "/wsman/ea/jmxws";

Map env = new HashMap();

// credentials for basic authentication with the server
String[] creds = new String[2];
creds[0] = user;
creds[1] = pass;

env.put(JMXConnector.CREDENTIALS, creds);

// provide a specific socket factory to avoid the need to setup
// a truststore

// Create JMX Agent URL over https
JMXServiceURL url = new JMXServiceURL("ws-secure", host, port, urlPath);

// System.out.println("WSMAN client opening a connection with url " + url.toString());

// Connect the JMXConnector
connector = JMXConnectorFactory.connect(url, env);

// Get the MBeanServerConnection
mbsc = connector.getMBeanServerConnection();

} catch (Exception ex) {
System.out.println("Got an exception while trying to open a WSMAN connection : " + ex.toString());
throw ex;

public static void main(String[] args) {
if ((args.length == 0) || (args.length > 3)) {
System.err.println("Usage: user password [target]");

String userName = args[0];
String userPass = args[1];

hostname = "localhost";

if (args.length == 3) {
hostname = args[2];

try {
// Open the WSMAN connection and get the MBeanServerConnection
setupConnection(hostname, userName, userPass);

// get details on these xVM servers

} catch (Exception ex) {

System.out.println("WSMAN client error : " + ex.toString());

} finally {
// close connection if necessary
if (connector != null) {
try {
} catch (Exception dc) {


private static void serverService() throws Exception {

try {
XVMServerServiceMXBean xssmxb = ServerClientServices.getXVMServerService(mbsc, false);

// get the list of xVM servers
ObjectName[] servers = xssmxb.getXVMApplianceDetailsObjectNames(null, null);

if ((servers == null) || (servers.length == 0)) {
System.out.println("No xVM server detected on " + hostname);

GuestServiceMXBean guestmxb = ServerClientServices.getGuestService(mbsc, false);

// get details on these xVM servers
for (int i = 0; i < servers.length; i++) {

XVMApplianceDetails details = xssmxb.getXVMApplianceDetails(servers[i]);

if (details != null) {
OperatingSystem os = details.getOperatingsystem();

Server svr = details.getServer();

VirtServerContainer vsc = details.getVirtservercontainer();

XVMServer xsvr = details.getXvmserver();

if (xsvr != null) {
System.out.println("xVM Server name = " + xsvr.getApplianceName());


// get guests on this xVM server
ObjectName[] guests = guestmxb.getGuestObjectNames(servers[i], null, null);

Java.util.Map<java.lang.String, Java.util.Set<java.lang.string>> map = null;
GuestDetails guestDetails[] = guestmxb.getGuestDetails(guests, map);

String s = guests[0].getCanonicalName();

if ((guests == null) || (guests.length == 0)) {
System.out.println("No guest on this xVM server");
if (guestDetails != null) {
for (int k = 0; k < guestDetails.length; k++) {
VirtServerOperatingSystem virt = guestDetails[k].getVirtServerOperatingSystem();
System.out.println("guest hostname is " + k + ": " + virt.getHostname());

} catch (Exception ex) {
System.err.println("Got Exception while testing the xVM server service : " + ex.toString());
throw ex;


public class ServerClientServices
    \* getCacheManagerService
    \* @param mbsc
    \* @return CacheManagerMXBean
    \* @throws java.lang.Exception
   public static CacheManagerMXBean getCacheManagerService(MBeanServerConnection mbsc, boolean verbose)
          throws Exception
       CacheManagerMXBean cmmxb = null;

       try {
           // build the objectname to access the service
           ObjectName cmson = new ObjectName(CacheManagerMXBean.DOMAIN +
                                             ":type=" + CacheManagerMXBean.TYPE);

           // verify that the service is currently deployed
           if (!mbsc.isRegistered(cmson)) {
               System.out.println("Cache Manager service is not registered : aborting");
               throw new Exception("MXBean for Cache Manager service not registered");

           // create a proxy to access the service
           cmmxb = JMX.newMXBeanProxy(mbsc,

           if (verbose) {
               System.out.println("Proxy for Cache Manager service : OK");
           return cmmxb;

       } catch (Exception ex) {
           System.err.println("Got Exception while creating the Cache Manager service proxy : " + ex.toString());
           throw ex;

Monday Jan 12, 2009

Solaris iSCSI Server

This document describes how to build Iscsi server based on Solaris platform on sun X4500 server.

On the target (server)

The server has six controllers each with eight disks and I have built the storage pool to spread I/O evenly and to enable me to build 8 RAID-Z stripes of equal length.

zpool create -f  tank \\
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \\
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \\
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \\
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \\
raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \\
raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \\
raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \\
raidz c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \\
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0

After the pool is created, the zfs utility can be used to create 50GB ZFS volume.

zfs create -V 50g tank/iscsivol000

Enable the Iscsi service

svcadm enable iscsitgt

Verify that the service is enabled.

svcs –a | grep iscsitgt

To view the list of commands, iscsitadm can be run without any options:


Usage: iscsitadm -?,-V,--help Usage: iscsitadm create [-?]  [-?] [ Usage: iscsitadm list [-?]  [-?] [ Usage: iscsitadm modify [-?]  [-?] [ Usage: iscsitadm delete [-?]  [-?] [ Usage: iscsitadm show [-?] 

[-?] [ For more information, please see iscsitadm(1M)

To begin using the iSCSI target, a base directory needs to be created.

This directory is used to persistently store the target and initiator configuration that is added through the iscsitadm utility.

iscsitadm modify admin -d /etc/iscsi

Once the volumes are created, they need to be exported to an initiator

iscsitadm create target -b /dev/zvol/rdsk/tank/iscsivol000 target-label

Once the targets are created, iscsitadm's "list" command and "target" subcommand can be used to display the targets and their properties:

iscsitadm list target -v

On the initiator (client)

Install iscsi client from 

Wednesday Jan 07, 2009

IGT2008 video and presentation

IGT2008 video and presentations are now available online:

Wednesday Dec 03, 2008

IGT2008 - The World Summit of Cloud Computing

Yesterday  I want to the  IGT 2008  event  In addition to exhibiting at the event, I  delivered a opening demo presentation and a hands-on

xVM server workshop



This blog covers cloud computing, big data and virtualization technologies


« July 2016