Tuesday Jun 30, 2009

Quick Blog Links

Just wanted to push out some quick links to other blog write-ups involving the Sun HPC Software, Linux Edition. Earlier this week (or was it late last week) we put up the blog entry for using the next version of the stack and Virtual Box (if you missed it, here it is). And here are some other write-ups about using the upcoming 2.0 release you shouldn't miss: As time goes on, we'll post other write-ups that we come across.

Sunday Jun 28, 2009

Coming soon ... 2.0

Just wanted to get a note out there that 2.0 is on it's way. It's had its delays, but we're close to final release on it. Included will be the following major changes:
  • Support for RHEL 5.3, CentOS 5.3 and now SLES 10
  • OFED 1.3.1
  • Lustre 1.8.0.1
When we finally get the release out the door, we'll send out more information. In the meantime, you can take a sneak peek at the documentation or check out how to build a 2.0 cluster using virtual box here.

Building a Sun HPC Virtual Cluster

(Today's blog post comes from our intern Liu Lei.)

Overview

Introduction

Sun HPC Software, Linux Edition (“Sun HPC Software”) is an integrated open-source software solution for Linux-based HPC clusters running on Sun hardware. It provides a framework of software components to simplify the process of deploying and managing large-scale Linux HPC clusters.

For those who are interested in trying out the stack but have no access to a lot of hardware, Sun xVM VirtualBox could turn your laptop into a HPC development platform. VirtualBox 2.2.2 supports 64bit virtualization on top of both 32 bit and 64 bit host OS.

System Requirements

Here are the hardware specs of my Lenovo T61 laptop and a list of software I used.

Hardware Specification

Software Configuration

Intel(R) Core(TM)2 Duo CPU T7300
2G RAM
150G harddisk

Ubuntu 9.04
Sun xVM VirtualBox 2.2.2 amd64 edition
CentOS 5.3
SunHPC Stack 2.0

Please Note:
If you want to go through the entire tutorial and configure the lustre file system as instructed, at least 60 GB of free disk space is required which includes the space for downloading the required software (CentOS 5.3, VirtualBox 2.2.2 and Sun HPC Stack) and for accommodating six Virtual Machines. Also, 2 GB RAM is the minimum requirement for running six Virtual Machines smoothly simultaneously. 4G RAM is recommended configuration as some of readers have reported some failures with the barely enough memory.


Installing The Cluster

Obtaining the software

You will need access to the following software or software repositories.

Installation Overview

The installation procedure described in this guide installs the Sun HPC Software on a cluster configured similar to that shown in Figure 1. This example cluster contains:

  • Head Node - As part of the Sun HPC Software installation process, the Cobbler and oneSIS provisioning tools are installed on the head node. These tools are used for the provisioning of diskful and diskless cluster nodes. The head node must be connected to the cluster-wide provisioning network.
  • Client Nodes - All nodes provisioned from the head node are referred to as clients of the head node. A client node may be provisioned in either a diskful or diskless configuration. Each client node must be connected to the cluster-wide provisioning network and are provisioned using the Cobbler/oneSIS-based provisioning system.

The Cobbler provisioning tool facilitates provisioning (via DHCP/PXE) of diskful or diskless node configurations. For diskless nodes, Cobbler uses the oneSIS system administration tool to provide NFS-mounted root filesystems for each node class, such as a Lustre server, compute node, or login node.The following figure shows example cluster configuration using an Ethernet or Infiniband network as the compute network.

Create and configure the virtual machine for our Head Node

  1. Create a virtual machine “mgmt1” in VirtualBox. I assigned it 512MB ram (1G ram is recommended.) and created a 40G virtual disk image for it. When choosing to use the dynamic disk option, the actual disk space occupied by the virtual machine will dynamically adjust within the confines of 40GB according to the use of the disk, so it doesn't necessarily matter if you assign a bigger initial disk size.
  2. Mount CentOS-5.3-x86_64-bin-DVD.iso image
  3. Make sure the "VT-x/AMD-V" feature is selected
  4. Enable two network adapters. Adapter1 is attached NAT and Adapter2 is attached to the internal network noted as "intnet". In order for the Virtual Machine to get the right MAC address when booting, the Adapter Type of both Adapter 1 and Adapter 2 should be chosen as PCnet-FAST III (Am79C973).

  5. Start the virtual machine and install CentOS 5.3 on it. The installation procedure should be pretty straightforward using the instalation wizard. I basically click next all the way except assigning host name as "mgmt1".

  6. During the post installation configuration after reboot, I disabled both firewall and selinux since I am not interested in testing these security features.
  7. Restart the virtual machine. By default network adapter 2, "eth1", is likely deactivated.
    [root@mgmt1 ~]# ifconfig -a
    eth0      Link encap:Ethernet  HWaddr 08:00:27:EB:B8:42  
              inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
              inet6 addr: fe80::a00:27ff:feeb:b842/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:1189 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1294 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:1358518 (1.2 MiB)  TX bytes:85153 (83.1 KiB)
              Interrupt:177 Base address:0xc020
    
    eth1      Link encap:Ethernet  HWaddr 08:00:27:9B:A4:22  
              BROADCAST MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
              Interrupt:193 Base address:0xc240
    
    We need to assign a static IP address for eth1 in order to active it. We can use the Network Configuration tool in Gnome to do this.

    Select "eth1" and click "Edit". Then type in static IP address as below. Tick the option of "Activate devices when computer starts".

    Sometimes, it might be a bit tricky to find "OK" button because of resolution of screen. The solution is to hold down the ALT key and click/drag the window to a new position.

    Click "Activate" to activate eth1.

    Now we should have Internet access both through eth0 and an internal network on eth1.

    [root@RHEL-mgmt1 ~]# wget www.sun.com
    --12:40:24--  http://www.sun.com/
    Resolving www.sun.com... 72.5.124.61
    Connecting to www.sun.com|72.5.124.61|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: unspecified [text/html]
    Saving to: `index.html.1'
    
        [       <=>                              ] 30,441      10.2K/s   in 2.9s  
    
    12:40:34 (10.2 KB/s) - `index.html.1' saved [30441]
    
  8. Enable ssh access to mgmt1

    I personally find it very convenient to be able to connect to the virtual box through SSH. In particular, I could copy & paste some long commands in the ssh terminal before I install Guest Additions.

    We have a Guest Machine (mgmt1) with a running ssh server which accepts connections on the TCP port 22. Our goal is to make any packet arriving at a given TCP port (i.e. 2222) of the Host machine to be forwarded to the TCP port 22 of the Guest Machine. Fortunately, there is Virtualbox command which permits us to do it almost instantly.

    Open a terminal on the host machine and enter the following commands:

    larry@LiuLei:~$ VBoxManage setextradata "mgmt1" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP
    VirtualBox Command Line Management Interface Version 2.2.2
    (C) 2005-2009 Sun Microsystems, Inc.
    All rights reserved.
    
    larry@LiuLei:~$ VBoxManage setextradata "mgmt1" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
    VirtualBox Command Line Management Interface Version 2.2.2
    (C) 2005-2009 Sun Microsystems, Inc.
    All rights reserved.
    
    larry@LiuLei:~$ VBoxManage setextradata "mgmt1" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
    VirtualBox Command Line Management Interface Version 2.2.2
    (C) 2005-2009 Sun Microsystems, Inc.
    All rights reserved.
    

    The HostPort must be greater than or equal to 1024 since listening on ports 0-1023 needs root permissions (and Virtualbox usually doesn't). Instead, GuestPort has to be equivalent to the one on which the virtualized ssh is listening.

    Once you have typed the above commands, you need to close the Guest Machine(a reboot won't be sufficient), restart it and then connect via ssh with following commands:

    larry@LiuLei:~$ ssh -p 2222 root@localhost
    root@localhost's password:
    Last login: Wed May  8  08:59:05 2009
    [root@mgmt1 ~]#
    
    larry@LiuLei:~$ sftp -o port=2222 root@localhost
    Connecting to localhost...
    root@localhost's password:
    sftp>
    

    The virtual machine, "mgmt1", is ready to be the head node of virtual clusters.

    Additional Information:
    If you want an ssh connection to more than one Guest Machine, you may assign different numbers to the HostPort (for example 2222 and 2301), GuestPort could remain the same(22), then close and start (a reboot won't be sufficient) all the Guest Machines (e.g. mgmt1 and mgmt2).

    larry@LiuLei:~$ ssh -p 2222 root@localhost                                                                                        
    root@localhost's password:                                                                                                                                  
    Last login: Fri May  8 14:01:37 2009 from 10.0.2.2                                                                                               
    [root@mgmt1 ~]# 
    
    larry@LiuLei:~$ ssh -p 2301 root@localhost
    root@localhost's password:
    Last login: Fri May 8 14:05:20 2009
    [root@mgmt2 ~]#
    

    If you encounter the following problem, remove \*/.ssh/known_hosts and use ssh to log in again.

    larry@LiuLei:~$ ssh -p 2222 root@localhost
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    83:86:f1:ac:b9:ca:f5:53:03:49:92:97:8a:49:5a:f3.
    Please contact your system administrator.
    Add correct host key in /home/larry/.ssh/known_hosts to get rid of this message.
    Offending key in /home/larry/.ssh/known_hosts:1
    RSA host key for [localhost]:2222 has changed and you have requested strict checking.
    Host key verification failed.
    

Install Sun HPC software stack on the head node

  1. Create a software repository for CentOS 5.3
    1. Use sftp to upload CentOS-5.3-x86_64-bin-DVD.iso into mgmt1 (For this example, it is placed in /root/.)
    2. Add the following line to /etc/fstab:
      /root/centos-5.3-x86_64-bin-DVD.iso /mnt/centos5.3 iso9660 ro,loop 0 0
      
    3. Mount the CentOS 5.3 ISO image to the mount point /mnt/centos5.3.
      [root@mgmt1 ~]# mkdir -p /mnt/centos5.3
      [root@mgmt1 ~]# mount -a
      
    4. Create a configuration file for the RHEL repository:
      # cat /etc/yum.repos.d/centos.repo
      [centos]
      name=CentOS DVD
      baseurl=file:///mnt/centos5.3
      enabled=1
      gpgcheck=0
      
  2. Check if dialog is installed by entering:
    [root@mgmt1 ~]# rpm -qa | grep dialog
    
  3. If dialog is not installed, use the appropriate command below to install it.
    [root@mgmt1 ~]# yum install dialog
    
  4. Use sftp to put sun-hpc-linux-rhel-2.0.iso to mgmt1 (For this example, it is placed in /root/.) and mount it to /media/sun_hpc_linux
    1. Add the following line to /etc/fstab:
      /root/sun-hpc-linux-rhel-2.0.iso /media/sun_hpc_linux iso9660 ro,loop 0 0
      
    2. Mount the file containing the ISO image by entering
      [root@mgmt1 ~]#  mkdir -p /media/sun_hpc_linux
      [root@mgmt1 ~]#  mount -a
      
  5. Install Sun HPC Software stack
    [root@mgmt1 ~]# rpm -ivh /media/sun_hpc_linux/SunHPC/sunhpc-release.rpm
    Preparing...                ########################################### [100%]
       1:sunhpc-release         ########################################### [100%]
    
  6. Run the sunhpc_install script by entering:
    [root@mgmt1 ~]# sunhpc_install
    
    The following screens indicates that the installation is successful

Note: we received some feedbacks that sunhpc_install might fail strangely on the virtual machine with small memory. Increasing the RAM size, e.g. to 1G, and then re-runing "sunhpc_install" would get through.

Prepare the Head Node to Provision the Cluster

To set up a Cobbler profile on a head node running CentOS 5.3, follow the procedure below. The examples assume that the headnode has two network interfaces: eth0 connects to the Internet or public network; eth1 connects to the rest of the HPC cluster nodes and serves as a DHCP interface. The Cobbler profile is used to provision the compute cluster.

  1. Check that both the CentOS 5.3 ISO image and the Sun HPC Software ISO image are mounted on the head node. The output of the mount command should contain the snippets below:
    [root@mgmt1 ~]# mount
    --snip--  
    /root/CentOS-5.3-x86_64-bin-DVD.iso on /mnt/centos5.3 type iso9660 (rw,loop=/dev/loop0)
    /root/sun-hpc-linux-rhel-2.0.iso on /media/sun_hpc_linux type iso9660 (rw,loop=/ dev/loop1)
    --snip--
    
  2. Complete the options below to provision diskful and diskless clients,as appropriate for your HPC cluster.
    • If diskful client is to be provisioned:
      1. Enter the following command where the head node connects to the client nodes on Ethernet interface eth1:
        [root@mgmt1 ~]# sunhpc_setup --profile=centos5.3 --distro-image=/mnt/centos5.3 \\
           --sunhpc-repo=/media/sun_hpc_linux --netif=eth1 --bootdisk=hda
           Initializing Cobbler configuration... Done
           Disabling the iptables... Done
           Restarting dhcpd/cobblerd/httpd... Done
           Generating SSH key in /root/.sshDone
           Copying /mnt/centos5.3 to /var/www/cobbler/ks_mirror/centos5.3... Done
           Created 'sunhpc_base_centos5.3' repo and copying... Done
           Created 'sunhpc_extras_centos5.3' repo and copying... Done
           Created 'sunhpc_lustre_centos5.3' repo and copying... Done
           Created 'sunhpc_updates_centos5.3' repo and copying... Done
           Creating distro 'centos5.3' in cobbler... Done
           Creating profile 'centos5.3' in cobbler... Done
           Creating profile 'centos5.3-lustre' in cobbler... Done
        
      2. Generate a Cobbler profile report to check that the Cobbler profiles centos5.3 and centos5.3-lustre have been created. centos5.3 is the profile for diskful Lustre client nodes. centos5.3-lustre is the profile for diskful Lustre server nodes, which will run on a Lustre patched kernel. Please refer to the Lustre operation manual for more information.
        [root@mgmt1 ~]# cobbler profile list
           centos5.3
           centos5.3-lustre
        
    • If diskless client is to be provisioned:
      1. Enter the following command where the head node connects to the client nodes on Ethernet interface eth1:
        [root@mgmt1 ~]# sunhpc_setup --profile=centos5.3-onesis --diskless \\
           --netif=eth1 --onesis-exclude=/root --bootdisk=hda
           Initializing Cobbler configuration... Done
           Disabling the iptables... Done
           Restarting dhcpd/cobblerd/httpd... Done
           Copying / to /var/lib/oneSIS/image/centos5.3-onesis... Done
           Creating initrd... Done
           Applying OneSIS configuration... Done
           Updated /etc/exports and restarting NFS... Done
           Copying /var/lib/oneSIS/image/centos5.3-onesis to /var/lib/oneSIS/image
           /centos5.3-onesis-lustre ... Done.
           Un-specializing centos5.3-onesis-lustre ... Done.
           Removing SunHPC Lustre Client group from centos5.3-onesis-lustre ... Done.
           Installing perl-TimeDate from distro... Done.
           Installing compat-libcom_err from distro... Done.
           Installing uuidd from distro... Done.
           Installing libnet from distro... Done.
           Installing python-xml from distro... Done.
           Upgrading e2fsprogs for ldiskfs support... Done.
           Removing base kernel from centos5.3-onesis-lustre ... Done.
           Installing SunHPC Lustre Server group to centos5.3-onesis-lustre ... Done.
           Creating oneSIS initrd for centos5.3-onesis-lustre ... Done.
           Converting centos5.3-onesis-lustre to oneSIS rootfs image ... Done.
           Adding /var/lib/oneSIS/image/centos5.3-onesis-lustre to /etc/exports ... Done.
           Now (re)starting NFS... Done.
           Creating distro 'centos5.3-onesis' in cobbler... Done
           Creating distro 'centos5.3-onesis-lustre' in cobbler... Done
           Creating profile 'centos5.3-onesis' in cobbler... Done
           Creating profile 'centos5.3-onesis-lustre' in cobbler... Done
        

        This command creates two oneSIS system images, one for diskless Lustre client nodes and one for diskless Lustre server nodes, in the directory /var/lib/oneSIS/image on the head node.

        [root@mgmt1 ~]# ls /var/lib/oneSIS/image
           centos5.3-onesis  centos5.3-onesis-lustre
        
      2. Generate a Cobbler profile report to check that the Cobbler profiles centos5.3-onesis and centos5.3-onesis-lustre have been created. centos5.3-onesis is the profile for diskless Lustre client nodes. centos5.3-lustre-onesis is the profile for diskless Lustre server nodes, which will run on a Lustre patched kernel.
        [root@mgmt1 ~]# cobbler profile list
           centosl5.3-onesis
           centos5.3-onesis-lustre
        
  3. Cfengine is used to distribute configuration files to the client nodes before they are provisioned. To set up Cfengine, run the following on the head node:
      [root@mgmt1 ~]# setup_cfengine
      parsing setting
      fix cfengine settings in gtdb
      update configuration files
      set up cfengine
      setup cfengine done
    

Note:
For more information about provisioning the Head Node, please refer to pages 20-22 of the Sun HPC Software, Linux Edition manual.

Provision the Client Nodes

The Sun HPC Software manages the client node provisioning process using the Sun HPC Software, Linux Management Database (gtdb) provided with the Sun HPC Software. To provision the client nodes in the compute cluster, you will first populate gtdb using the Sun HPC Software, Linux Management Tools (gtt). You will then generate configuration files for provisioning tools such as Cobbler, which will be used to provision each node in the cluster from the head node.

The Sun HPC Software provides support for provisioning four types of client nodes using a Cobbler service on the head node:

  • Diskful Lustre client mode - Runs an unpatched Red Hat Linux or SLES SP 2 kernel and a number of software packages on a local disk, such as an MPI program, SGE execution host program, Lustre client software, and InfiniBand software. A diskful Lustre client node generally serves as a compute node in the HPC cluster and has access to a Lustre file system.
  • Diskful Lustre server node - Runs the kernel patched with Lustre server software. Although other software packages can be installed on a diskful Lustre server node, the major role of this node type is to serve as a metadata server (MDS) or object storage server (OSS) node in a Lustre file system.
  • Diskless Lustre client node - Runs on a oneSIS image on the head node through an NFS mount. A diskless Lustre client node uses the same kernel as the head node. It generally serves as a compute node in the HPC cluster and has access to a Lustre file system.
  • Diskless Lustre Server Node - Runs on a oneSIS image on the headnode through an NFS mount. A diskless Lustre server node uses a Lustre patched kernel. It typically serves as a metadata server (MDS) or object storage server (OSS) node in a Lustre file system.

The tables below shows the clients we use to provision a lustre file system.

Node ID Configuration Role Provisioning Interface
MAC Address
Provisioning Interface
IP Address
mgmt1 Diskful Management   10.1.80.1
dfmds01 Diskful Lustre MDS 08:00:27:5E:C2:77 10.1.80.4
dflcn001 Diskful Luster client/compute node 08:00:27:60:12:CB 10.1.80.8
dloss01 Diskless Luster OSS 08:00:27:AF:AB:A8 10.1.80.12
dloss02 Diskless Luster OSS 08:00:27:C9:6C:C7 10.1.80.13
dllcn001 Diskless Luster client/compute node 08:00:27:74:65:51 10.1.80.15

Notes:
For more information about provisioning client nodes, please refer to pages 28-31 of the Sun HPC Software, Linux Edition manual.

Populating the Sun HPC Software Management Database (gtdb)

The Sun HPC Software, Management Database (gtdb) is provided to relieve the HPC cluster administrator of repetitive and error-prone service configuration management. After populating the database with information about the HPC cluster (e.g. hostnames, network addresses, etc.), the administrator can then generate configuration files for supported services (e.g. ConMan, PowerMan, SLURM) and system databases (e.g. /etc/hosts, /etc/genders, etc.).

Follow the procedure below to population the Sun HPC Software Management Database (gtdb) and generate configuration files for provisioning.

  1. Before provisioning clients, you need to create five virtual machines for client nodes (including dflcn001, dllcn001, dloss01, dloss02, dfmds01).
    • Enable only Adapter1 which is attached to the internal network "intnet". The Adapter Type of Adapter 1should be chosen as PCnet-FAST III (Am79C973).
    • Assign 20GB dynamic free disk space to each client node.
    • Assign 128MB memory size to each client node
  2. Before using gtt to add clients, you need to get the mac address of the five client nodes to be added, we take dflcn001 for example

  3. Use the gtt command to add clients in gtdb.
    [root@mgmt1 ~]# gtt host --add --name dflcn001 --network "hwaddr=08:00:27:60:12:CB,ipaddr=10.1.80.8,device=eth0,bootnet=true" --attribute "profile=centos5.3" --attribute static
    Host added successfully: dflcn001
    Network added successfully: eth0
    Attribute added successfully to compute1: profile
    Attribute added successfully to compute1: static
    
    [root@mgmt1 ~]# gtt host --add --name dllcn001 --network "hwaddr=08:00:27:74:65:51,ipaddr=10.1.80.15,device=eth0,bootnet=true" --attribute "profile=centos5.3-onesis" --attribute static
    Host added successfully: dllcn001
    Network added successfully: eth0
    Attribute added successfully to compute1: profile
    Attribute added successfully to compute1: static
    
    [root@mgmt1 ~]# gtt host --add --name dfmds01 --network "hwaddr=08:00:27:5E:C2:77,ipaddr=10.1.80.4,device=eth0,bootnet=true" --attribute "mds" --attribute "profile=centos5.3-lustre" --attribute static
    Host added successfully: dfmds01
    Network added successfully: eth0
    Attribute added successfully to compute1: profile
    Attribute added successfully to compute1: static
    
    [root@mgmt1 ~]# gtt host --add --name dloss01 --network "hwaddr=08:00:27:C9:6C:C7,ipaddr=10.1.80.12,device=eth0,bootnet=true" --attribute "oss" --attribute "profile=centos5.3-onesis-lustre" --attribute static
    Host added successfully: dloss01
    Network added successfully: eth0
    Attribute added successfully to compute1: profile
    Attribute added successfully to compute1: static
    
    
    [root@mgmt1 ~]# gtt host --add --name dloss02 --network "hwaddr=08:00:27:AF:AB:A8,ipaddr=10.1.80.13,device=eth0,bootnet=true" --attribute "oss" --attribute "profile=centos5.3-onesis-lustre" --attribute static
    Host added successfully: dloss02
    Network added successfully: eth0
    Attribute added successfully to compute1: profile
    Attribute added successfully to compute1: static
    

    The option --attribute static enables clients to be provided with a static IP address after provisioning. Without this attribute, the clients will be provided with a dynamic IP address allocated by the DHCP server running on the head node.

  4. Generate a set of configuration files from gtdb
    [root@mgmt1 ~]# gtt config --update all
    Updating config: cfservd
    /var/lib/sunhpc/cfengine/var/cfengine/inputs/cfservd.conf: Wrote 35 lines
    Updating config: cfupdate
    /var/lib/sunhpc/cfengine/var/cfengine/inputs/update.conf: Wrote 70 lines
    Updating config: cobbler
    /var/lib/sunhpc/cfengine/tmp/cobbler.csv: Wrote 2 lines
    Updating config: conman
    /var/lib/sunhpc/cfengine/etc/conman.conf: Wrote 179 lines
    Updating config: genders
    \^[[B/var/lib/sunhpc/cfengine/etc/genders: Wrote 3 lines
    Updating config: hosts
    /var/lib/sunhpc/cfengine/etc/hosts: Wrote 5 lines
    Updating config: ntp
    /var/lib/sunhpc/cfengine/etc/ntp.conf: Wrote 24 lines
    Updating config: powerman
    /var/lib/sunhpc/cfengine/etc/powerman/powerman.conf: Wrote 5 lines
    Updating config: slurm
    /var/lib/sunhpc/cfengine/etc/slurm/slurm.conf: Wrote 37 lines
    
  5. Update the local configuration files on the head node by running cfagent, which will copy files from /var/lib/sunhpc/cfengine into the appropriate places.
    [root@mgmt1 ~]# cfagent -q
    
  6. Generate data for Cobbler from the Cobbler configuration file cobbler.csv
    [root@mgmt1 ~]# populate_cobbler_system /var/lib/sunhpc/cfengine/tmp/cobbler.csv
    Internet Systems Consortium DHCP Server V3.0.5-RedHat
    Copyright 2004-2006 Internet Systems Consortium.
    All rights reserved.
    For info, please visit http://www.isc.org/sw/dhcp/
    Shutting down dhcpd: [ OK ]
    Starting dhcpd: [ OK ]
    
  7. Use cobbler list to get a summary of clients (referred to as the "system" in Cobbler) and client profiles.
    [root@mgmt1 ~]# cobbler list
    distro centos5.3
       profile centos5.3
          system dflcn001
       profile centos5.3-lustre
          system dfmds01
    distro centos5.3-onesis
       profile centos5.3-onesis
          system dllcn001
    distro centos5.3-onesis-lustre
       profile centos5.3-onesis-lustre
          system dloss01
          system dloss02
    repo sunhpc_base_centos5.3
    repo sunhpc_extras_centos5.3
    repo sunhpc_lustre_centos5.3
    repo sunhpc_updates_centos5.3
    
  8. Run the following command to remove console=ttyS0,9660 so that the provision process would show up on the VM windows of the client nodes.
    [root@mgmt1 ~]# cobbler profile edit --name=centos5.3 --kopts="selinux=0" 
    [root@mgmt1 ~]# cobbler profile edit --name=centos5.3-lustre --kopts="selinux=0" 
    [root@mgmt1 ~]# cobbler profile edit --name=centos5.3-onesis --kopts="selinux=0" 
    [root@mgmt1 ~]# cobbler profile edit --name=centos5.3-onesis-lustre --kopts="selinux=0" 
    

The client nodes are now ready to boot.

Notes:
If you ever reboot the headnode during provision process, you will need to manually start httpd and nfs before continuing provision.

[root@mgmt1 ~]# /etc/init.d/httpd status
httpd is stopped
[root@mgmt1 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
                                                           [  OK  ]
[root@mgmt1 ~]# /etc/init.d/httpd status
httpd (pid 4319 4318 4317 4316 4315 4314 4313 4312 4310) is running...

[root@mgmt1 ~]# /etc/init.d/nfs status
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
[root@mgmt1 ~]# /etc/init.d/nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ] 

Provision Diskful Client Node(dflcn001)

Boot dflcn001 from network and the installation started.

Notes:
You need to keep an eye on the installation process and after the installation finished when dflcn001 starts to reboot, you must manually shut down it and boot from local disk, otherwise, the installation process will repeat once more.

Once the client provisioning completes, run the following commands to test password-less ssh access to the provisioned clients and add them to .ssh/known_hosts

[root@mgmt1 ~]# PDSH_SSH_ARGS_APPEND="-o StrictHostKeyChecking=no" pdsh -g profile hostname

Warning messages similar to the following are displayed to indicate the clients have been added to the known_hosts list:

dflcn001: Warning: Permanently added 'dflcn001,10.1.80.8' (RSA) to the list of known hosts.

[root@mgmt1 ~]# ssh dflcn001
[root@dflcn001 ~]

Provision Diskless Client Node(dllcn001)

Boot dllcn001 from network and after dllcn001 boots up, just press "Enter" if the following screen pops up

[root@mgmt1 ~]# PDSH_SSH_ARGS_APPEND="-o StrictHostKeyChecking=no" pdsh -g profile hostname
--snip--
dllcn001: Warning: Permanently added 'dllcn001,10.1.80.15' (RSA) to the list of known hosts.
--snip--
[root@mgmt1 ~]# ssh dllcn001
[root@dllcn001 ~]#

Notes:
During the process of provisioning diskless nodes(including dllcn001, dloss01 and dloss02), you may encounter the following problem. You need to wait for a while and there is a chance that the problem will go away by itself (nfs: server xx OK). However, if "not responding" problem persists, you have no choice but to reboot the node until the problem is gone. Although it sounds awkward and is sometime time-consuming, there seems to be no better solution.

Provision Diskful Server Node(dfmds01)

Boot dfmds01 from network and after the installation completes manually reboots it from local disk as you did when provisioning dflcn001.

After dfmds01 boots up,you should be able to ssh it from the headnode

[root@mgmt1 ~]# PDSH_SSH_ARGS_APPEND="-o StrictHostKeyChecking=no" pdsh -g profile hostname
--snip--
dfmds01: Warning: Permanently added 'dfmds01,10.1.80.4' (RSA) to the list of known hosts.
--snip--
[root@mgmt1 ~]# ssh dfmds01
[root@dfmds01 ~]#

Provision Diskless Server Nodes(dloss01 dloss02)

Boot dloss01 from network. The following screen may or may not pop up shortly after you start dloss01, if it does pop up you need to choose centos5.3-onesis-lustre in time

After dloss01 boots up you can ssh it from headnode

[root@mgmt1 ~]# PDSH_SSH_ARGS_APPEND="-o StrictHostKeyChecking=no" pdsh -g profile hostname
--snip--
dloss01: Warning: Permanently added 'dloss01,10.1.80.12' (RSA) to the list of known hosts.
--snip--
[root@mgmt1 ~]# ssh dloss01
bash-3.2# PS1=['\\u@\\H:\\w]# '
[root@dloss01:~]#

Then use the same method to provision dloss02

After provisioning the client nodes, run a simple pdsh command to check if all the provisioned clients are accessible. A typical result is:

[root@mgmt1 ~]# pdsh -g profile uptime
dllcn001: 14:25:36  up  2:45, 0 users, load average: 0.02, 0.02, 0.00
dloss01: 13:25:52  up  2:44, 0 users, load average: 0.09, 0.03, 0.00
dflcn001: 14:25:59  up  2:45, 0 users, load average: 0.01, 0.01, 0.00
dfmds01: 19:25:39  up  1:49, 2 users, load average: 0.00, 0.00, 0.08
dloss02: 18:25:49  up  1:55, 0 users, load average: 0.00, 0.00, 0.04

Lustre Configuration

Once the client nodes have been provisioned, they can serve as Lustre server nodes or Lustre client nodes, regardless of whether they are diskful or diskless.

Configure MDS node

  1. Edit modprobe.conf file to specify Lustre network option, add the following red line.
    [root@dfmds01 ~]# vi /etc/modprobe.conf
    --snip--
    alias eth0 pcnet32
    alias eth1 pcnet32
    alias scsi_hostadapter ata_piix
    alias snd-card-0 snd-intel8x0
    options snd-card-0 index=0
    options snd-intel8x0 index=0
    options lnet networks=tcp
    --snip--
    
  2. Since there is only one disk on dfmds01 which is already occupied, we create a tmp file as mdt device
    [root@dfmds01 ~]# mkfs.lustre --mdt --mgs --device-size=100000 /tmp/mdt
    
       Permanent disk data:
    Target:     lustre-MDTffff
    Index:      unassigned
    Lustre FS:  lustre
    Mount type: ldiskfs
    Flags:      0x75
                  (MDT MGS needs_index first_time update )
    Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
    Parameters: mdt.group_upcall=/usr/sbin/l_getgroups
    
    checking for existing Lustre data: not found
    2 6 18
    formatting backing filesystem ldiskfs on /dev/loop0
        target name  lustre-MDTffff
        4k blocks     25000
        options        -i 4096 -I 512 -q -O dir_index,uninit_groups -F
    mkfs_cmd = mkfs.ext2 -j -b 4096 -L lustre-MDTffff  -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/loop0 25000
    Writing CONFIGS/mountdata
    
  3. Mount mdt device
    [root@dfmds01 ~]# mkdir /mnt/mdt
    [root@dfmds01 ~]# mount -t lustre -o loop /tmp/mdt /mnt/mdt
    
  4. Verify if mds is functioning
    [root@dfmds01 ~]# lctl list_nids
    10.1.80.4@tcp
    [root@dfmds01 ~]#  df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/hda3              19G  1.7G   16G  10% /
    /dev/hda1             122M   12M  104M  11% /boot
    tmpfs                 250M     0  250M   0% /dev/shm
    /dev/loop0             86M  4.2M   77M   6% /mnt/mdt
    

Configure OSS nodes

  1. Edit modprobe.conf file to specify Lustre network option
    [root@mgmt1 ~]# vi /var/lib/oneSIS/image/centos5.3-onesis-lustre/etc/modprobe.conf
    --snip--
    alias eth0 pcnet32
    alias eth1 pcnet32
    alias scsi_hostadapter ata_piix
    alias snd-card-0 snd-intel8x0
    options snd-card-0 index=0
    options snd-intel8x0 index=0
    options lnet networks=tcp
    --snip--
    
  2. Create OST
    [root@dloss01:~]# fdisk -l
    Disk /dev/hda: 21.4 GB, 21474836480 bytes
    16 heads, 63 sectors/track, 41610 cylinders
    Units = cylinders of 1008 \* 512 = 516096 bytes
    Disk /dev/hda doesn't contain a valid partition table
    
    [root@dloss01:~]# mkfs.lustre --ost --mgsnode=10.1.80.4@tcp /dev/hda
    Permanent disk data:
    Target:     lustre-OSTffff
    Index:      unassigned
    Lustre FS:  lustre
    Mount type: ldiskfs
    Flags:      0x72
                  (OST needs_index first_time update )
    Persistent mount opts: errors=remount-ro,extents,mballoc
    Parameters: mgsnode=10.1.80.4@tcp
    
    checking for existing Lustre data: not found
    device size = 20480MB
    2 6 18
    formatting backing filesystem ldiskfs on /dev/hda
        target name  lustre-OSTffff
        4k blocks     0
        options        -J size=400 -i 16384 -I 256 -q -O dir_index,uninit_groups -F
    mkfs_cmd = mkfs.ext2 -j -b 4096 -L lustre-OSTffff  -J size=400 -i 16384 -I 256 -q -O 
    
    dir_index,uninit_groups -F /dev/hda
    Writing CONFIGS/mountdata
    
  3. Create the mount point on the headnode
    [root@mgmt1 ~]# mkdir -p /var/lib/oneSIS/image/centos5.3-onesis-lustre/mnt/ost0
    
  4. Mount OST on the diskless server node
    [root@dloss01:~]# mount -t lustre /dev/hda /mnt/ost0
    

    Notes:
    You have to make sure that mds(dfmds01) is running when creating OST. If you ever reboot dfmds01, you have to remount mdt device in order for mds to function.

    [root@dloss01:~]# mount -t lustre /dev/hda /mnt/ost0
    mount.lustre: mount /dev/hda at /mnt/ost0 failed: Input/output error
    Is the MGS running?
    [root@dfmds01 ~]# mount -t lustre -o loop /tmp/mdt /mnt/mdt
    [root@dfmds01 ~]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/hda3              19G  1.7G   16G  10% /
    /dev/hda1             122M   12M  104M  11% /boot
    tmpfs                 124M     0  124M   0% /dev/shm
    /dev/loop0             86M  4.2M   77M   6% /mnt/mdt
    [root@dloss01:~]# mount -t lustre /dev/hda /mnt/ost0
    [root@dloss01:~]#
    
  5. Use dloss02 to create another OSS using the steps above. (mount point is .../ost1)

Mount Lustre file system on the client node

  • Using diskless luster client
    1. Edit modprobe.conf on the head node
      [root@mgmt1 ~]# vi /var/lib/oneSIS/image/centos5.3-onesis/etc/modprobe.conf
      --snip--
      alias eth0 pcnet32
      alias eth1 pcnet32
      alias scsi_hostadapter ata_piix
      alias snd-card-0 snd-intel8x0
      options snd-card-0 index=0
      options snd-intel8x0 index=0
      options lnet networks=tcp
      --snip--
      
    2. Create mount point on the head node
      [root@mgmt1 ~]# mkdir /var/lib/oneSIS/image/centos5.3-onesis/mnt/lustre
      
    3. Mount lustre file system on the client
      [root@dllcn001 ~]# mount -t lustre 10.1.80.4@tcp:/lustre /mnt/lustre
      [root@dllcn001 ~]# df -h
      Filesystem            Size  Used Avail Use% Mounted on
      rootfs                 38G   19G   17G  53% /
      10.1.80.1:/var/lib/oneSIS/image/centos5.3-onesis
                             38G   19G   17G  53% /
      /dev/ram              100M  1.5M   99M   2% /ram
      none                   61M   88K   61M   1% /dev
      tmpfs                  61M     0   61M   0% /dev/shm
      10.1.80.4@tcp:/lustre
                             40G  890M   37G   3% /mnt/lustre
      

      You can see that he size of Lustre file system is 40G which is equal to the sum of harddisks(/dev/hda) on dloss01 and dloss02.

  • Using diskful luster client
    1. Verify if kernel-ib and kernel are consistent.
      [root@dflcn001 ~]# uname -a
      Linux dflcn001 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 
      GNU/Linux
      [root@dflcn001 ~]# rpm -qa|grep kernel-ib
      kernel-ib-1.3.1-2.6.18_128.el5
      
    2. Edit modprobe.conf to specify Lustre network option
      [root@dflcn001 ~]# vi /etc/modprobe.conf 
      +options lnet networks=tcp
      
    3. Mount Lustre File System
      [root@dflcn001 ~]# mkdir /mnt/lustre
      [root@dflcn001 ~]# mount -t lustre 10.1.80.4@tcp:/lustre /mnt/lustre
      [root@dflcn001 ~]# df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/hda3              19G  1.6G   16G  10% /
      /dev/hda1             122M   12M  104M  10% /boot
      tmpfs                  61M     0   61M   0% /dev/shm
      10.1.80.4@tcp:/lustre
                             20G  445M   19G   3% /mnt/lustre
      

      Now we should able to copy files into /mnt/lustre and access these files from both Lustre clients.

      [root@dllcn001:~]# cp ~/sun-hpc-linux-rhel-2.0.iso /mnt/lustre/
      [root@dflcn001 ~]# ls /mnt/lustre/
      sun-hpc-linux-rhel-2.0.iso
      [root@dflcn001 ~]# df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/hda3              19G  1.6G   16G  10% /
      /dev/hda1             122M   12M  104M  10% /boot
      tmpfs                  61M     0   61M   0% /dev/shm
      10.1.80.4@tcp:/lustre
                             40G  1.5G   36G   4% /mnt/lustre
      

Conclusion

This tutorial illustrated the procedure of utilizing Sun HPC software stack, Linux Edition 2.0 and Sun xVM Virtual Box to set up a High Performance Computing development platform on a moderate laptop. We demonstrated a complete open source solution for HPC software developers who has no access to computer clusters and proprietary softwares. All of software components in Sun HPC software stack, Linux Edition are exactly as same as those running on world leading supercomputers. We hope more and more HPC software developers could take advantage of such a "mobile" development platform in their projects. We will also be eager to receive feedbacks and wishlists for the future Sun HPC software stack releases. Download the Sun HPC software stack and Join the community today!


References

Friday Jan 09, 2009

Sun HPC Software, Linux Edition 1.2 Now Available!

The Sun HPC Software Stack, Linux Edition team (aka Giraffe) is pleased to announce the release of version 1.2 of our HPC Software Stack.

This release builds on the features included in version 1.1 of the stack, including:

  • Support for Red Hat 5.2 and CentOS 5.2
  • OFED 1.3.1
  • Lustre 1.6.6
  • Optional Kickstart-based installation

Improvements included in version 1.2 of the stack include:

  • Boot Over Infiniband (BoIB) support
  • New versions of oneSIS (2.0.1) and SLURM (1.3.10)
  • Inclusion of Mellanox Firmware Tools (version 2.5.0)
  • Optional perfctr-patched kernel for compute nodes

Sun HPC Software, Linux Edition is an integrated open-source software solution for Linux-based HPC clusters running on Sun hardware. It provides a framework of software components to simplify the process of deploying and managing large-scale Linux HPC clusters.

For more information or to download this new version, please visit the Sun HPC Software, Linux Edition product page at the following URL: http://www.sun.com/software/products/hpcsoftware/

Thanks to everyone in the community who helped us by providing early feedback and testing.

Friday Nov 14, 2008

Sun HPC Software, Linux Edition 1.1

The announcement went out today on our mailing lists (are you on the mailing list yet? no? you should click here and sign up) of the 1.1 release of the Sun HPC Software stack.

First, you might be asking, "What happened to 1.0?". Well, that was released and got some really good feedback, it just seems that the one who writes these blog entries never quite got around to actually writing one for 1.0. Please ridicule him as much as possible, since he really deserves it. But, all of this is in the past now; on to 1.1.

As some might know, 1.0 was based on CentOS 5.1. The 1.1 release brings us up to a CentOS 5.2 base, but also includes the option to deploy on a RedHat Enterprise Linux 5.2 OS as well. We had heard a few requests for this inclusion, and here we go. Gone also is the "build-an-iso" option seen from 1.0 (if you don't know what that is, it's ok, we've all moved on). We now provide a new installation option based on Kickstart that allows a system administrator to perform a semi- or fully-automated installation of an RPM-based Linux system.

Other important updates in this version include OFED 1.3.1 as well as Lustre 1.6.6. We've also included cobbler to aid in the creation of diskfull clusters as well as diskless clusters.

So, please visit the Sun HPC Software, Linux Edition product page to read more about the release, and start using it.

Giraffe Members at SC08

Well, SC08 is just around the corner (quite literally, since I've taken so long to write this). Just wanted to mention that members from your friendly, neighborhood Sun HPC Software, Linux Edition team will be there showing off our work. So, stop by the Sun Booth and ask lots of questions, or taunt us, whichever seems to be the preferred method of communication.

In addition, we will also be giving talks at the Sun HPC Consortium prior to SC08, so there's all sorts of ways to find and consort with members of Giraffe.

Wednesday Jun 18, 2008

Giraffe pre-release of the Sun HPC Software, Linux Edition

After a quick sprint from February to June, and some mild bumps in between, Giraffe came to the Sun HPC Consortium and International SuperComputing '08 and announced our first public pre-release of Sun's HPC Software, Linux Edition. While we're still working out some of the kinks and bettering the stack in general, we wanted to be able to finally get the stack into the hands of others.

This pre-release offering comes in the form of a "build your own ISO" package, which includes tools that can be run on a local RPM-based system (RedHat, CentOS, Scientific Linux, etc.) that go out and fetch all the relevant pieces and build an ISO that can then be burned to DVD. Utilizing this shiny, new DVD, one can install a management node and build their very own linux-based HPC cluster.

Please, don't forget to also sign up to our community mailing list, that way we can all get involved in the conversation and really help each other figure out how to better not only the stack, but perhaps linux and HPC as well.

About

A forum to allow those of us in Giraffe to help the Linux community in our own little ways.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Feeds