Saturday Jan 16, 2010

MySQL on EBS with OpenSolaris based AMI

Amazon Elastic Block Store (EBS) is the persistent storage which can be attached to the AMI instance. Data can be safely placed and accessed independent of the life of the instance. There are plenty of benefits using EBS for MySQL. For detailed explanation click here.



Following steps describes how to use MySQL on EBS volume with OpenSolaris based AMI.

1. Create a new EBS Volume.

$ ec2-create-volume --size 10 --availability-zone us-east-1a
VOLUME    vol-688d4c01    10        us-east-1a    creating    2010-01-13T21:54:57+0000

$ ec2-describe-volumes vol-688d4c01
VOLUME    vol-688d4c01    10        us-east-1a    available    2010-01-13T21:54:57+0000

Click here for ec2-create-volume API help.

2. Create a new MySQL 5.1 AMI Instance based on OpenSolaris.

$ ec2-run-instances ami-752ccb1c -k my-keypair
RESERVATION    r-2486124c    987654321123    default
INSTANCE    i-9e6858f6    ami-752ccb1c            pending    my-keypair    0        m1.small    2010-01-17T00:01:24+0000    us-east-1a    aki-6552b60c    ari-6452b60d        monitoring-disabled

Click here for more details on this AMI.

3. Attach EBS volume to this AMI instance

$ ec2-attach-volume -i i-9e6858f6 -d 2 vol-688d4c01
ATTACHMENT    vol-688d4c01    i-9e6858f6    2    attaching    2010-01-13T22:02:51+0000

In the above command -d option specifies the device number. In OpenSolaris devices are named from 0 to 23. Device number 0 and 1 are used for ephemeral storage. Therefore, we can use the devices starting from 2 until 23.

$ ec2-describe-volumes vol-688d4c01
VOLUME    vol-688d4c01    10        us-east-1a    in-use    2010-01-13T21:54:57+0000
ATTACHMENT    vol-688d4c01    i-9e6858f6    2    attached    2010-01-13T22:02:51+0000

Click here for ec2-attach-volume API help.
Click here for ec2-describe-volumes API help.

4. Create a ZFS storage pool on the previously attached EBS volume.

Once the instance is up and running, login into the instance using ssh with your key-pair.
To verify new EBS volume attached, execute the format command and press Control + C to come out.

root@domU-12-31-39-03-21-E4:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c3d0 <DEFAULT cyl 1274 alt 0 hd 255 sec 63>
          /xpvd/xdf@0
       1. c3d1 <DEFAULT cyl 19464 alt 0 hd 255 sec 63>
          /xpvd/xdf@1
       2. c3d2 <DEFAULT cyl 1305 alt 0 hd 255 sec 63>
          /xpvd/xdf@2
Specify disk (enter its number): \^C
root@domU-12-31-39-03-21-E4:~#

Create a ZFS storage pool on the EBS volume.

root@domU-12-31-39-03-21-E4:~# zpool create ebsmysql c3d2

root@domU-12-31-39-03-21-E4:~# zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
ebsmysql  9.94G  73.5K  9.94G     0%  ONLINE  -
mnt            149G  73.5K   149G     0%  ONLINE  -
rpool         9.75G  3.50G  6.25G    35%  ONLINE  -
root@domU-12-31-39-03-21-E4:~#

root@domU-12-31-39-03-21-E4:~# zpool status ebsmysql
  pool: ebsmysql
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ebsmysql    ONLINE       0     0     0
          c3d2         ONLINE       0     0     0

errors: No known data errors
root@domU-12-31-39-03-21-E4:~#

Move MySQL on ZFS storage pool based on EBS

Make sure MySQL service is disabled.

root@domU-12-31-39-03-21-E4:~# svcs svc:/application/mysql:default
STATE          STIME    FMRI
disabled       21:57:48 svc:/application/mysql:default
root@domU-12-31-39-03-21-E4:~#

If MySQL service is running make sure to disable it using  svcadm disable svc:/application/mysql:default

Move MySQL installation directory to /ebsmysql

root@domU-12-31-39-03-21-E4:~# mv /usr/local/mysql/ /ebsmysql/
root@domU-12-31-39-03-21-E4:~#

Create a soft link to /ebsmysql/mysql/ under /usr/local with the name 'mysql'

root@domU-12-31-39-03-21-E4:~# cd /usr/local/
root@domU-12-31-39-03-21-E4:/usr/local# ln -s /ebsmysql/mysql/ mysql

root@domU-12-31-39-03-21-E4:/usr/local# ls -la
total 5
drwxr-xr-x  2 root root  3 2010-01-13 22:08 .
drwxr-xr-x 32 root sys  46 2009-02-19 23:20 ..
lrwxrwxrwx  1 root root 16 2010-01-13 22:08 mysql -> /ebsmysql/mysql/
root@domU-12-31-39-03-21-E4:/usr/local#

Enable MySQL service

root@domU-12-31-39-03-21-E4:~# svcadm enable svc:/application/mysql:default
root@domU-12-31-39-03-21-E4:~# svcs svc:/application/mysql:default
STATE          STIME    FMRI
online         22:09:43 svc:/application/mysql:default
root@domU-12-31-39-03-21-E4:~#

Test MySQL

root@domU-12-31-39-03-21-E4:~# /ebsmysql/mysql/bin/mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \\g.
Your MySQL connection id is 3
Server version: 5.1.25-rc Source distribution

Type 'help;' or '\\h' for help. Type '\\c' to clear the buffer.

mysql> show databases;
+--------------------+
| Database             |
+--------------------+
| information_schema |
| mysql                        |
| sbtest                         |
| test                            |
+--------------------+
4 rows in set (0.01 sec)

mysql>

5. Detach EBS volume from AMI instance

root@domU-12-31-39-03-21-E4:~# svcadm disable svc:/application/mysql:default

root@domU-12-31-39-03-21-E4:~# svcs svc:/application/mysql:default
STATE          STIME    FMRI
disabled        3:09:07 svc:/application/mysql:default
root@domU-12-31-39-0A-0E-64:~#

root@domU-12-31-39-03-21-E4:~# zpool export ebsmysql
root@domU-12-31-39-03-21-E4:~# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
mnt     149G  73.5K   149G     0%  ONLINE  -
rpool  9.75G  3.22G  6.53G    33%  ONLINE  -
root@domU-12-31-39-03-21-E4:~#

$ ec2-detach-volume vol-688d4c01
ATTACHMENT    vol-688d4c01   i-9e6858f6    2    detaching    2010-01-17T02:58:33+0000

$ ec2-describe-volumes vol-688d4c01
VOLUME    vol-688d4c01    10        us-east-1a    available    2010-01-17T02:50:34+0000
divyen-patels-macbook:ec2 divyenpatel$

6. Re-attach EBS volume to the AMI instance

$ ec2-attach-volume -i i-9e6858f6 -d 2 vol-688d4c01
ATTACHMENT    vol-5f0ccd36    i-9e6858f6    2    attaching    2010-01-17T03:18:25+0000

$ ec2-describe-volumes vol-688d4c01
VOLUME    vol-688d4c01    10        us-east-1a    in-use    2010-01-17T02:50:34+0000
ATTACHMENT    vol-688d4c01    i-9e6858f6    2    attached    2010-01-17T03:18:25+0000

Import the pool

root@domU-12-31-39-03-21-E4:~# zpool import ebsmysql

root@domU-12-31-39-03-21-E4:~# zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
ebsmysql  9.94G   285M  9.66G     2%  ONLINE  -
mnt        149G  73.5K   149G     0%  ONLINE  -
rpool     9.75G  3.22G  6.53G    33%  ONLINE  -
root@domU-12-31-39-03-21-E4:~#

7. Add another EBS Volume to pool

$ ec2-create-volume --size 10 --availability-zone us-east-1a
VOLUME    vol-990ccdf0    10        us-east-1a    creating    2010-01-17T03:24:24+0000

$ ec2-describe-volumes vol-990ccdf0
VOLUME    vol-990ccdf0    10        us-east-1a    available    2010-01-17T03:24:24+0000

$ ec2-attach-volume -i i-9e6858f6 -d 3 vol-990ccdf0
ATTACHMENT    vol-990ccdf0    i-9e6858f6    3    attaching    2010-01-17T03:25:02+0000

$ ec2-describe-volumes vol-990ccdf0
VOLUME    vol-990ccdf0    10        us-east-1a    in-use    2010-01-17T03:24:24+0000
ATTACHMENT    vol-990ccdf0    i-9e6858f6    3    attached    2010-01-17T03:25:02+0000

root@domU-12-31-39-03-21-E4:~# zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
ebsmysql  9.94G   285M  9.66G     2%  ONLINE  -
mnt        149G  73.5K   149G     0%  ONLINE  -
rpool     9.75G  3.22G  6.53G    33%  ONLINE  -

root@domU-12-31-39-03-21-E4:~# zpool add ebsmysql c3d3

root@domU-12-31-39-03-21-E4:~# zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
ebsmysql  19.9G   285M  19.6G     1%  ONLINE  -
mnt        149G  73.5K   149G     0%  ONLINE  -
rpool     9.75G  3.22G  6.53G    33%  ONLINE  -
root@domU-12-31-39-03-21-E4:~#

root@domU-12-31-39-03-21-E4:~# zpool status ebsmysql
  pool: ebsmysql
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ebsmysql    ONLINE       0     0     0
          c3d2      ONLINE          0     0     0
          c3d3      ONLINE          0     0     0

errors: No known data errors
root@domU-12-31-39-03-21-E4:~#

Important Links:

MySQL on ZFS:
http://dev.mysql.com/tech-resources/articles/mysql-zfs.html

EBS With OpenSolaris AMI:
http://blogs.sun.com/ec2/entry/ebs_is_supported_on_opensolaris

Amazon EBS, OpenSolaris, and ZFS Guide to Getting Started
http://blogs.sun.com/ec2/resource/OpenSolaris-ZFS-EBS-v1.3.pdf


Sunday Dec 27, 2009

Connecting MySQL server on Amazon EC2 instance using ssh tunnel

SSH tunnel basically does the port forwarding via secure ssh service.

Let's start with launching a latest hardened AMP stack AMI (ami-e0b05389) instance.
Once the instnace is up and running, open the terminal client and connect to the instance using ssh.

Edit the /etc/ssh/sshd_config file and set AllowTcpForwarding to yes
AllowTcpForwarding yes

Restart ssh service
svcadm restart svc:/network/ssh:default

Comment out skip-networking from /etc/mysql/my.cnf and save the file
#skip-networking

Restart MySQL service
svcadm restart svc:/application/database/mysql:version_51

exit the ssh connection and recreate ssh connection using following command.
ssh -i <keypair> -L 3307:localhost:3306 root@ec2-xxx-xxx-xxx-xx.compute-1.amazonaws.com

Keep the terminal opened, and start MySQL Query Browser / MySQL administrator from your local machine. and  connect to the MySQL server on remote EC2 instance via the port 3307, as shown in the following sanpshots.


When done close the ssh connection and it should close the ssh tunnel.

Friday Oct 30, 2009

Creating paravirtualized CentOS 5.3 DomU on OpenSolaris 2009.06 Dom0

Setup OpenSolaris Dom0 

First Step is to setup xVM Hypervisor on the OpenSolaris 2009.06 machine.

Follow the steps mentioned at following link and setup Dom0 OpneSolaris 2009.06.
http://hub.opensolaris.org/bin/view/Community+Group+xen/2008_11_dom0

Create a RAW disk image for the DomU installation

mkdir /centos53-pv/
cd /centos53-pv/
dd if=/dev/zero of=CentOS-5.3-x86_64.img bs=1k seek=10000k count=1

Enable xvm/virtd service on Dom0

svcadm enable svc:/system/xvm/virtd:default

Begin the Paravirtualized CentOS DomU installation with the virt-install command

virt-install -n centos53_x86_64 \\
-r 1024 \\
-f /centos53-pv/CentOS-5.3-x86_64.img \\
-p --nographics \\
-l http://mirrors.kernel.org/centos/5.3/os/x86_64/

In the above command -n is for the name of the domu, -r is for the memory, -f is for specifying location of the raw disk on which installation will be performed, -p is for specifying paravirtualized mode and -l is for the location containing an installable distribution image.

Follow the regular CentOS installation steps and reboot the DomU when prompted.

Following snapshots walk you through CentOS DomU installation.

After reboot enable ssh login for root user on DomU

uncomment the line "PermitRootLogin yes" from "/etc/ssh/sshd_config" file and save it.

Extract kernel and ramdisk from the running DomU image

Copy "/boot/vmlinuz-2.6.18-128.el5xen" and "/boot/initrd-2.6.18-128.el5xen.img" files to Dom0 machine at "/centos53-pv/".

scp  /boot/vmlinuz-2.6.18-128.el5xen root@dom0-ip:/centos53-pv/
scp  /boot/initrd-2.6.18-128.el5xen.img root@dom0-ip:/centos53-pv/ 

Halt the DomU.

Shutdown domU issuing halt command inside domU and remove a domain from Xend domain management using following command.

# xm delete centos53_x86_64

Create a domain configuration file as given below. 

Create "/centos53-pv/domu.py" file as given below

# cat domu_64.py 
name = "centos53_x86_64"
memory = "2058"
kernel = "/centos53-pv/vmlinuz-2.6.18-128.el5xen"
ramdisk = "/centos53-pv/initrd-2.6.18-128.el5xen.img"
disk = ['file:/centos53-pv/CentOS-5.3-x86_64.img,xvda,w']
vif = ['']

Relaunch the DomU by passing extracted kernel and ramdisk images using py script. 

# xm create domu.py 
Using config file "./domu.py".
Started domain centos53_x86_64

Access domU console

 # xm console centos53_x86_64

Cent OS 5.3 Paravirtualized DomU VNC Session on OpenSolaris 2009.06 Dom0

Tuesday Jun 16, 2009

How to Setup iPhone AWS manager on OpenSolaris EC2 Instance


Project Link: http://code.google.com/p/iphone-aws-manager/

I was looking for the free iPhone application to manage my ec2 account and came across this article: "Mobile Monitoring and Administration of Amazon Web Services: Developing an iPhone-Optimized Web Application".

This article not only describes in details each steps on how to begin with developing iPhone application to manage AWS but also provides source code that you easily download from Google Code Repository and setup in-house.

In this blog entry I will walk you through steps on setting up this application on our popular OpenSolaris AMP Stack AMI.

Following Steps are also valid for the setup of "iPhone AWS Manager" on local machine with OpenSolaris and AMP Stack.

Lets begin with launching new OpenSolaris AMP Stack AMI Instance.
ec2-run-instances ami-c7cf28ae -k my-keypair

Get the public DNS address of an Instance and login into with your keypair
ec2-describe-instances i-xxxxxxxx
ssh -i <path_to_your_keypair> root@ec2-xx-xxx-xx-xxx.compute-1.amazonaws.com

Install SVN Package from the Repository
pkg refresh
pkg install SUNWsvn

Checkout iPhone AWS Manager code form the Google Code Repository
cd /var/apache2/2.2/htdocs/

svn co http://iphone-aws-manager.googlecode.com/svn/trunk/ aws-iphone
A aws-iphone/amis.php
A aws-iphone/class.s3.php
A aws-iphone/reboot.php
A aws-iphone/deleteobj.php
A aws-iphone/contents.php
A aws-iphone/getbuckets.php
A aws-iphone/delete.php
A aws-iphone/create.php
A aws-iphone/run.php
A aws-iphone/index.php
A aws-iphone/about.php
A aws-iphone/size.php
A aws-iphone/class.ec2.php
A aws-iphone/lib
A aws-iphone/lib/iui
A aws-iphone/lib/iui/backButton.png
A aws-iphone/lib/iui/grayButton.png
A aws-iphone/lib/iui/loading.gif
A aws-iphone/lib/iui/pinstripes.png
A aws-iphone/lib/iui/toolbar.png
A aws-iphone/lib/iui/iuix.js
A aws-iphone/lib/iui/toolButton.png
A aws-iphone/lib/iui/whiteButton.png
A aws-iphone/lib/iui/iui.js
A aws-iphone/lib/iui/toggleOn.png
A aws-iphone/lib/iui/thumb.png
A aws-iphone/lib/iui/toggle.png
A aws-iphone/lib/iui/iuix.css
A aws-iphone/lib/iui/blueButton.png
A aws-iphone/lib/iui/cancel.png
A aws-iphone/lib/iui/selection.png
A aws-iphone/lib/iui/listArrow.png
A aws-iphone/lib/iui/listGroup.png
A aws-iphone/lib/iui/iui.css
A aws-iphone/lib/iui/listArrowSel.png
A aws-iphone/iPhone AWS Source Code.zip
A aws-iphone/instances.php
A aws-iphone/terminate.php
Checked out revision 3.


Set the Document Root in the httpd.conf file
vi /etc/apache2/2.2/httpd.conf

Set the DocumentRoot as given below
DocumentRoot "/var/apache2/2.2/htdocs/aws-iphone"

edit /etc/php/5.2/php.ini and set following variables to On
register_globals = On
register_long_arrays = On

Create "key.php" file at "/var/apache2/2.2/htdocs/aws-iphone/keys.php" with following code

<?php 
define('my_access_key','XXXXXXXXXXXXXXXXXXXX'); define('my_secret_key','XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX');
?>

Edit "/var/apache2/2.2/htdocs/aws-iphone/index.php" file and replace php code block at the top of the page with following code.
<?php
        // start the session
        session_start();

        // include the EC2 and S3 libraries
        require_once("class.ec2.php");
        require_once("class.s3.php");
        include_once("keys.php");
        // create session variables for AWS keys
        session_register("AMAZON_KEY");
        session_register("AMAZON_PRIVATE_KEY");

        $HTTP_SESSION_VARS["AMAZON_KEY"]=my_access_key;
        $HTTP_SESSION_VARS["AMAZON_PRIVATE_KEY"]=my_secret_key;
?>

change your keypair in "/var/apache2/2.2/htdocs/aws-iphone/class.ec2.php" file
look for the function definition and replace your keyName

function runInstances($imageId, $min = 1, $max = 1, $keyName = "divyen-personal-aws-key")


Restart the Apache Server
svcadm disable svc:/network/http:apache22
svcadm enable svc:/network/http:apache22

Open Safari Browser in your iPhone and type in EC2 Instance DNS Address





Create a Shortcut icon so that you don't need to type in URL every time





Snapshots of the basic operations supported on this application are described as given below

Describe AMIs - Both Public and Private





Create Instance, Reboot Instance, Terminate Instance





Create Bucket, Delete Bucket





Thursday Jun 04, 2009

How to start VNC session on OpenSolaris 2009.06 EC2 AMI

Following instructions describes steps to access GUI on OpenSolaris 2009.06 EC2 AMI Instance using VNC Viewer.

Enable following desktop related services

svcadm enable svc:/application/graphical-login/gdm:default
svcadm enable svc:/application/desktop-cache/icon-cache:default
svcadm enable svc:/application/desktop-cache/desktop-mime-cache:default
svcadm enable svc:/application/desktop-cache/mime-types-cache:default
svcadm enable svc:/application/desktop-cache/input-method-cache:default
svcadm enable svc:/application/desktop-cache/gconf-cache:default
svcadm enable svc:/application/desktop-cache/pixbuf-loaders-installer:default

Set xvm_vnc property to true for x11-server SMF service.

svccfg -s x11-server setprop options/xvm_vnc = boolean: true

Set the root password, you may need it to get back to your session in case if screen saver starts.

root@domU-12-31-39-02-5C-D1:~# passwd
passwd: Changing password for root
New Password:
Re-enter new Password:
passwd: password successfully changed for root
root@domU-12-31-39-02-5C-D1:~#

Setup VNC server

Create .vnc directory under /root if it does not exist
mkdir /root/.vnc/

Create /root/.vnc/xstartup file with following lines.
#!/bin/sh
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
gnome-terminal -geometry 100x60 -title "$VNCDESKTOP Desktop" &
/usr/bin/gnome-session
Note: if xstartup file is already exist, replace it with above lines to start gnome terminal session for VNC.

Provide execute permission on xtartup file.
chmod +x /root/.vnc/xstartup

Start the vncserver and set the password.

root@domU-12-31-39-02-5C-D1:~# vncserver

You will require a password to access your desktops.

Password:
Verify:
xauth: creating new authority file /root/.Xauthority

New 'domU-12-31-39-02-5C-D1:1 (root)' desktop is domU-12-31-39-02-5C-D1:1

Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/domU-12-31-39-02-5C-D1:1.log

root@domU-12-31-39-02-5C-D1:~#

Authorize VNC Port
Here we are done with configuring EC2 instance for GUI Remote Access.
Finally we need to authorize the corresponding VNC port on users machine where VNC client is installed.

divyen-patels-macbook:~ divyen$ ec2-authorize default -p 5801
GROUP default
PERMISSION default ALLOWS tcp 5801 5801 FROM CIDR 0.0.0.0/0
divyen-patels-macbook:~ divyen$


Screen Shots
VNC client connection configuration
Snapshot1:http://blogs.sun.com/divyen/resource/snap_1_ec2_opensolaris_2009_06_vnc.jpg
Use public DNS address associated with EC2 instance as the host name.

VNC client connected to OpenSolaris 2009.06 EC2 Instance.
Snapshot2:http://blogs.sun.com/divyen/resource/snap_2_ec2_opensolaris_2009_06_vnc.jpg

For more details on OpenSolaris AMIs visit http://blogs.sun.com/ec2/
About

Divyen Patel

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today