ec2ify:me 5 optional steps to parametrize instance

Based on my previous Linux on Amazon AWS experience I define 5 optional Solaris ec2ify steps:

This entry is part of 'OpenSolaris on Amazon EC2' workshop

1) info get instance runtime data, show them login and generate shell include script
2) userdata get instace parameterizing with userdata in form of file
3) updater make adhock needed small fixes
4) mount format and mount data drives, create additional swaps
5) syncer synchronize running instance with data on S3

Plus wrapping script and service ec2ify:me

1. Sample of indent output /etc/motd showed on login with generated instance data:

-------100------ Welcome to OpenSolaris on Amazon AWS (EC2,S3) Prototypes --------100-------

Image: JeOS RNSP - OpenSolaris JeOS with Reduced Network Server Profile os2008.05 snv_86

This AMI image is Prof-of-Concept and is presented for evaluation, demostration or learning
purposes only. This image can contain software and/or represent procedures in BETA quality.

JeOS [ Ver: os2008.05, Pkgs: 112, Sizes: [Pkgs 482MB, Inst 785MB, ZIP 358MB, EC2 363MB] ]
Shells [ sh bash screen mc ] Editors [ joe vi vim ] Langs [ Perl Ruby Java tcl ]
Archvers [ tar gtar gzip bzip2 zip ] SysTools [ traceroute ntpdate cron top logadm sfMANs ]
ec2ify [ /opt/ec2ify: s3tool ec2ident(SMF) ec2ify(SMF) [info mount params updater syncer] ]
Apps [ None (Empty JeOS) ]
DBs [ None (Empty JeOS) ]
To list installed packages use command 'pkg list -s', optional software is installed in /opt

For OpenSolaris on Amazon EC2 program http://www.sun.com/third-party/global/amazon/index.jsp
For more info about JeOS creation and more tips look in blog http://blogs.sun.com/VirtualGuru

OS type: solaris.indiana Manifest: sun-osol/2008.05_JeOS_32_1.0.img.manifest.xml
Type: m1.small Index: 0 i-ID: i-8c19dbe5 AMI: ami-0d50b564 AKI: aki-7846a311 ARI: ari-7d46a314
Con: ec2-75-101-253-189.compute-1.amazonaws.com->ip-10-251-123-177.ec2.internal
Con: 75.101.253.189->10.251.123.177 AZ: us-east-1c SG: default KEY: osol-rk159669-keypair

2. How parameterizing and updade of Amazon EC2 instance work

Using Parameterized Launches to Customize Your AMIs

We will use file based parameterizing, max size is 16KB and data is automatically uu-encoded by Amazon EC2 tools. Its what you pass, that you will get service. Add instances running with one command will get same data.

ec2-run-instances ami-1234567 -f parfile.zip

curl http://169.254.169.254/latest/user-data -O dat.file

unzip /tmp/payload.zip 

Patching AMI Instances with Updates on Amazon S3

Updating/Backuping to S3

Most popular is s3 tool ir Ruby based s3sync & s3cmd tools

Thread: Best Practice for securing Access Key ID/Secret Key
AWS article about using s3sync on EC2
How I automated my backups to Amazon using s3sync
Series on using s3sync to backup a Linksys NSLU2
How_to_backup_the_Synology_Server_to_Amazon_S3

3. ec2ify:me main script and service

cat ec2ifyme.sh
#!/bin/bash

# This wrapper script call all ec2ify-me scripts
# To do: add error checking

# Allow execute aditional ec2ify scrips, which user can customize by executing custom actions
# Time out is set to 1200s =  20 minutes

.  /lib/svc/share/smf_include.sh

echo "ec2ify got $1 !"

# Amazon AWS use this internal DNS names:
# domU-12-31-39-00-69-D8.ec2.internal or domU-12-31-39-00-2D-61.compute-1.internal

myHOSTNAME=`/usr/bin/hostname`
myFQDN=`cat /etc/hosts | /usr/bin/grep $myHOSTNAME | /usr/bin/awk '{print \\$2}'`

echo "ec2ify: Runing on hostname $myHOSTNAME with FQDN $myFQDN"

# We need to use PERL FQDN !!!
#IsAmazonAWS=`echo ${myFWDN} | /usr/bin/grep "internal\\$"`
#if [ $? = 0 ]; then
#    echo "ec2ify: YES, Amazon AWS internal network!"
#else
#    echo "ec2ify: NO, not Amazon AWS internal network!"
#    exit = $SMF_EXIT_OK
#fi

if [ -f /mnt/develop ] ; then
 echo "ec2ify: Develolers DRY run code "
 exit $SMF_EXIT_OK
fi

    if  [ -f /opt/ec2ify/scripts/ec2ifyme_info.sh ] ; then
        echo "Executing ec2ifyme_info.sh"
        /opt/ec2ify/scripts/ec2ifyme_info.sh $1
    fi

    if  [ -f /opt/ec2ify/scripts/ec2ifyme_userdata.sh ] ; then
       echo "Executing ec2ifyme_userdata.sh"
       /opt/ec2ify/scripts/ec2ifyme_userdata.sh $1
    fi

    if  [ -f /opt/ec2ify/scripts/ec2ifyme_updater.sh ]  ; then
       echo "Executing ec2ifyme_updater.sh"
       /opt/ec2ify/scripts/ec2ifyme_updater.sh $1
    fi

    if  [ -f /opt/ec2ify/scripts/ec2ifyme_mount.sh ]  ; then
       echo "Executing ec2ifyme_mount.sh"
       /opt/ec2ify/scripts/ec2ifyme_mount.sh $1
    fi

    if  [ -f /opt/ec2ify/scripts/ec2ifyme_syncer.sh ]  ; then
       echo "Executing ec2ifyme_syncer.sh"
       /opt/ec2ify/scripts/ec2ifyme_syncer.sh  $1
    fi

exit $SMF_EXIT_OK
cat ec2ifyme.xml
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='ec2ify:me'>

<service name='ec2ify/me' type='service' version='1'>
        <create_default_instance enabled='true' />

        <single_instance />
        <dependency name='fs-local' grouping='require_all' restart_on='none' type='service'>
            <service_fmri value='svc:/system/filesystem/local' />
        </dependency>
        <dependency name='network-service' grouping='require_all' restart_on='none' type='service'>
            <service_fmri value='svc:/network/service' />

        </dependency>
        <exec_method type='method' name='start' exec='/opt/ec2ify/scripts/ec2ifyme.sh start' timeout_seconds='1200' />
        <exec_method type='method' name='stop' exec='/opt/ec2ify/scripts/ec2ifyme.sh stop' timeout_seconds='1200' />
        <exec_method type='method' name='refresh' exec='/opt/ec2ify/scripts/ec2ifyme.sh refresh' timeout_seconds='1200' />
        <exec_method type='method' name='restart' exec='/opt/ec2ify/scripts/ec2ifyme.sh restart' timeout_seconds='1200' />
        <property_group name='startd' type='framework'>

                <propval name='duration' type='astring' value='transient' />
        </property_group>
        <stability value='Unstable' />
        <template>
                <common_name>
                        <loctext xml:lang='C'>Amazon EC2 ec2ify me script - updating Solaris image for Amazon AWS infrastruture</loctext>

                </common_name>
        </template>
</service>
cat smfadd-me.sh
#!/usr/bin/sh
rm -f /var/svc/log/ec2ify-me:default.log
/usr/sbin/svccfg validate /opt/ec2ify/scripts/ec2ifyme.xml
/usr/sbin/svccfg import /opt/ec2ify/scripts/ec2ifyme.xml
sleep 3
/usr/sbin/svcadm enable svc:/ec2ify/me:default

4. ec2ifyme_info script

cat ec2ifyme_info.sh
#!/bin/bash

# !!! You need to use full path to bins in SMF scrips !!!

# ec2ifyme_info - getting Amazon EC2 instance data
# Create script which can be sourced in other scripts

echo "ec2ifyme_info got $1"

case $1 in

 'start')

    echo "Getting data from Amazon AWS "
    echo "#!/usr/bin/bash" >/opt/ec2ify/rundata/ec2envs.run
    echo ec2_ami_id=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/ami-id --silent` >>\\
/opt/ec2ify/rundata/ec2envs.run
    echo ec2_ami_launch_index=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/ami-launch-index \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_ami_manifest_path=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/ami-manifest-path \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_hostname=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/hostname \\
--silent` >>/opt/ec2ify/rundata/ec2envs.run
    echo ec2_instance_id=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/instance-id \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_instance_type=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/instance-type \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_kernel_id=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/kernel-id \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_local_hostname=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/local-hostname  \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_local_ipv4=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/local-ipv4 \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_placement=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/placement/availability-zone \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_profile=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/profile \\
 --silent` >>/opt/ec2ify/rundata/ec2envs.run
    echo ec2_public_hostname=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/public-hostname \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_public_ipv4=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/public-ipv4 \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_public_keys=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/public-keys/ \\
--silent | awk -F= '{ print $2}'` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_ramdisk_id=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/ramdisk-id \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_reservation_id=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/reservation-id \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo ec2_security_groups=`curl --connect-timeout 10 --retry 3 --retry-delay 2 http://169.254.169.254/latest/meta-data/security-groups \\
--silent` >> /opt/ec2ify/rundata/ec2envs.run
    echo export ec2_ami_id ec2_ami_launch_index ec2_ami_manifest_path ec2_hostname ec2_kernel_id \\ 
>> /opt/ec2ify/rundata/ec2envs.run
    echo export ec2_local_hostname ec2_local_ipv4 ec2_placement ec2_profile \\
>> /opt/ec2ify/rundata/ec2envs.run
    echo export ec2_public_hostname ec2_public_ipv4 ec2_public_keys \\
 >> /opt/ec2ify/rundata/ec2envs.run
    echo export ec2_ramdisk_id ec2_reservation_id ec2_security_groups ec2_instance_id ec2_instance_type \\
>> /opt/ec2ify/rundata/ec2envs.run
    chmod 0777 /opt/ec2ify/rundata/ec2envs.run

    echo "Updating /etc/motd with instance data"

. /opt/ec2ify/rundata/ec2envs.run

    cat /opt/ec2ify/etc/motd >/etc/motd
    echo OS type: $ec2_profile Manifest: $ec2_ami_manifest_path \\
>>/etc/motd
    echo Type: $ec2_instance_type Index: $ec2_ami_launch_index i-ID: i-8c19dbe5 AMI: $ec2_ami_id AKI: $ec2_kernel_id ARI: $ec2_ramdisk_id \\
>> /etc/motd
    echo Con: $ec2_public_hostname-\\>$ec2_local_hostname >>\\
/etc/motd
    echo Con: $ec2_public_ipv4-\\>$ec2_local_ipv4 AZ: $ec2_placement SG: $ec2_security_groups KEY: $ec2_public_keys \\ 
>> /etc/motd
;;

esac

exit 0

6. ec2ifyme_userdata script

cat ec2ifyme_userdata.sh
#!/bin/bash

# !!! You need to use full path to bins in SMF scrips !!!

. /opt/ec2ify/rundata/ec2envs.run

echo "ec2ifyme_data got $1"

# Get data file archive from Amazon AWS user-data as file param
# data file archive is passed to instance as parameter
# Hmm, it always get file , error as XML :-(

/usr/bin/curl --connect-timeout 60 --retry 3 --retry-delay 2 \\
  http://169.254.169.254/latest/user-data -o /opt/ec2ify/rundata/userdata.dat

if [ $? -eq 0 ] ; then

 echo "Got OK user-data "
 current=$PWD
 cd /opt/ec2ify/rundata

else

 echo "Can't get user-data !"
 exit 0

fi

# Detect type of archive and unpack it (in two runs)
# zip, gtar.[gzip|bzip2|Z], cpio.[gzip|bzip2|Z], executable

# First RUN, comressors, executable

filetype1=`{ LC_ALL=C /usr/bin/file /opt/ec2ify/rundata/userdata.dat | /usr/bin/awk '{print \\$2}' ; } 2>/dev/null`

case $filetype1 in

'executable')

  chmod 0700 /opt/ec2ify/rundata/userdata.dat
  /opt/ec2ify/rundata/userdata.dat
  cd $current
  exit 0

;;


'ZIP')

 /usr/bin/unzip /opt/ec2ify/rundata/userdata.dat

;;

'gzip')

 # mv to .gz extension, ungzip it /opt/ec2ify/rundata

;;

'bzip2')

 # mv to .bz2 extension, unbzip2 it /opt/ec2ify/rundata

;;

'compressed')

# mv to .Z extension, ununcompress it into /opt/ec2ify/rundata

;;

\*)

 # Can't detect type on first run, try second

 echo "ec2ify_userdata: Can't detect type on first run, trying second run"

;;

esac

# Second RUN archives

filetype2=`{ LC_ALL=C /usr/bin/file /opt/ec2ify/rundata/userdata.dat | /usr/bin/awk '{print $2}' ; } 2>/dev/null`

case $filetype2 in


'cpio')

 # rename to .cpio , to call cpio and unpack it into /opt/ec2ify/rundata/
 # Important

;;

'USTAR')

 # rename to .tar , to call gtar and unpack it into /opt/ec2ify/rundata/
 # Important (YES for LINUX compatiblity use GNU TAR !!!)

;;

\*)

 # Can't detect type on second run, leaving untouched

 echo "ec2ify_userdata: Can't detect type on second run, leaving untouched"

;;

esac

cd $current

exit 0

7. ec2ifyme_updater script

cat ec2ifyme_updater.sh
#!/bin/bash

# !!! You need to use full path to bins in SMF scrips !!!

. /opt/ec2ify/rundata/ec2envs.run

# ec2ifyme_updater - update Amazon EC2 instance with
# script from instance user data

echo "ec2ifyme_updater got $1"

case $1 in

 'start')

    echo "Checking update script "

    if [ -f /opt/ec2ify/rundata/updater.run ]; then

      echo "Got update script, executing"

      chmod 0700 /opt/ec2ify/rundata/updater.run

      /opt/ec2ify/rundata/updater.run

    fi
;;

esac

exit 0

8. ec2ifyme_mount script

cat ec2ifyme_mount.sh
#!/bin/bash

# !!! You need to use full path to bins in SMF scrips !!!

# More info about ZFS tips see blog http://blogs.sun.com/VirtualGuru

echo "ec2ifyme_mount got $1"

. /opt/ec2ify/rundata/ec2envs.run

case $1 in

 'start')

# If alternate mount script is in /etc/ec2ify/rundata exec it instead

  if [ -f /etc/ec2ify/rundata/mount.run ]; then
    echo "Executing alternative mount script"
    chmod 0700 /etc/ec2ify/rundata/mount.run
    /etc/ec2ify/rundata/mount.run
    exit 0
  fi

# If we don't want mounts
  if [ -f /etc/ec2ify/rundata/mount.no ]; then
    echo "Bypassing mount script"
    exit 0
  fi

# Dirty execution for now, if exist cmds will fail, also must detect UFS !

  echo "Trying to import mypool, dirty OS reboot failower"

  /usr/sbin/zpool import mypool

  echo "Creating ZFS data partition, mounting it in /mnt, adding 2GB or 16GB SWAP"

 if [ -s /dev/dsk/c4d1p0 ] ; then

      echo "We are on OpenSolaris Indiana line c4dNPN"

         if [ $ec2_instance_type = m1.small ]; then /usr/sbin/zpool create mypool c4d1p0 ; fi
         if [ $ec2_instance_type = c1.medium ]; then /usr/sbin/zpool create mypool c4d1p0 ; fi
         if [ $ec2_instance_type = m1.large ]; then /usr/sbin/zpool create mypool mirror c4d1p0 c4d2p0 ; fi
         if [ $ec2_instance_type = m1.xlarge ]; then /usr/sbin/zpool create mypool c4d1p0 c4dp02 c4d3p0 c4d4p0 ; fi
         if [ $ec2_instance_type = c1.xlarge ]; then /usr/sbin/zpool create mypool c4d1p0 c4dp02 c4d3p0 c4d4p0 ; fi

      else

      echo "We are on OpenSolaris Nevada line c0dNpN"
         if [ $ec2_instance_type = m1.small ]; then /usr/sbin/zpool create mypool c0d1p0 ; fi
         if [ $ec2_instance_type = c1.medium ]; then /usr/sbin/zpool create mypool c0d1p0 ; fi
         if [ $ec2_instance_type = m1.large ]; then /usr/sbin/zpool create mypool mirror c0d1p0 c0d2p0 ; fi
         if [ $ec2_instance_type = m1.xlarge ]; then /usr/sbin/zpool create mypool c0d1p0 c0d2p0 c0d3p0 c0d4p0 ; fi
         if [ $ec2_instance_type = c1.xlarge ]; then /usr/sbin/zpool create mypool c0d1p0 c0d2p0 c0d3p0 c0d4p0 ; fi
 fi

  /usr/sbin/zpool list

# Create swap 2Gb for 32bit or 16Gb for lagre instances

# On SXDE79a m1.small and c1.medium as 32bit can use only 2 GB

  if [ $ec2_instance_type = m1.small ]; then /usr/sbin/zfs create -V 2G mypool/swapvol ; fi
  if [ $ec2_instance_type = c1.medium ]; then /usr/sbin/zfs create -V 2G mypool/swapvol ; fi
  if [ $ec2_instance_type = m1.large ]; then /usr/sbin/zfs create -V 16G mypool/swapvol ; fi
  if [ $ec2_instance_type = m1.xlarge ]; then /usr/sbin/zfs create -V 16G mypool/swapvol ; fi     
  if [ $ec2_instance_type = c1.xlarge ]; then /usr/sbin/zfs create -V 16G mypool/swapvol ; fi     

# There is a potential issue with SWAP on ZFS volume when system is low with memory
# You can disable SWAP on ZFS with 'swap -d /dev/zvol/dsk/mypool/swapvol'
#
# See more:
# 6528296 System hangs when swapping to a zvol
# http://bugs.opensolaris.org/view_bug.do?bug_id=6528296

  /usr/sbin/swap -a /dev/zvol/dsk/mypool/swapvol

  /usr/sbin/swap -h -l

  /usr/sbin/zfs create mypool/mnt

# /mnt must be empty ???

  /usr/sbin/zfs set mountpoint=/mnt mypool/mnt

  /usr/bin/df -h | /usr/bin/grep /mnt

;;

esac

exit 0

9. ec2ifyme_syncer script

Hint: s3sync tools is part of ec2ify dirs under /opt
cat ec2ifyme_syncer.sh
#!/bin/bash

# !!! You need to use full path to bins in SMF scrips !!!

. /opt/ec2ify/rundata/ec2envs.run

# Will call script which can sync data from s3, its separate becuase it can
# and need to be run as completely new process, to don't be distrubed by SMF timeout ?
# s3sync and s3cmd are in /opt/ec2ify/tools dir, keys will be sourced from ec2ify-userdata
# syncer will be rin on all SMF events: start, stop, restart,
# Its your responciblity to impement locking so when system system for example:
# !!! shutdown quickly after start up you will not backup harmed data !!!

if [ -f /opt/ec2ify/rundata/s3keys.run ] ; then

. /opt/ec2ify/rundata/s3keys.run

fi

# ec2ifyme_updater - update Amazon EC2 instance with
# script from instance user data

echo "ec2ifyme_syncer got $1"

    echo "Checking syncer script "

    if [ -f /opt/ec2ify/rundata/syncer.run ]; then

      echo "Got syncer script, executing"

      chmod 0700 /opt/ec2ify/rundata/syncer.run

      /opt/ec2ify/rundata/syncer.run $1

    fi

exit 0

10. Debugging ec2ify:me

It will be offline\* until all scrips are finished

svcs -a | grep ec2ify/me
offline\*       11:43:38 svc:/ec2ify/me:default

svcs -a | grep ec2ify
online         11:43:35 svc:/ec2ify/me:default

svcs -l svc:/ec2ify/me:default
fmri         svc:/ec2ify/me:default
name         Amazon EC2 ec2ify me script - updating Solaris image for Amazon AWS infrastruture
enabled      true
state        online
next_state   none
state_time   Wed May 21 11:43:35 2008
logfile      /var/svc/log/ec2ify-me:default.log
restarter    svc:/system/svc/restarter:default
dependency   require_all/none svc:/system/filesystem/local (online)
dependency   require_all/none svc:/network/service (online)

cat   /var/svc/log/ec2ify-me:default.log | more

tail -f /var/svc/log/ec2ify-me:default.log
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Hands-on experience with Virtualization

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today