X

The latest cloud infrastructure announcements, technical solutions, and enterprise cloud insights.

Auto-Detecting and Attaching Block Storage Service Volumes

The use of iSCSI enables you to use commonly understood System Administration processes to manage storage and perform common operations, such as moving storage among instances, and ensure instances attach only to the particular storage devices they are explicitly allowed.  However, most operating systems do not have a mechanism for automatically/periodically scanning and attaching to iSCSI storage devices and enabling them for use.

Oracle, unlike other cloud providers, uses industry standard iSCSI to attach block volume storage to instances. This means that one of the challenges in adding Oracle Cloud block volumes to instances involves copying and pasting iscsiadm commands out of the console and to the command line.  Cutting and pasting command lines is not a complex task, but could introduce errors into your environment.  Since we only provide operating systems without any Oracle-specific customization, and we do not install or run any type of “agent” code after the operating system is provisioned on your instance, there is not a way to provide a pre-built tool to enable automatic attachment of iSCSI volumes to instances.

Attaching the block volumes in an automated fashion inside the Oracle Cloud is straightforward.  To attach iSCSI block volumes to an operating system on-premises, you have to know the IP address and target ID of the volume before you can perform any attachment operations.  Within the Oracle Cloud, explicitly gathering this information is not required.  Because we control how volumes are presented to instances you only need follow a few simple rules:

  • The instances are only exposed to the iSCSI volumes that have explicitly been attached via the console or API interfaces.
  • The IP address of the volumes will always fall within the range of 169.254.2.0 - 169.254.2.254. 
  • We only activate the first 34 addresses of this range.  The .1 address is reserved for the boot volume.  The address count reflects our current limit of 32 block volumes per instances (see https://docs.us-phoenix-1.oraclecloud.com/Content/Block/Concepts/overview.htm).
  • Each volume is assigned a specific IP address, depending on the order in which they were attached to the instance. So, the first volume attached would be assigned 169.254.2.2, the next .3, and so on.

Also, since only volumes intended for the instance are ever exposed, any iSCSI target detected is a valid one. This means you don't have to explicitly know which volume target ID to use - just use the discovery mode from iscsiadm to find all the targets.  Lastly, we always use the standard iSCSI port (3260), so a simple loop which spans the first 34 IP addresses will reveal all the iSCSI targets available to any particular instance.  It is then a simple process to write a script to loop through the targets and add them to the system configuration. The script might look something like this:

 

iscsiadd.sh

#!/bin/bash

 

BASEADDR="169.254.2.2"

# Set a base address counter to increment so we can loop through all the

# addresses.

addrCount=0

 

while [ ${addrCount} -le 32 ]

do

        CURRADDR=`echo ${BASEADDR} | awk -F\. '{last=$4+'${addrCount}';print $1"."$2"."$3"."last}'`

        clear

        echo "Attempting connection to ${CURRADDR}"

        mkfifo discpipe

        # Find all the iSCSI Block Storage volumes attached to the instance but

        # not configured for use on the instance.  Basically, get a list of the

        # volumes that the instance can see, the loop through the ones it has,

        # and add volumes not already configured on the instance.

        #

        # First get the list of volumes visible (attached) to the instance

        iscsiadm -m discovery -t st -p ${CURRADDR}:3260 | grep -v uefi | awk '{print $2}' > discpipe 2> /dev/null &

 

        # If the result is non-zero, that generally means that there are no targets available or

        # that the portal is reachable but not active.  We make no distinction between the two

        # and simply skip ahead.

        result=$?

        if [ ${result} -ne 0 ]

        then

               (( addrCount = addrCount + 1 ))

               continue

        fi

        

        # Loop through the list (via the named FIFO pipe below)

        while read target

        do

            mkfifo sesspipe

            # Get the list of the currently attached Block Storage volumes

            iscsiadm -m session -P 0 | grep -v uefi | awk '{print $4}' > sesspipe 2> /dev/null &

            

            # Set a flag, and loop through the sessions (attached, but not configured)

            # and see if the volumes match.  If so, skip to the next until we get

            # through the list.  Session list is via the pipe.

            found="false"

            while read session

            do

               if [ ${target} = ${session} ]

               then

                   found="true"

                   break

               fi

            done < sesspipe

            

            # If the volume is not found, configure it.  Get the resulting device file.

            if [ ${found} = "false" ]

            then

               iscsiadm -m node -o new -T ${target} -p ${CURRADDR}:3260

               iscsiadm -m node -o update -T ${target} -n node.startup -v automatic

               iscsiadm -m node -T ${target} -p ${CURRADDR}:3260 -l

               sleep 10

            fi

        done < discpipe

       

        (( addrCount = addrCount + 1 ))

        find . -maxdepth 1 -type p -exec rm {} \;

done

echo "Scan Complete."

The main work of finding the volumes happens at the bottom of the script.  Basically, it uses the 'discovery' mode of iscsiadm to find all targets that are possible to attach to this instance (it filters out the 'uefi' target because that is the boot device and we do not want to touch it). It then simply takes the list of targets (provided via the awk filter) and creates 'new' entries in the iscsiadm list.  The three commands in the loop add the target, update the target to start automatically at boot, and then 'login' to the target, making it available on the instance.  You don't even need to check for ones that are already attached, because the operating system call verifies that the entry is not a duplicate.

What if you want to have your instances detect the attached volumes once when initially deployed?  Simply provide a base64 encoded version of the script as user metadata when you deploy the instance.  The script will run once immediately after deployment and make any block volumes attached to the instance available for use.

If you want to periodically check if new volumes have been attached to your instance, and configure them if new ones are detected, put the script in the root crontab and schedule it to run as you wish.  For instance, the below entry in the root crontab would check for volumes once a day at midnight:

crontab entry for iscsiattach.sh

0 0 * * * * /usr/local/bin/iscsiadd.sh

Note, that while any new volumes are available to the operating system, you must still partition and format the volumes before you mount them.  That is, of course, if you want filesystems on them in the first place.

If you use block volumes for a file system and want it to be mounted at boot time via /etc/fstab, it is vitally important you use the correct mount options.  Failure to do so will result in your instance becoming completely inaccessible.  When you add the entry for the volume to /etc/fstab, be sure to use the "_netdev" as part of the mount point entries.  This option ensures the volume is not mounted until the network is enabled.  Your /etc/fstab entry per filesystem should look something like this:

/etc/fstab

/dev/sdb1      <your mount point>     ext4    defaults,_netdev,noatime       0       0

If you want to get really creative, you can install the Oracle Cloud CLI, create a user that only has the ability to create/attach volumes to an instance, and embed that into the script above. Then, you would have the ability to create and attach volumes to an instance without ever having to open the console!  But that is a subject for another blog...

Using the method described above of detecting and configuring block volumes will save you from having to cut and paste three iscsiadm commands for each volume. Cutting and pasting command lines is may not be a complex task, but could introduce errors into your environment. The information presented here may save you time and effort next time you need to attach block volumes to your instance.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha