Using Solaris zfs + iscsi targets with Oracle VM

I was playing with my Oracle VM setup and needed some shared storage that was block based. I did not have a storage array available but I did have a solaris box, that I use for Oracle VDI, available.

I set up a few iscsi targets on this solaris server and exported them to my 2 Oracle VM servers. Here's how I did this :

(1) On the solaris side :

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
rpool 544G 129G 415G 23% ONLINE -

I just have a simple zpool, called rpool, on this box. It has plenty of space available for my needs. So I will use rpool and I will create 5 50gb vols :

zfs create -V 50G rpool/ovm1
zfs create -V 50G rpool/ovm2
zfs create -V 50G rpool/ovm3
zfs create -V 50G rpool/ovm4
zfs create -V 50G rpool/ovm5

I want to use these volumes for iscsi so I have to enable them as shared iscsi devices :

zfs set shareiscsi=on rpool/ovm1
zfs set shareiscsi=on rpool/ovm2
zfs set shareiscsi=on rpool/ovm3
zfs set shareiscsi=on rpool/ovm4
zfs set shareiscsi=on rpool/ovm5

The command iscsitadm list target should list these devices so make sure they show up.

# iscsitadm list target
Target: rpool/ovm1
iSCSI Name: iqn.1986-03.com.sun:02:896c766c-0943-4da5-d47e-9575b5a0be36
Connections: 2
Target: rpool/ovm2
iSCSI Name: iqn.1986-03.com.sun:02:a3116b46-73e0-e8c2-e80c-9a4f71aff069
Connections: 2
Target: rpool/ovm3
iSCSI Name: iqn.1986-03.com.sun:02:a838c400-2730-c0d6-f2c2-ee186a0261c1
Connections: 2
Target: rpool/ovm4
iSCSI Name: iqn.1986-03.com.sun:02:2e046afb-d66d-4f3f-c5de-8115e0ddd931
Connections: 2
Target: rpool/ovm5
iSCSI Name: iqn.1986-03.com.sun:02:66109fbe-81ac-ef05-f85e-ab8c1f34cb43
Connections: 2

At this point I want to make sure that I have some access control on these devices. To make it easier, I will create an alias for my 2 servers and use the alias for the ACL.

get the iqn from the 2 servers
on my 2 ovm servers (wcoekaer-srv1, wcoekaer-srv2) get the content of
/etc/iscsi/initiatorname.iscsi (for each server)

InitiatorName=iqn.1986-03.com.sun:01:2a7526f0ffff

On the solaris side create the aliases :

iscsitadm create initiator -n iqn.1986-03.com.sun:01:2a7526f0ffff wcoekaer-srv1
iscsitadm create initiator -n iqn.1986-03.com.sun:01:e31b08110f1 wcoekaer-srv5

Add the ACL to the targets :

iscsitadm modify target -l wcoekaer-srv1 rpool/ovm1
iscsitadm modify target -l wcoekaer-srv1 rpool/ovm2
iscsitadm modify target -l wcoekaer-srv1 rpool/ovm3
iscsitadm modify target -l wcoekaer-srv1 rpool/ovm4
iscsitadm modify target -l wcoekaer-srv1 rpool/ovm5
iscsitadm modify target -l wcoekaer-srv5 rpool/ovm1
iscsitadm modify target -l wcoekaer-srv5 rpool/ovm2
iscsitadm modify target -l wcoekaer-srv5 rpool/ovm3
iscsitadm modify target -l wcoekaer-srv5 rpool/ovm4
iscsitadm modify target -l wcoekaer-srv5 rpool/ovm5

(2) the Oracle VM side

On each server just do 2 simple things :

# iscsiadm -m discovery -t sendtargets -p ca-vdi1 where ca-vdi1 is my solaris server name

# service iscsi restart

When I do cat /proc/partitions on my servers I will see the devices show up

# cat /proc/partitions
major minor #blocks name

8 0 160836480 sda
8 1 104391 sda1
8 2 3148740 sda2
8 3 1052257 sda3
253 0 6377804 dm-0
253 1 6377804 dm-1
253 2 6377804 dm-2
8 16 52428800 sdb
8 32 52428800 sdc
8 48 52428800 sdd
8 80 52428800 sdf
8 64 52428800 sde

These 5 new devices sd[b..f] are shared storage for Oracle VM and can be used to pass through to the VM's as phy: devices or put ocfs2 on it and use as shared filesystem storage for dom0 repositories.

I am setting up an 11gR2 rac template (the cool stuff Saar did) so I am using my devices to create a 2 node RAC cluster with phy: devices.

(updated content : )

To make sure we have persistent device names to pass through to the guest, it is important to have udev rules set up. This is what I did :

on one of the 2 servers, I ran, for each of the iscsi targets, the following commands :

# scsi_id -g -u -s /block/sdb
3600144f04d1cc91e00001e6857e16a00
# scsi_id -g -u -s /block/sdc
3600144f04d1cc91d00001e6857e16a00
# scsi_id -g -u -s /block/sdd
3600144f04d1cc91c00001e6857e16a00
# scsi_id -g -u -s /block/sde
3600144f04d1cc91f00001e6857e16a00
# scsi_id -g -u -s /block/sdf
3600144f04d1cc91b00001e6857e16a00

these are the unique IDs we can use with udev to create fixed entries on each server.

The next step is to add these id's to udev rules. I simply edited :
/etc/udev/rules.d/50-udev.rules and for each of the devices added a line like below :

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="36001
44f04d1cc91e00001e6857e16a00", SYMLINK+="rac1"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="36001
44f04d1cc91d00001e6857e16a00", SYMLINK+="rac2"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="36001
44f04d1cc91c00001e6857e16a00", SYMLINK+="rac3"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="36001
44f04d1cc91f00001e6857e16a00", SYMLINK+="rac4"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="36001
44f04d1cc91b00001e6857e16a00", SYMLINK+="rac5"

Next, restart iscsi service iscsi restart

and you will see the fixed devices in /dev :


[root@wcoekaer-srv1 ~]# ls -l /dev/rac*
lrwxrwxrwx 1 root root 3 Dec 30 14:35 /dev/rac1 -> sdd
lrwxrwxrwx 1 root root 3 Dec 30 14:35 /dev/rac2 -> sdf
lrwxrwxrwx 1 root root 3 Dec 30 14:35 /dev/rac3 -> sde
lrwxrwxrwx 1 root root 3 Dec 30 14:35 /dev/rac4 -> sdc
lrwxrwxrwx 1 root root 3 Dec 30 14:35 /dev/rac5 -> sdb

and the last step is to add these devices to my vm.cfg :

in the disk = [] section I just add :
'phy:/dev/rac1,xvdc,w!',
'phy:/dev/rac2,xvdd,w!',
'phy:/dev/rac3,xvde,w!',
'phy:/dev/rac4,xvdf,w!',
'phy:/dev/rac5,xvdg,w!',

now we're all set.

Comments:

iscsitadm is gone, you should be using COMSTAR instead

Posted by ignacio marambio catan on January 02, 2011 at 10:27 AM PST #

As I mentioned, this was using solaris 10.

Posted by wim.coekaerts on January 08, 2011 at 03:20 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

You can follow him on Twitter at @wimcoekaerts

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today