Oracle Linux, virtualization , Enterprise and Cloud Management Cloud technology musings

  • January 3, 2013

dm nfs

A little known feature that we make good use of in Oracle VM is called dm nfs. Basically the ability to create a device mapper device directly on an nfs-based file/filesystem. We use this in Oracle VM 3 if your shared storage for the cluster is nfs based.

Oracle VM clustering relies on the OCFS2 clusterstack/filesystem that is native in the kernel (uek2/2.6.39-x). When we create an HA-enabled pool, we create, what we call, a pool filesystem. That filesystem contains an ocfs2 volume so that we can store cluster-wide data. In particular we store shared database files that are needed by the Oracle VM agents on the nodes for HA. It contains info on pool membership, which VMs are in HA mode, what the pool IP is etc...

When the user provides an nfs filesystem for the pool, we do the following :

  • mount the nfs volume in /nfsmnt/
  • create a 10GB sized file ovspoolfs.img
  • create a dm nfs volume(/dev/mapper/ovspoolfs> on this ovspoolfs.img file
  • create an ocfs2 volume on this dm nfs device
  • mount the ocfs2 volume on /poolfsmnt/
  • If someone wants to try out something that relies on block-based shared storage devices, such as ocfs2, but does not have iSCSI or SAN storage, using nfs is an alternative and dm nfs just makes it really easy.

    To do this yourself, the following commands will do it for you :

  • to find out if any such devices exist just type dmsetup table --target nfs
  • to create your own device, do something like this:
  • mount mynfsserver:/mountpoint /mnt
    dd if=/dev/zero of=/mnt/myvolume.img bs=1M count=2000
    dmsetup create myvolume --table "0 4096000 nfs /mnt/myvolume.img 0"

    So mount the nfs volume, create a file which will be the container of the blockdevice, in this case a 2GB file and then create the dm device. The values for the dmsetup command are the following:

    myvolume = the name of the /dev/mapper device. Here we end up with /dev/mapper/myvolume

    table = start (normally always 0), number of blocks/length, this is in 512byte blocks, so you double the number,nfs since this is on nfs, filename of the nfs based file,
    offset (normally always 0)

    So now you have /dev/mapper/myvolume, it acts like a normal block device. If you do this on multiple servers, you can actually create an ocfs2 filesystem on this block device and it will be consistent across the servers.

    Credits go to Chuck Lever for writing dm nfs in the first place, thanks Chuck :)
    The code for dm nfs is here.

    Join the discussion

    Comments ( 2 )
    • ksopyla Saturday, February 4, 2017


      I'm one of creator PLON datascience platform, we use docker heavily and I wonder if dm-nfs can help to run docker on nfs.

      We are trying to run docker on nfs, but only device mapper storage driver will allow for that operation. Unfortunately, default mode for Docker running the devicemapper storage driver is loop-lvm, so I started seeking the solution where I create block device on nfs. Your method seems ok, but when I run

      dmsetup create myvolume --table "0 4096000 nfs /mnt/myvolume.img 0"

      on my 14.04 Ubuntu box I get the error message:

      "device-mapper: reload ioctl on myvolume failed: Invalid argument

      Command failed"

      Could you help me, what is wrong? I noticed that when I list kernel modules "lsmod" I didn't see any df-nfs related module. How can I add this module.

      Thanks in advanced.

    • jbdenis Tuesday, December 12, 2017

      ksopyla > I guess you need to use unbreakable linux.

      Is there any development still happening on this feature ? Is it still used ?

      I would love to see this outside Oracle ecosystem. Any git repository available somewhere ?

      Thank you !
    Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.