Why are my NFS mount options being ignored?

Have you ever specified a particular NFS mount option, when mounting an NFS filesystem, only to find that the system seems to have ignored you, and the mount option isn’t set as you specified?

Here’s an example:

$ sudo mount -o nconnect=4 ol9-nfs:/export/cdm2 /mnt/mnt2

$ nfsstat -m
    …
/mnt/mnt2 from ol9-nfs:/export/cdm2
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.1,local_lock=none,addr=192.168.1.2

In the above example, we’ve specified that we would like to set the nconnect mount option to ‘4’.

Yet when we look at the currently mounted NFS filesystems, with nfsstat -m, we don’t see the specified value for nconnect. In fact we don’t see it mentioned at all.

It’s not because we misspelt the name of the mount option, or it isn’t supported. If that were the case, we’d see an error, e.g:

$ sudo mount -o zconnect=4 ol9-nfs:/export/cdm2 /mnt/mnt2
mount.nfs: an incorrect mount option was specified

The reason that nfsstat -m is not showing us a value for nconnect is that it only shows values for nconnect if they are different from the default, which is 1.

[Actually, nfsstat -m merely shows us what the kernel is reporting for NFS mounts, via the file /proc/mounts. Arguably, the kernel is somewhat inconsistent here, since some values/flags are reported even when set to their defaults, e.g. rw]

So why is the value of nconnect still set to ‘1’, the default, when we asked for it to be set to ‘4’? To explain that we need to talk about RPC transports.

RPC Transports

NFS clients talk to NFS servers over a network connection using the NFS protocol. The NFS protocol is itself implemented using RPC: the Remote Procedure Call protocol. When an RPC client (e.g. our NFS client) wants to first make an RPC remote procedure call to an RPC server (our NFS server), the Linux kernel will create an RPC network transport, which encapsulates the information necessary to make repeated RPC calls to that RPC server.

It’s important to know that NFS will, where possible, try to share the use of already existing RPC transports between NFS operations. One consequence is that, in general:

NFS mounts of different exported filesystems, from the same NFS server, will all share a single RPC transport.

NFS mount options, per-filesystem or per-transport?

It’s also important to realise that:

Some NFS mount options are specific to the filesystem being mounted, whereas others are specific to the RPC transport being used for this mounted filesystem, which may itself be shared with other NFS filesystems mounted from the same NFS server.

We can now see the full sequence of events that caused the situation seen in the first example above:

$ sudo mount -o ol9-nfs:/export/cdm1 /mnt/mnt1

$ nfsstat -m
/mnt/mnt1 from ol9-nfs:/export/cdm1
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.1,local_lock=none,addr=192.168.1.2


$ sudo mount -o nconnect=4 ol9-nfs:/export/cdm2 /mnt/mnt2

$ nfsstat -m
/mnt/mnt1 from ol9-nfs:/export/cdm1
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.1,local_lock=none,addr=192.168.1.2

/mnt/mnt2 from ol9-nfs:/export/cdm2
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.1,local_lock=none,addr=192.168.1.2

Above, we see an initial mount of filesystem 1; no value for nconnect is given in the mount command, so it defaults to ‘1’, and is thus not shown by nfsstat -m. Here, we assume that there were no other NFS filesystems already mounted from this NFS server. That means that the NFS client kernel will have created a new RPC transport, for NFS operations to this NFS server. The nconnect value associated with that RPC transport is now set to ‘1’, the default.

We then mount a second filesystem 2, from the same NFS server. Since we already have an existing RPC transport for that NFS server (from the mount of filesystem 1), that will be reused, so that now filesystems 1 and 2 are sharing the RPC transport for that NFS server.

Importantly, that means that filesystems 1 and 2 are now sharing the RPC transport-specific mount options that were used for the first mount, that of filesystem 1. Since the mount of filesystem 1 did not specify a value for nconnect, and was thus given the default value of ‘1’, that is the value that will be used for filesystem 2, also.

Any RPC transport-specific mount options given, during the mount of a subsequent filesystem, where there is already a filesystem mounted from a given NFS server, will be silently ignored (no error or warning will be given), and the existing value of that option will be used for all NFS filesystems mounted from that server, sharing its single RPC transport.

To put it another way:

RPC transport-specific options must be specified when mounting the first filesystem from a particular NFS server.

To see that in action, we need to first make sure that all filesystems from our NFS server are unmounted:

sudo umount /mnt/mnt1
sudo umount /mnt/mnt2

The unmount of the final NFS filesystem using an RPC transport will cause that transport to be removed (but see the further point noted below).

We then mount the filesystems once more, taking care to specify our value for nconnect during the first mount:

$ sudo mount -o nconnect=4 ol9-nfs:/export/cdm1 /mnt/mnt1
$ sudo mount -o ol9-nfs:/export/cdm2 /mnt/mnt2

$ nfsstat -m
/mnt/mnt1 from ol9-nfs:/export/cdm1
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.1,local_lock=none,addr=192.168.1.2

/mnt/mnt2 from ol9-nfs:/export/cdm2
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.1,local_lock=none,addr=192.168.1.2

The first mount, of filesystem 1, causes a new RPC transport to be created, and given the nconnect value of ‘4’, as specified. It can be seen above that the second filesystem 2 also uses this value, despite it not being specified during the mount of filesystem 2, because both filesystems are sharing the same RPC transport, and thus its options.

A few further points should also be noted…

• in some cases, the RPC transport may not be freed immediately, upon unmount of the last NFS filesystem using it. The NFS client kernel may keep the transport around for a short time in order to cache connection state for reuse, or to avoid premature NFSv4 lease expiry. If the original issue isn’t resolved, following unmount of all filesystems from a particular NFS server, then try unmounting them again, then waiting a few minutes before remounting, to allow time for the kernel to decide to free the associated transport.
• there’s a further oddity in relation to the mount option nconnect, when it is used in association with pNFS servers. In this case, RPC transports for mounts from pNFS Data Servers (DS) will share the nconnect value used for the associated pNFS MetaData Server (MDS).
• the nconnect mount option does not apply when using pNFS with SCSI layouts where the pNFS client accesses the block storage from the DS directly via iSCSI.

Which NFS mount options are RPC transport-specific?

Fortunately, there aren’t many NFS mount options which are RPC transport-specific; the main ones are:

• nconnect
• max_connect
• retrans
• retry
• timeo

In general, if an NFS mount option does not seem to be applied, the first thing to check is whether there are existing NFS mounts from the same NFS server, indicating that an RPC transport has already been created for this NFS server, and the RPC transport-specific options used for its first mount will be shared for all subsequent mounts from this NFS server (whilst a previous mount is still mounted).

Further discussion

A future blog will show in more detail how to examine what RPC transports are being used, for which NFS mounted filesystems.

Ironically, the nconnect mount option itself — along with max_connect — give some control over how many RPC transports may be created and shared amongst NFS mounts, to aid in performance tuning.

• nconnect=n — specifies the number of network connections that will be established for operation to this NFS server IP address (which will be shared between all NFS mounts from this server). The maximum is 16, and the default is 1.
• max_connect=n — specifies the maximum number of trunked NFSv4.1 (or higher) network connections (which may be to different server IP addresses on the same NFS v4.1 server). The maximum is 16, and the default is 1.

A future blog will investigate the use of these options in more detail.