NFSv4 Courteous Server

Introduction

The NFSv4 Courteous Server allows clients to continue operation even after experiencing a transient network outage by permitting clients’ uncontested locks to remain valid on the server even if the network outage lasts longer than the NFSv4 lease period. This feature was developed by Oracle for upstream Linux (v5.19) and available in UEK7 Update 1 as part of our ongoing effort to improve NFS for Linux users.

With Courteous Server, lock conflicts are handled automatically depending on the operational state of the network infrastructure, more details in the sections below.

The NFSv4 courteous Server feature is useful in configurations that have complex network infrastructure with many routers and switches, or where data has to travel long distant to reach its destination. In these configurations there are many points of failure that increases the possibility of transient network outage. Another unintended benefit of the NFSv4 Courteous Server is for NFSv4 laptop users. A user can mount an NFSv4 export, does some work on it then closes the lid to put the machine into hibernation. Days or weeks later the user can open up the lid and continue to work without having to remount the export, reopen files or reacquire any previous locks.

Background

The NFSv4 protocol is stateful. The NFSv4 server maintains a set of state tokens which describe state, such as file locks, delegations and share reservations, for as long as the client needs them. The collection of state managed by the server for a particular client is known as a lease. In order for this state to remain valid on the server, the client must maintain the lease either implicitly by performing a stateful operation, or explicitly via the RENEW (NFSv4.0) or stand-alone SEQUENCE (NFSv4.x) operation periodically. The default lease period for LINUX NFSv4.x server is 90 seconds.

Network Partition

A network partition refers to any loss of network connectivity between the NFS client and the NFS server, regardless of its root cause. This includes NIC failures, NIC driver bugs, network misconfigurations and administrative errors, routers and switches crashing and/or having software updates applied, even down to cables being physically pulled. In most cases, these network failures are transient, although the duration is unknown.

Network partition prevents the client from renewing its lease resulting in the sever destroying the entire client’s state, forcing the client to recover its state after network partition heals and the client reconnects to the server.

Effects of Network Partition on NFSv4 Operation

After the network partition heals and when the client reconnects to the server it has to perform a recovery procedure to re-establish its previous state. The time it takes the client to complete its recovery depends on the amount of state needing to be recovered. For a large configuration with multiple clients where each client has lots of state, the recovery process might take a long time before the clients can continue to send new requests.

The recovery process is also usually complex and error prone for many reasons. Avoiding having the client to recover its state during transient network partition improves the reliability and robustness of NFS operation.

Courteous Server

Courteous Server is a server which does not immediately expunge the client state on lease expiration. A Courteous Server continues to recognize previously generated state tokens as valid until conflict arises between the expired state and the requests from another client, or the server reboots.

Courtesy Client

The Courteous Server feature introduces the notion of a Courtesy client. A Courtesy client is a client whose lease has already expired but is still allowed to remain valid on the server. A Courtesy client has the following characteristics:

• Is an expired client that still has state on the server.
• There is no waiter (conflict) for any locks owned by the expired client.
• There is no conflict for any delegations granted to the expired client.
• When there are conflicts with other clients’ requests, the entire lease of the Courtesy client is destroyed, not just the conflicted state.

Types of Conflict

There are 3 types of conflict between the expired state of the Courtesy client and new requests from other clients that the Courteous Server needs to handle:

• File delegations are granted by the server to allow the client temporary exclusive control of the file. This allows the client to cache the data safely until there is conflict with another client. A Courtesy client might own delegations that conflict with OPEN requests from another NFSv4 client.
• Share reservation: is a mechanism to control access and sharing of a file. The Courtesy client might has a file opened with access mode, or deny mode, that conflict with OPEN from another NFSv4 client.
• File locks: NFSv4 file-range lock. The Courtesy client might own locks that conflict with LOCK request from another client.

Sources of Conflct

Conflict with the Courtesy client’s state can come from request of:

• Local thread: a local thread can request a lock that is owned by the Courtesy client.
• Another NFSv4 client: request from another NFSv4 client that resulted in a conflict with the state owned by the Courtesy client.
• NFSv3 client with NLM (Network Lock Manager): a NFSv3 client requests a lock, via NLM, that is owned by the Courtesy client.

Conflict Resolution

When a new request from another client comes in which conflicts with an existing lock (be it deny modes, delegations, or file locks) associated with an expired lease, the server is obliged to revoke the conflicting lock and grant access to the new client. The server has the choice to either (1) remove the specific state of the Courtesy client that causes the conflict or (2) destroy the client lease which includes all its expired state. Option 1 is the most efficient solution but with significant complexity. For the initial implementation of Courteous Server, option 2 is selected to introduce the feature with minimal risk to the NFS server’s stability.

Admin Tool

The /proc/fs/nfsd/clients interface was enhanced to allow the user to display Courtesy client status, seconds from last renew and to manually destroy the Courtesy client.

To display Courtesy client status:

[root]# cat /proc/fs/nfsd/clients/2/info
clientid: 0xf0d156a662a0deec
address: "10.0.0.1:1003"
status: courtesy
seconds from last renew: 198
name: "Linux NFSv4.1 nfstest-client1"
minor version: 1
Implementation domain: "kernel.org"
Implementation name: "Linux 5.18.0-rc6+ #1 SMP PREEMPT_DYNAMIC Fri May 27 22:29:45 GMT 2022 x86_64"
Implementation time: [0, 0]
callback state: UP
callback address: 10.0.0.1:0
[root@nfsvmf24 nfsd]#

To manually destroy the Courtesy client:

[root]# echo expire > /proc/fs/nfsd/clients/2/ctl

References

• Learn more about this in the original submission here!