The NFSv4 Courteous Server allows clients to continue operation even after experiencing a transient network outage by permitting clients’ uncontested locks to remain valid on the server even if the network outage lasts longer than the NFSv4 lease period. This feature was developed by Oracle for upstream Linux (v5.19) and available in UEK7 Update 1 as part of our ongoing effort to improve NFS for Linux users.
With Courteous Server, lock conflicts are handled automatically depending on the operational state of the network infrastructure, more details in the sections below.
The NFSv4 courteous Server feature is useful in configurations that have complex network infrastructure with many routers and switches, or where data has to travel long distant to reach its destination. In these configurations there are many points of failure that increases the possibility of transient network outage. Another unintended benefit of the NFSv4 Courteous Server is for NFSv4 laptop users. A user can mount an NFSv4 export, does some work on it then closes the lid to put the machine into hibernation. Days or weeks later the user can open up the lid and continue to work without having to remount the export, reopen files or reacquire any previous locks.
The NFSv4 protocol is stateful. The NFSv4 server maintains a set of state tokens which describe state, such as file locks, delegations and share reservations, for as long as the client needs them. The collection of state managed by the server for a particular client is known as a lease. In order for this state to remain valid on the server, the client must maintain the lease either implicitly by performing a stateful operation, or explicitly via the RENEW (NFSv4.0) or stand-alone SEQUENCE (NFSv4.x) operation periodically. The default lease period for LINUX NFSv4.x server is 90 seconds.
A network partition refers to any loss of network connectivity between the NFS client and the NFS server, regardless of its root cause. This includes NIC failures, NIC driver bugs, network misconfigurations and administrative errors, routers and switches crashing and/or having software updates applied, even down to cables being physically pulled. In most cases, these network failures are transient, although the duration is unknown.
Network partition prevents the client from renewing its lease resulting in the sever destroying the entire client’s state, forcing the client to recover its state after network partition heals and the client reconnects to the server.
After the network partition heals and when the client reconnects to the server it has to perform a recovery procedure to re-establish its previous state. The time it takes the client to complete its recovery depends on the amount of state needing to be recovered. For a large configuration with multiple clients where each client has lots of state, the recovery process might take a long time before the clients can continue to send new requests.
The recovery process is also usually complex and error prone for many reasons. Avoiding having the client to recover its state during transient network partition improves the reliability and robustness of NFS operation.
Courteous Server is a server which does not immediately expunge the client state on lease expiration. A Courteous Server continues to recognize previously generated state tokens as valid until conflict arises between the expired state and the requests from another client, or the server reboots.
The Courteous Server feature introduces the notion of a Courtesy client. A Courtesy client is a client whose lease has already expired but is still allowed to remain valid on the server. A Courtesy client has the following characteristics:
There are 3 types of conflict between the expired state of the Courtesy client and new requests from other clients that the Courteous Server needs to handle:
Conflict with the Courtesy client’s state can come from request of:
When a new request from another client comes in which conflicts with an existing lock (be it deny modes, delegations, or file locks) associated with an expired lease, the server is obliged to revoke the conflicting lock and grant access to the new client. The server has the choice to either (1) remove the specific state of the Courtesy client that causes the conflict or (2) destroy the client lease which includes all its expired state. Option 1 is the most efficient solution but with significant complexity. For the initial implementation of Courteous Server, option 2 is selected to introduce the feature with minimal risk to the NFS server’s stability.
/proc/fs/nfsd/clients interface was enhanced to allow the user to display Courtesy client status, seconds from last renew and to manually destroy the Courtesy client.
To display Courtesy client status:
[root]# cat /proc/fs/nfsd/clients/2/info clientid: 0xf0d156a662a0deec address: "10.0.0.1:1003" status: courtesy seconds from last renew: 198 name: "Linux NFSv4.1 nfstest-client1" minor version: 1 Implementation domain: "kernel.org" Implementation name: "Linux 5.18.0-rc6+ #1 SMP PREEMPT_DYNAMIC Fri May 27 22:29:45 GMT 2022 x86_64" Implementation time: [0, 0] callback state: UP callback address: 10.0.0.1:0 [root@nfsvmf24 nfsd]#
To manually destroy the Courtesy client:
[root]# echo expire > /proc/fs/nfsd/clients/2/ctl