Good post by Cormac Hogan (thank you)
In one of my very first blog posts last year around the new improvements made to VMFS-5 in vSphere 5.0, one of the enhancements I called out was related to the VAAI primitive ATS (Atomic Test & set). In the post, I stated that the ‘Hardware Acceleration primitive, Atomic Test & Set (ATS), is now used throughout VMFS-5 for file locking.’ This recently led to an obvious, but really good question (thank you Cody) - What are the operations that ATS now does in vSphere 5.0/VMFS-5 that it didn’t do in vSphere 4.1/VMFS-3?
Well, before we delve into that, I thought it might be a good idea to provide a general overview of VMFS locking.
VMFS is a distributed journaling filesystem. All distributed file systems need to synchronize operations between multiple hosts as well as indicate the liveliness of a host. In VMFS, this is handled through the use of an ‘on-disk heartbeat structure’. The heartbeat structure maintains the lock states as well as the pointer to the journal information.
In order to deal with possible crashes of hosts, the distributed locks are implemented as lease-based. A host that holds a lock must renew a lease on the lock (by changing a “pulse field” in the on-disk lock data structure) to indicate that it still holds the lock and has not crashed. Another host can break the lock if the lock has not been renewed by the current holder for a certain period of time.
When another ESXi host wants to access a file, it will check to see if that timestamp has been increased. If it has not been increased, this host can take over ownership of the file by removing the stale lock, place its own lock on the file, and generating a new timestamp.
On-disk Metadata Locks
VMFS has on-disk metadata locks which are used to synchronize metadata updates. Metadata updates are required when creating, deleting or changing file metadata, in particular, file length. Earlier versions of VMFS (an VMFS built on LUNs from non-VAAI arrays) use SCSI reservations to acquire the on-disk metadata locks. It is important to note that the SCSI reservations are not in place to do the actual metadata update. They are used to get the on-disk lock only, and once the lock has been obtained, the SCSI reservation is released. Therefore, to address a common misconception, the LUN is not locked with a SCSI reservation for the duration of metadata updates. They are only reserved to get the on-disk lock.
Acquiring on-disk locks is a very short operation compared to the latency of metadata updates. However you should not infer from this that metadata updates themselves are long; it is simply that metadata updates are typically longer than the the time taken to acquire the lock.
The VMFS version released with ESX version 3.5 introduced an updated distributed locking called ‘optimistic locking’. Basically, the actual acquisition of on-disk locks (involving SCSI reservations) is postponed as late as possible in the life cycle of a VMFS metadata transactions. Optimistic locking allows the number and duration of SCSI reservations to be reduced. This in turn reduces the impact of SCSI reservations on Virtual Machine I/O and other VMFS metadata I/O originating from other ESX hosts that share the volume.
When locks are acquired in non-optimistic mode, one SCSI reservation is used for each lock that we want to acquire.
In optimistic mode, we use one SCSI reservation for each set of locks required by a particular journal transaction. Optimistic locking uses one SCSI reservation per transaction as opposed to one SCSI reservation per lock.
We don’t commit the transaction to the on-disk journal unless we are able to update all optimistic locks to physical locks (using a single SCSI reservation). You may see the message: Optimistic Lock Acquired By Another Host. This means that a lock which was held optimistically (not yet acquired on-disk) during a transaction was found to have been acquired on-disk by a different host. If we are unable to do said update we roll back the transaction’s in-memory changes (no on-disk changes would have been made) and then we simply retry the transaction. But, as per the description, we are optimistic that this won’t occur very often, and that in the vast majority of cases, optimistic locks will be upgraded to physical locks without any contention.
Read on here