Performance Improvements for the File System Cache in Solaris 10
By rmc on Jul 12, 2005
One of the major improvements in Solaris 10 was the improvement of the file system cache. Here is a quick summary of the perf changes that I grabbed recently using Filebench.
The file system caching mechanism was overhauled to eliminate the majority of the MMU overhead associated with the page cache. The new segkpm facility providesfast mappings for physical pages within the kernel's address space. It is used by the file systems to provide a virtual addres that can be used to copy data to and from the user's address space for file system I/O.
Since the available virtual address range in a 64 bit kernel is always larger than physical memory size, the entire physical memory can be mapped into the kernel. This eliminates the need to have to map/unmap pages everytime they are accessed via segmap, significantly reducing code path and the need for TLB shoot downs. In addition, segkpm can use large TLB mappings to minimize TLB miss overhead.
Some quick measurements on Solaris 8 and 10 show the code path reduction. There are three important cases:
- Small In-Memory: When the file fits in memory, and is smaller than the size of the segmap cache
- Large In-Memory: When the file fits in memory, and is larger than the size of the segmap cache
- Large Non-Cached: When the file does not fit in memory
Measurements show that the most significant gain is a reduction of CPU per system call when a file is in memory (fits in the page cache), and is larger than segmap (which is 12% of physmem on SPARC, and 64MB on x64). Importantly, this is the most common case I've seen with real applications, too.
|Random Read of 8k||Solaris 8||Solaris 10||Small In-Memory (Less than segmap)||89us||83us|
|Large In-Memory (Greater than segmap but less then physmem)||181us||98us|
The throughput gains for a CPU constrained system are shown here:
With Solaris 10 you should expect significant improvements for applications which are I/O intensive into the file system cache. The actual improvement will vary, and will be greater on systems with higher CPU counts. You should also expect to see the cross call count drop (see xcal in mpstat), and a significantly reduced amount of system time.
Technorati Tag: OpenSolaris
Technorati Tag: Solaris