Memory page coalescing update and Solaris 10
By mrbenchmark on Apr 01, 2005
Well, some of you may remember my december technical brief
talking about the Solaris memory page coalescing on high-end servers.
If you don't know what I'm talking about feel free to to send me an email at
email@example.com and I will give you the link.
Since this technical brief, I have received a lot of request on this topic
that I would like to answer today :
-> Request 1 : What is the list of issues linked to this one ?
Here is the list with bugIds so you have a complete picture :
4802594 - Idle loop degrades IO performance on large psets
5059920 - Idle loop is not scalable on large systems
5054052 - disp_getwork() is greedy and negatively impacts dispatch latency
5050686 - Solaris mutexes should be made more efficient under contention
5095432 - Oracle startup takes too long due to memory fragmentation
5046939 - kcage_freemem grows too large when large ISM segments assigned on SF15k
4904187 - page_freelist_coalesce() holds the page freelist locks for too long
--> Request 2 : You mentioned that the fixes for this issues are available in
IDRs for Solaris 8 and Solaris 9. What is the status of the patches ?
Good news here. Solaris 8 patches have just been released. The fixes for this issues
are available in the following kernel update patches (KUP) :
Solaris 8 Solaris 9
Sparc 117350-23 117171-17
x86 117351 117172
Now, you are ready to ask me : what about Solaris 10 ?
The answer for Solaris 10 (and nevada) is : in progress....
I tested last month some of this issues on Solaris 10 and while the
problems are still there (the page_freelist_coalesce() routine is in the common Solaris
code), the impact is much,much lower. As an example, the 10G Oracle startup testcase we
built took 50s on a normal system. With no 4M pages available , it took up to 2 hours on
Solaris 8, up to 15 minutes on Solaris 9 and up to 3 minutes on Solaris 10.
--> Request 3 : It is very complicated to get a memory picture on our system.
vmstat or sar data are not detailed enough. Can you help ?
No need here for complex packaged tools. The best kept secret of Solaris is
the numerous options of mdb. So if you write a little script like :
# Displaying the memory map...
echo Browsing memory...
mdb -k 2>/dev/null <<!
You will get this output :
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 36480 285 1%
Anon 12891 100 0%
Exec and libs 5106 39 0%
Page cache 208799 1631 5%
Free (cachelist) 139913 1093 3%
Free (freelist) 3780231 29533 90%
Total 4183420 32682
Physical 4116397 32159
Fri Apr 1 10:18:16 PST 2005
--> Request 4 : Solaris 10 provide updated memory structures and the page freelist is now available. Can we use it to get the amount of free 4M pages ?
This request came last week from the VOS escalation team. And the answer is : yes but it requires a close look at how the page_freelists is implemented to get the right number.
We worked on this question with my good friend Mike C. in December and here is
the updated script for Solaris 10 (yes, mdb again) :
# Walking the page_freelist in Solaris 10 to get the amount of 4M pages...
mdb -k 1>/tmp/1 2>&1 <<!
page_freelists+30::array uintptr_t 1 | \\
::print uintptr_t | ::array uintptr_t 0t18 | \\
::print uintptr_t | ::array uintptr_t 0t2 | \\
::print uintptr_t | ::grep ".!=0" | ::list page_t p_vpnext
cat /tmp/1 |grep -v failed|wc -l
And on my v490, I have :
v490 # ./4m_s10.sh
That's it for now...