Using large pages support in Sun Web Server

  Solaris has a multiple page size support which has been explained here:
http://www.solarisinternals.com/wiki/index.php/Multiple_Page_Size_Support

I was running specweb ecommerce php benchmark and noticed that 4-8% of gains can be obtained on CMT systems by using 256M pages. When process address space getting bigger and bigger (specially for 64 bit), number of tlb entries grow significantly. I observed that in ecommerce specweb php runs, around 7% of CPU time was spent to handle tlb misses (trapstat output).  When I used 256M pages on CMT systems, I found out that CPU time spent because of tlb misses reduced to 2%. One of the interesting thing I observed that Sun Web Server tends to slow down with time. In my setting, I had set the file cache heap size to 2.5GB. I was wondering why WebServer is slowing down with time. What was really happening, that when Web Server starts, it initally only allocate 256 MB so used process address space was smaller and hence less number of tlb misses.  After some time as file cache starts expanding, more and more tlb misses starts happening. So server sounds sluggish after 20 minutes or so.
So if you are running Sun Web Server on CMT systems, you should try 256M page support.

To enable mpss support in Sun Web Server (64 bit) put the following in
bin/startserv:

LD_PRELOAD_64=/usr/lib/sparcv9/mpss.so.1; export LD_PRELOAD_64     
MPSSHEAP=256M; export MPSSHEAP

For 32 bit Web Server :
LD_PRELOAD=/usr/lib/mpss.so.1; export LD_PRELOAD
MPSSHEAP=256M; export MPSSHEAP

"pagesizes -a" can be used to determine the max page sizes supported on your system. For systems which have only 4M pages, following can be used :
LD_PRELOAD_64=/usr/lib/64/mpss.so.1; export LD_PRELOAD_64     
MPSSHEAP=4M; export MPSSHEAP

pmap -sx gives the following information :
# pmap -sx 1938
1938:   webservd -d /opt/SUNWwbsvr/https-nsapiphp/config -r /opt/SUNWwbsvr -t
         Address     Kbytes        RSS       Anon     Locked Pgsz Mode   Mapped File
...
0000000100B00000       6080       6080          -          -    - rwx--    [ heap ]
00000001010F0000     244800     244800     244224          -  64K rwx--    [ heap ]
0000000110000000    3932160    3932160    3932160          - 256M rwx--    [ heap ]
0000000200000000     262144     262144     262144          - 256M rwx--    [ heap ]
FFFFFFFF68000000       4096       4096       4096          -  64K rwx--    [ anon ]
...

Note that few page sizes are mapped with 256M pages while others are mapped
64K pages.


mpss setting has been used in specweb jsp publications.

Comments:

One caveat: On CMT systems with the recent updates of Solaris 10 running, it may be necessary to set the "max_uheap_lpsize" parameter to a value of 256M in /etc/system.

eg.,
\* 256M pages
set max_uheap_lpsize=0x10000000

I guess it wasn't clearly documented anywhere in the official documentation. Due to this, max_uheap_lpsize parameter may or may not have any impact on the maximum page size being selected for the process heap (you may see different behavior with different updates of Solaris 10 - setting this parameter has no negative impact though). If you suspect that 256M pages are not being used even when they are supposed to, try this parameter. It may help.

Posted by Giri Mandalika on May 20, 2009 at 06:26 PM PDT #

In my performance test results, pmap -sx output showed that it is using 256M pages.

Posted by Basant Kukreja on May 21, 2009 at 06:16 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

Basant Kukreja

Search

Top Tags
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today