Using large DTLB page sizes

The TLB is a structure on the chip that handles the mapping of virtual memory addresses (used by the application) into physical memory addresses (used by the hardware). It is a list of such mappings, each mapping describes a range of memory (called the page size), the default on SPARC in 8KB page sizes, but it can be configured up to impressively large sizes (eg 256MB for UltraSPARC T1). The command to display what page sizes the hardware supports is pagesize:

pagesize -a

If the application requests a virtual to physical translation that is not mapped in the TLB, then there's a TLB miss. On UltraSPARC III/IV the process of fetching a TLB entry takes about a hundred cycles.

Using a larger page size will reduce the number of TLB misses. Of course a large page size requires a large chunk of contiguous physical memory, and it's not always possible to get this.

An application can request large pages in one of three ways:

  • Using the ppgsz command to set the preferred page sizes.
  • Using the compiler flag -xpagesize= to set the preferred page size at compile time.
  • Preloading the mpss.so.1 library and using the MPSSHEAP, MPSSSTACK environment variables to describe the page size.

When an application is running it is possible to inspect the page sizes of the allocated memory using the command:

pmap -xs <pid>
Comments:

Just curious: Why isn't there any application in the OS/Net tree which uses larger pagesizes than the default (just looking at UltraSPARC which defaults to 8k but has 64k pages which would be usefull in many cases) ?

Posted by Roland Mainz on April 18, 2007 at 09:29 AM PDT #

I agree. 64KB page sizes would seem a good compromise in many situations - hopefully netting an 8x reduction in TLB misses. On his blog Denis describes some work in Solaris 10 for UltraSPARC T1 that tries to improve out-of-the-box performance by automatically selecting larger page sizes. In some ways this is the ideal way to go, but practically many folks will want to tune their own page sizes - hence the choice.

Posted by Darryl Gove on April 18, 2007 at 01:33 PM PDT #

Mhhh... but the automated selection of the pagesize does not affect the output of <code>|getpagesize()|</code> nor <code>|sysconf()|</code> which most applications use for "guidance".
In such cases weired things may happen, including that something like a custom memory allocator generates havoc because it allocates 64k pages "8k-page"-style (in one of the applications I am working with the result was excessive memory consumption because most of the 64k pages were only partially filled... fun... <code>;-(</code>).
Somehow I wish the old "64k-kernel"-project could be brought back to live (or Sun just release the patches made long ago) ...

Posted by Roland Mainz on April 18, 2007 at 01:51 PM PDT #

BTW: Just for the log: OS/Net soon gets it's first application which uses 64k pages for stack+heap by default... lets hope others follow that lead...

Posted by Roland Mainz on April 18, 2007 at 01:58 PM PDT #

Post a Comment:
Comments are closed for this entry.
About

Darryl Gove is a senior engineer in the Solaris Studio team, working on optimising applications and benchmarks for current and future processors. He is also the author of the books:
Multicore Application Programming
Solaris Application Programming
The Developer's Edge

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
5
6
8
9
10
12
13
14
15
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
The Developer's Edge
Solaris Application Programming
Publications
Webcasts
Presentations
OpenSPARC Book
Multicore Application Programming
Docs