Monday Mar 19, 2007

Solaris memory, the recurring question

 

One of most common questions I get asked by system administrators is around memory statistics displayed by utilities like “prstat” in Solaris , especially environments with Oracle databases. The confusion normally comes in with “prstat” displaying tens of gigabyte memory used when there is less than that in terms of physical memory installed.

The confusion is not something that is easy to resolve , it is basically a decision of how to account for memory usage. In the example below, a piece of shared memory is allocated by one process and then attached to by other processes, for arguments sake , the allocated memory size is 100MB.

Shared Memory 

The question is how to display this situation, if the shared memory is only displayed as part of the process that created it, the sum of all the used memory would be more accurate. This would still not be right though as every process attached is using the memory, if they were executing on separate machines they would probably each have their own segment of memory allocated ?

In our case they all use the same 100MB segment, but when seen as separate processes, they each have 100MB available.

If the shared memory segment is displayed as part of each process, the 100MB shared segment will be double counted as it really only exists once , but is attached to multiple processes. This is the way Solaris prefers to display the usage.

Lets look at a more practical example, below we have two very simple programs, one to create the shared memory segment (“shm_creator”) and one that attaches to this segment and maps it into its address space (“shm_attach”).

shm_creator

/\*  Simple program to demonstrate SHM allocation 
and its effect on the memory statistics
displayed by utilities like "prstat".
\*/
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>


#define SHMSZ 104857600 /\* 100MB \*/
#define true 1

main()
{
char c;
int shmid;
key_t key;
char \*shm, \*s;

/\*
\* We'll name our shared memory segment
\* "123321".
\*/
key = 123321;

/\*
\* Create the segment.
\*/
if ((shmid = shmget(key, SHMSZ, IPC_CREAT | 0666)) < 0) {
perror("shmget");
exit(1);
}

/\*
\* Now we attach the segment to our data space.
\*/
if ((shm = shmat(shmid, NULL, 0)) == (char \*) -1) {
perror("shmat");
exit(1);
}

/\*
\* Now put some things into the memory for the
\* other process to read.
\*/
s = shm;
strcpy(s,"Hello From Server ....");


/\*
\* Now we just sleep forever sothat we have time
\* to play with prstat etc.
\*/

while (true) {
sleep(60);
}

\*s = NULL;

exit(0);
}


 shm_attach

/\*  Simple program to demonstrate SHM allocation
and its effect on the memory statistics
displayed by utilities like "prstat".
\*/
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>

#define SHMSZ 104857600 /\* 100MB \*/
#define true 1

main()
{
int shmid;
key_t key;
char \*shm, \*s;

/\*
\* We need to get the segment named
\* "123321", created by the server.
\*/
key = 123321;

/\*
\* Locate the segment.
\*/
if ((shmid = shmget(key, SHMSZ, 0666)) < 0) {
perror("shmget");
exit(1);
}

/\*
\* Now we attach the segment to our data space.
\*/
if ((shm = shmat(shmid, NULL, 0)) == (char \*) -1) {
perror("shmat");
exit(1);
}

s = shm;

/\*
\* Sleep , print the string the server left
\* for us in the shared memory segment, Sleep again
\* do this forever .........
\*/

while (true) {
printf("String from SHM : %s \\n",s);
sleep (10);
}

\*s = NULL;


exit(0);
}

To help us distinguish our test processes from the rest of the Solaris statistics we will create a new user and Project to allocate these processes to. Do not know about Projects, then you have to read the documents referred to at the bottom of this page.

# groupadd -g 2222 test
# projadd -U test -p 2222 testproj
# projmod -c 'Test Project for SHM' testproj 

Lets run these programs, we will use the "newtask" command, which allows us to run processes in a specified Project. First the server process (“shm_creator”) which will allocate the shared memory segment.

# newtask -p testproj ./shm_creator

If we inspect the memory allocated to this Project with “prstat” we see that +- 100MB is used.

# prstat -J

Memory Statistics

Now we start the process that attaches to this memory segment, remember thus process does not allocate an additional 100MB of memory, it just attaches to the same shared memory segment created by the first process.

# newtask -p testproj ./shm_attach
 String from SHM :  Hello From Server ....

Inspecting the memory used via “prstat”, the project now shows a total of + 200MB allocated , but remember the real memory allocated is 100MB.

# prstat -J

Memory Statistics

 

Other tools that can be used to further inspect the shared memory used include “ipcs” and “pmap”. The “ipcs” command is used to display shared memory segments, message queues and semaphores in Solaris. For our purposes we will use it to have a look at the shared memory segment we have created :

# ipcs -m -J -t -p -o
IPC status from <running system> as of Tue Mar 20 00:53:02 GMT+4 2007
T ID KEY MODE OWNER GROUP NATTCH CPID LPID ATIME DTIME CTIME PROJECT
Shared Memory:
m 26 0x1e1b9 --rw-rw-rw- root root 2 3063 3065 0:25:23 no-entry 0:16:18 testproj

From the output we can see that there are two processes that are attached to this segment (NATTCH) and that the creator of this segment belongs to the “testproj” Project (PROJECT).

The second tool we will use is “pmap” , this utility displays detail of the address space for a particular process. In our case we will use it to look at the process that created the shared memory segment and the process that attached to it.

First we will look at at the process that created the shared segment :

# pmap -x -a `pgrep shm_creator`
3063: ./shm_creator
Address Kbytes RSS Anon Locked Mode Mapped File
08046000 8 8 8 - rwx-- [ stack ]
08050000 4 4 - - r-x-- shm_creator
08060000 4 4 4 - rwx-- shm_creator
F8A00000 102400 4 4 - rwxs- [ shmid=0x1a ]
FEED0000       4       4       4       - rwx--    [ anon ]
FEEE0000 736 724 - - r-x-- libc.so.1
FEFA8000 24 24 24 - rw--- libc.so.1
FEFAE000 8 8 8 - rw--- libc.so.1
FEFC0000 24 12 12 - rwx-- [ anon ]
FEFC9000 132 132 - - r-x-- ld.so.1
FEFFA000 4 4 4 - rwx-- ld.so.1
FEFFB000 8 8 8 - rwx-- ld.so.1
-------- ------- ------- ------- -------
total Kb 103356 936 76 -

In the above output one can see the 100MB segment, identified with the “shmid=0x1a” note.

If we look at the process that attached to this segment, we will see the same segment attached :

# pmap -x -a `pgrep shm_attach`
3065: ./shm_attach
Address Kbytes RSS Anon Locked Mode Mapped File
08046000 8 8 8 - rwx-- [ stack ]
08050000 4 4 - - r-x-- shm_attach
08060000 4 4 4 - rwx-- shm_attach
F8A00000 102400 4 4 - rwxs- [ shmid=0x1a ] FEED0000 4 4 4 - rwx-- [ anon ] FEEE0000 736 724 - - r-x-- libc.so.1 FEFA8000 24 24 24 - rw--- libc.so.1 FEFAE000 8 8 8 - rw--- libc.so.1 FEFC0000 24 12 12 - rwx-- [ anon ] FEFC9000 132 132 - - r-x-- ld.so.1 FEFFA000 4 4 4 - rwx-- ld.so.1 FEFFB000 8 8 8 - rwx-- ld.so.1 -------- ------- ------- ------- ------- total Kb 103356 936 76 -

This is only two processes sharing 100MB, if one was to look at an Oracle database, the pictures seems more distorted if one does not know how to interpret the statistics.

Every Oracle server process as well as every client shadow process maps the shared memory segments for the database. Combine this with the fact that these shared memory segments in Oracle is often multiple gigabyte big one can see why “prstat” could display large amounts of memory consumed.

As a last note, a similar case exists for shared libraries, as these are normally loaded only once in the operating system and mapped to all processes that use the same shared library. As these are normally a lot smaller than the gigabyte size shared areas allocated by databases they normally go unnoticed.

 

References: Administering Projects 

 


 

 

 

About

Hein

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today