Oracle on Solaris 10 : Fixing the 'ORA-27102: out of memory' Error
By Giri Mandalika on Nov 21, 2008
(Crossposting the 2+ year old blog entry from my other blog hosted on blogger. Source URL:
As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an
ORA-27102: out of memory error message. The system had enough free memory to serve the needs of Oracle.
SQL> startup ORA-27102: out of memory SVR4 Error: 22: Invalid argument
$ oerr ORA 27102 27102, 00000, "out of memory" // \*Cause: Out of memory // \*Action: Consult the trace file for details
Not so helpful. Let's look the alert log for some clues.
% tail -2 alert.log WARNING: EINVAL creating segment of size 0x000000028a006000 fix shm parameters in /etc/system or equivalent
Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example) responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in
Prior to Solaris 10,
shmsys:shminfo_shmmax parameter has to be set in
/etc/system with maximum memory segment value that can be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10 and later. On a Solaris 10 (or later) system, it can be verified as shown below:
% prtconf | grep Mem Memory size: 32760 Megabytes % id -p uid=59008(oracle) gid=10001(dba) projid=3(default) % prctl -n project.max-shm-memory -i project 3 project: 3: default NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT project.max-shm-memory privileged 7.84GB - deny - system 16.0EB max deny -
Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to create a memory segment (10G) larger than 8G. Hence the failure.
So, the solution is to configure the system with a value large enough for the shared segment being created, so Oracle succeeds in starting up the database instance.
On Solaris 9 and prior releases, it can be done by adding the following line to
/etc/system, followed by a reboot for the system to pick up the new value.
set shminfo_shmmax = 0x000000028a006000
shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter in
/etc/system even though it works as expected.
On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is how we do it on Solaris 10 and later:
% prctl -n project.max-shm-memory -r -v 10G -i project 3 % prctl -n project.max-shm-memory -i project 3 project: 3: default NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT project.max-shm-memory privileged 10.0GB - deny - system 16.0EB max deny -
Note that changes made with the
prctl command on a running system are temporary, and will be lost when the system is rebooted. To make the changes permanent, create a project with
projadd command and associate it with the user account as shown below:
% projadd -p 3 -c 'eBS benchmark' -U oracle -G dba -K 'project.max-shm-memory=(privileged,10G,deny)' OASB % usermod -K project=OASB oracleFinally make sure the project is created with
% projects -l ... ... OASB projid : 3 comment: "eBS benchmark" users : oracle groups : dba attribs: project.max-shm-memory=(privileged,10737418240,deny) % cat /etc/project ... ... OASB:3:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)
With these changes, Oracle would start the database up normally.
SQL> startup ORACLE instance started. Total System Global Area 1.0905E+10 bytes Fixed Size 1316080 bytes Variable Size 4429966096 bytes Database Buffers 6442450944 bytes Redo Buffers 31457280 bytes Database mounted. Database opened.
- What's New in Solaris System Tuning in the Solaris 10 Release?
- Resource Controls (overview)
- System Setup Recommendations for Solaris 8 and Solaris 9
- Man page of prctl(1)
- Man page of projadd
Addendum : Oracle RAC settings
Anonymous Bob suggested the following settings for Oracle RAC in the form of a comment for the benefit of others who run into similar issue(s) when running Oracle RAC. I'm pasting the comment as is (Disclaimer: I have not verified these settings):
Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.
# Recommended Oracle RAC system params ndd -set /dev/udp udp_xmit_hiwat 65536 ndd -set /dev/udp udp_recv_hiwat 65536 # For root processes like crsd prctl -n project.max-shm-memory -r -v 8G -i project system prctl -n project.max-shm-ids -r -v 512 -i project system # For oracle processes like sqlplus prctl -n project.max-shm-memory -r -v 8G -i project default prctl -n project.max-shm-ids -r -v 512 -i project default
So simple yet it took me a week working with Oracle and SUN to come up with that answer...Hope that helps someone out.
# posted by Blogger Bob : 6:48 AM, April 25, 2008