Oracle on Solaris 10 : Fixing the 'ORA-27102: out of memory' Error

(Crossposting the 2+ year old blog entry from my other blog hosted on blogger. Source URL:
http://technopark02.blogspot.com/2006/09/solaris-10oracle-fixing-ora-27102-out.html)

Symptom:

As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an ORA-27102: out of memory error message. The system had enough free memory to serve the needs of Oracle.

SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument

Diagnosis
$ oerr ORA 27102
27102, 00000, "out of memory"
// \*Cause: Out of memory
// \*Action: Consult the trace file for details

Not so helpful. Let's look the alert log for some clues.

% tail -2 alert.log
WARNING: EINVAL creating segment of size 0x000000028a006000
fix shm parameters in /etc/system or equivalent

Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example) responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in /etc/system.

Prior to Solaris 10, shmsys:shminfo_shmmax parameter has to be set in /etc/system with maximum memory segment value that can be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10 and later. On a Solaris 10 (or later) system, it can be verified as shown below:

% prtconf | grep Mem
Memory size: 32760 Megabytes

% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      7.84GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to create a memory segment (10G) larger than 8G. Hence the failure.

So, the solution is to configure the system with a value large enough for the shared segment being created, so Oracle succeeds in starting up the database instance.

On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system to pick up the new value.

set shminfo_shmmax = 0x000000028a006000

However shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter in /etc/system even though it works as expected.

On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is how we do it on Solaris 10 and later:

% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      10.0GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Note that changes made with the prctl command on a running system are temporary, and will be lost when the system is rebooted. To make the changes permanent, create a project with projadd command and associate it with the user account as shown below:

% projadd -p 3  -c 'eBS benchmark' -U oracle -G dba  -K 'project.max-shm-memory=(privileged,10G,deny)' OASB
% usermod -K project=OASB oracle
Finally make sure the project is created with projects -l or cat /etc/project commands.

% projects -l
...
...
OASB
        projid : 3
        comment: "eBS benchmark"
        users  : oracle
        groups : dba
        attribs: project.max-shm-memory=(privileged,10737418240,deny)

% cat /etc/project
...
...
OASB:3:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)

With these changes, Oracle would start the database up normally.

SQL> startup
ORACLE instance started.

Total System Global Area 1.0905E+10 bytes
Fixed Size                  1316080 bytes
Variable Size            4429966096 bytes
Database Buffers         6442450944 bytes
Redo Buffers               31457280 bytes
Database mounted.
Database opened.

Related information:

  1. What's New in Solaris System Tuning in the Solaris 10 Release?
  2. Resource Controls (overview)
  3. System Setup Recommendations for Solaris 8 and Solaris 9
  4. Man page of prctl(1)
  5. Man page of projadd


Addendum : Oracle RAC settings

Anonymous Bob suggested the following settings for Oracle RAC in the form of a comment for the benefit of others who run into similar issue(s) when running Oracle RAC. I'm pasting the comment as is (Disclaimer: I have not verified these settings):

Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.

# Recommended Oracle RAC system params
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536

# For root processes like crsd
prctl -n project.max-shm-memory -r -v 8G -i project system
prctl -n project.max-shm-ids -r -v 512 -i project system

# For oracle processes like sqlplus
prctl -n project.max-shm-memory -r -v 8G -i project default
prctl -n project.max-shm-ids -r -v 512 -i project default

So simple yet it took me a week working with Oracle and SUN to come up with that answer...Hope that helps someone out.

Bob
# posted by Blogger Bob : 6:48 AM, April 25, 2008

Comments:

Good.This doc helped me.

I had one more question.

I have a 3 node cluster , which is already configured,

When I do a ps -ef |grep oracle withy Project details , I see it running with user.oracle instead of the default system project.

How can I setup it up that way?

Regards,

vijaykumar Kammar

Posted by vijay on December 22, 2008 at 11:00 PM PST #

Hi,

Thanks a lot for this blog...helped me solve a critical issue!!

~Kaushik

Posted by Kaushik Sethna on March 06, 2009 at 02:49 AM PST #

Thanks a lot for this blog...helped me solve a critical issue!!

Posted by seo on March 09, 2009 at 04:35 AM PDT #

tnx

Posted by Evgeny on March 18, 2009 at 10:41 PM PDT #

We were struggling with the same problem for two days.
This article is written in very precise manner and we could solve our problem.

Thanks for such a nice article.

Posted by Tanweer Khan on April 07, 2009 at 04:33 PM PDT #

Thank you for this great blog. I was trying to install an SAP - Oracle central instance and it wasn't going well. Very informative and easy to follow. Thanks much!

Posted by Rick on April 21, 2009 at 04:11 AM PDT #

Thanks for such a good and concise article

Regards
Lanto

Posted by Lanto RAZAKALALAO on April 27, 2009 at 09:49 PM PDT #

good doc

Posted by Babu on June 07, 2009 at 02:42 PM PDT #

Thanks this blog.
but I found there have some issue when set default project:

bash-3.00# prctl -n project.max-shm-memory -r -v 8G -i project default
prctl: default: No controllable process found in task, project, or zone.

but I edited the file /etc/project as following, it worked.

zone1 oracle $ cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::project.max-shm-memory=(privileged,2147483648,deny)

Posted by Jason Wang on July 20, 2009 at 01:10 PM PDT #

So Sun, in its infinite wisdom, took a process where all one had to do was edit one file and replaced it with this debacle?

No wonder the company died.
How does a change like this even get approved?

Lets take a simple, one step process and replace it with projects, multiple settings and even more commands with untold flags that one has to know.

Brilliant. Thanks.

Posted by Ian on August 12, 2009 at 04:13 AM PDT #

So Sun, in its infinite wisdom, took a process where all one had to do was edit one file and replaced it with this debacle?

No wonder the company died.

http://www.gifico.com
http://www.lv-cheap.com

Posted by louis vuitton on November 18, 2009 at 03:11 PM PST #

I'm running into trouble using the projmod commands to set max-shm-memory for a group above 16GB. My system memory is 64GB. Is there a hard-cap on project memory of which I should be aware?

Posted by Aaron Long on May 20, 2010 at 08:14 AM PDT #

I was very impressed with this article. First go off the bat and the DBA could start the DB`s

Regards,
Avin...

Posted by Avin on November 17, 2010 at 11:42 AM PST #

Hi,
This is really a nice post and helped me "help" my sys-admins to increase the parameter value.

But, after many days after that activity, today when I checked the output of

--- > prctl -n project.max-shm-memory -i project 100

project: 100: user.oracle
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 20.0GB - deny -
privileged 60.0GB - deny -
system 16.0EB max deny -

Now it's not allowing me to increase oracle sga size greater than 20G.
Will using -x resolve this ??
Or any better idea how can I resolve this ??

Posted by Jitu Keshwani on January 23, 2011 at 07:03 AM PST #

impresive!!!!!
it helps for our problem, and went successfuly... Thank you!

Posted by esmael on January 24, 2011 at 08:46 PM PST #

One of the best article that ever been written on DBA stuff.

Posted by raj on February 09, 2011 at 04:01 AM PST #

This post helped to solve critical issue, great blog.
Thanks
Mohan

Posted by guest on July 12, 2011 at 10:30 AM PDT #

hi,i met the same problem as u. My OS is solaris 10, and my memory size is 16GB. While i try to install the Oracle 10g, i got a out of memory error. And i have set the max-shm-memory to 12GB. Still get the same problem

Posted by guest on August 29, 2011 at 02:03 PM PDT #

Its directly related to the memory size allocated on the configuration panel. It defaults to a very very large size (27GB in my case). Rather than change my system config using this article, I lowered the size allocation on the dbca config panel to a more reasonable size. Either way works.

Posted by guest on September 29, 2011 at 05:08 PM PDT #

Is this a valid setting where privileged is larger than system setting.

project.max-shm-memory
privileged 30.0GB - deny -
system 16.0EB max deny -

Posted by guest on October 14, 2011 at 05:01 AM PDT #

Hi,

we are facing out of memory issue while cold backup restoring on sparc solaris9 systems, which we moved from sparc hardware to another similar hardware.

even after we set below section:
On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system to pick up the new value.
set shminfo_shmmax = 0x000000028a006000

SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument

Posted by surya on March 24, 2012 at 10:46 AM PDT #

Thanks blog owner, for sharing this great article with us, really I needed this information for my college homework, thank you again

Posted by guillermo on May 25, 2012 at 03:54 PM PDT #

Gooooooooooood job, thanx a lot!

Posted by guest on July 27, 2012 at 05:47 AM PDT #

Hi,
I had the following error while startup databases
"ORA-27102: out of memory"
I ran the following command and all the databases (7) are up and running now.
"prctl -n project.max-shm-memory -r -v 10G -i project 3"
BUT...
How can I make this change permanent (after the reboot the sun server)?
Thanks in advance
-Joe

Posted by guest on October 12, 2012 at 07:16 AM PDT #

Joe: please check the rest of the blog entry for instructions to make that change permanent. Search for "projadd" command in the same page.

Posted by Giri Mandalika on October 12, 2012 at 11:07 AM PDT #

On our Solaris 10 server, it complains that project id already exists:

projadd: Invalid projid "3": must be >= 100
projadd: Duplicate projid "3"

Anyone face this issue ?

Posted by guest on March 22, 2013 at 02:24 AM PDT #

I faced the same issue regarding the project ID. I just set it to 100 and it worked fine.

Posted by guest on April 01, 2013 at 12:06 AM PDT #

Yes, actually, that's a flaw in the article- project ID number 3 is the pre-existing "default" project- you can't do a "projadd -p 3" unless you delete the default project first.

There are two workarounds for this-
You may create a new project ID number (the -p parameter) using an unused project id number 100 or greater (look at the second field in /etc/project for the in-use projids).

OR

Instead of using projadd, just edit the /etc/project file and change the "default" project manually to something like this:

default:3:oracle test:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)

If you change /etc/project, you may (or may not) need to reboot or restart one of the services that I cannot remember off the top of my head in order to instantiate your change.
------------------
On our Solaris 10 server, it complains that project id already exists:

projadd: Invalid projid "3": must be >= 100
projadd: Duplicate projid "3"

Anyone face this issue ?
-------------------

Posted by guest on April 05, 2013 at 07:05 PM PDT #

Thanks, It's good

Posted by guest on March 24, 2014 at 09:58 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Benchmark announcements, HOW-TOs, Tips and Troubleshooting

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today