PeopleSoft on Solaris 10: Fixing the "msgget: No space left on device" Error
By Giri Mandalika on Nov 30, 2008
When a large number of application server processes are configured in a single PeopleSoft domain or in multiple domains cumulative, it is very likely that the PeopleSoft application server domain boot process may fail with errors like:
Booting server processes ... exec PSSAMSRV -A -- -C psappsrv.cfg -D CS90SPV -S PSSAMSRV : Failed. 113954.ben15!PSSAMSRV.29746.1.0: LIBTUX_CAT:681: ERROR: Failure to create message queue 113954.ben15!PSSAMSRV.29746.1.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : msgget: No space left on device 113954.ben15!tmboot.29708.1.-2: CMDTUX_CAT:825: ERROR: Process PSSAMSRV at ben15 failed with /T tperrno (TPEOS - operating system error)
In this particular example, the PeopleSoft Enterprise is running on a Solaris 10 system. Fortunately the error message is very clear in this case; and the failure is related to the message queues. During the domain boot up process, there is a call to
msgget() to create a message queue. If the call to
msgget() succeeds, it returns a non-negative integer that serves as the identifier for the newly created message queue. However in the case of a failure, it returns -1 and sets the error number to EACCES, EEXIST, ENOENT or ENOSPC depending on the underlying reason.
From the above error messages it clear that the
msgget() failed with the errno set to ENOSPC (No space left on device). Man page of
msgget(2) has the following explanation for ENOSPC error code on Solaris:
ERRORS The msgget() function will fail if: ... ... ENOSPC A message queue identifier is to be created but the system-imposed limit on the maximum number of allowed message queue identifiers system wide would be exceeded. See NOTES. NOTES ... ... The system-imposed limit on the number of message queue identifiers is maintained on a per-project basis using the project.max-msg-ids resource control.
It has enough clues to suspect the configured number for the message queue identifiers.
Prior to the release of Solaris 10, the /etc/system System V IPC tunable,
msgsys:msginfo_msgmni, was used to control the maximum number of message queues that can be created. The default value on pre-Solaris 10 systems is 50.
With the release of Solaris 10, majority of the System V IPC tunables were obsoleted and equivalent resource controls were created for the remaining tunables to reduce the administrative overhead. On Solaris 10 and later versions, System V IPC can be tuned on a per project basis using the newly introduced resource controls.
On any Solaris 10 system, the resource control,
project.max-msg-ids, replaced the old /etc/system tunable,
msginfo_msgmni. And the default value has been raised to 128.
Now back to the failure in PeopleSoft environment. Let's first check the current value configured for
- Get the project ID.
% id -p uid=222227(psft) gid=2294(dba) projid=3(default)
- Examine the
project.max-msg-idsresource control for the project with ID 3, using the
% prctl -n project.max-msg-ids -i project 3 project: 3: default NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT project.max-msg-ids privileged 128 - deny - system 16.8M max deny -
Alternatively run the command
ipcs -q to check the number of active message queues. Note that the project with id '3' is configured to create a maximum of 128 (default) message queues. In any case, the number of active message queues from the
ipcs -q output may almost match with the configured value for the
Since it appears the configured PeopleSoft domain(s) needs more than 128 message queues in order to bring up all the application server processes that constitute the PeopleSoft Enterprise, the solution is to increase the value for the resource control,
project.max-msg-ids, to any value beyond 128. For the sake of simplicity, let's increase it to 256 (2 \* default value, that is). Again
prctl utility can be used to set the new value for the resource control.
- Assume the privileges of the 'root' user
% su Password:
- Increase the maximum value for the message queue identifiers to 256 using the
# prctl -n project.max-msg-ids -r -v 256 -i project 3
- Verify the new maximum value for the message queue identifiers
# prctl -n project.max-msg-ids -i project 3 project: 3: default NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT project.max-msg-ids privileged 256 - deny - system 16.8M max deny -
With this change, the PeopleSoft Enterprise should boot up at least with no
Failure to create message queue .. msgget: No space left on device errors.
Before we conclude, note that the above mentioned solution is not persistent across multiple operating system reboots. To make it persistent, create a new project using the
projadd command. The man page for projadd(1M) has an example showing the creation of a project.