• Categories
  • Search
Thursday, July 9, 2015

Multi-CPU bindings for Solaris Project

Traditionally, assigning specific processes to a certain set of CPUs has been
done by using processor sets (and resource pools). This is quite useful, but it
requires the hard partitioning of processors in the system. That means, we can't
restrict process A to run on CPUs 1,2,3 and process B to run on CPUs 3,4,5,
because these partitions overlap.

There is another way to assign CPUs to processes, which is called processor affinity, or Multi-CPU binding (MCB for short). Oracle Solaris 11.2 introduced MCB binding, as described in pbind(1M) and processor_affinity(2). With the release of Oracle Solaris 11.3, we have a new interface to assign/modify/remove MCBs, via Solaris project.

Briefly, a Solaris project is a collection of processes with predefined attributes. These attributes include various resource controls one can apply to processes that belong to the project. For more details, see projects(4) and resource_controls(5). What's new is that MCB becomes simply another resource control we can manage through Solaris projects.

We start by making a new project with MCB property. We assume that we have
enough privilege to create a project and there's no project called test-project
in the system, and all CPUs described by project.mcb.cpus entry exist in the
system and online. We also assume that the listed cpus are in the resource pool
to which current zone is bound. For manipulating project, we use standard
command line tools projadd(1M)/projdel(1M)/projmod(1M).

root@sqkx4450-1:~# projects -l test-project
projects: project "test-project" does not exist
root@sqkx4450-1:~# projadd -K project.mcb.cpus=0,3-5,9-11 -K project.mcb.flags=weak -K project.pool=pool_default test-project
root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.cpus=0,3-5,9-11
project.mcb.flags=weak
project.pool=pool_default

This means that processes in test-project will be weakly bound to CPUs 0,3,4,5,
9,10,11. (Note: For the concept of strong/weak binding, see
processor_affinity(2). In short, strong binding guarantees that processes will
run ONLY on designated CPUs, while weak binding does not have such a guarantee.)

Next thing is to assign some processes to test-project. If we know PIDs of
target processes, it can be done by newtask(1).

root@sqkx4450-1:~# newtask -c 4156 -p test-project
root@sqkx4450-1:~# newtask -c 4170 -p test-project
root@sqkx4450-1:~# newtask -c 4184 -p test-project

Let's check the result by using the following command.

root@sqkx4450-1:~# pbind -q -i projid 100
pbind(1M): pid 4156 weakly bound to processor(s) 0 3 4 5 9 10 11.
pbind(1M): pid 4170 weakly bound to processor(s) 0 3 4 5 9 10 11.
pbind(1M): pid 4184 weakly bound to processor(s) 0 3 4 5 9 10 11.

Good. Now suppose we want to change the binding type to strong binding. In that case, all we need to do is change the value of project.mcb.flags to "strong", or even delete the project.mcb.flag key, because the default value is set to "strong".

root@sqkx4450-1:~# projmod -s -K project.mcb.flags=strong test-project
root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.cpus=0,3-5,9-11
project.mcb.flags=strong
project.pool=pool_default

Things look good, but...

root@sqkx4450-1:~# pbind -q -i projid 100
pbind(1M): pid 4156 weakly bound to processor(s) 0 3 4 5 9 10 11.
pbind(1M): pid 4170 weakly bound to processor(s) 0 3 4 5 9 10 11.
pbind(1M): pid 4184 weakly bound to processor(s) 0 3 4 5 9 10 11.

Nothing changed actually! WARNING: By default, projmod(1M) only modifies project configuration file, but do not attempt to apply it to its processes. To do that, use the "-A" option.

root@sqkx4450-1:~# projmod -A test-project
root@sqkx4450-1:~# pbind -q -i projid 100
pbind(1M): pid 4156 strongly bound to processor(s) 0 3 4 5 9 10 11.
pbind(1M): pid 4170 strongly bound to processor(s) 0 3 4 5 9 10 11.
pbind(1M): pid 4184 strongly bound to processor(s) 0 3 4 5 9 10 11.

Now, suppose we want to change the list of CPUs, but oops, we made some typos.

root@sqkx4450-1:~# projmod -s -K project.mcb.cpus=0,3-5,13-17 -A test-project
projmod: Updating project test-project succeeded with following warning message.
WARNING: Following ids of cpus are not found in the system:16-17
root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.cpus=0,3-5,13-17
project.mcb.flags=strong
project.pool=pool_default

Our system has CPUs 0 to 15, not up to 17. In that case, we get some warnings. But the command succeeded anyway. The command simply ignores missing CPUs.

root@sqkx4450-1:~# pbind -q -i projid 100
pbind(1M): pid 4156 strongly bound to processor(s) 0 3 4 5 13 14 15.
pbind(1M): pid 4170 strongly bound to processor(s) 0 3 4 5 13 14 15.
pbind(1M): pid 4184 strongly bound to processor(s) 0 3 4 5 13 14 15.

And one more thing: If you want to check the validity of project file only, use projmod(1M) without any options.

root@sqkx4450-1:~# projmod
projmod: Validation warning on line 6, WARNING: Following ids of cpus are not found in the system:16-17

But projmod is not tolerant if it can't find any CPUs at all.

root@sqkx4450-1:~# projmod -s -K project.mcb.cpus=17-20 -A test-project
projmod: WARNING: Following ids of cpus are not found in the system:17-20
projmod: ERROR: All of given multi-CPU binding (MCB) ids are not found in the system: project.mcb.cpus=17-20
root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.cpus=0,3-5,13-17
project.mcb.flags=strong
project.pool=pool_default

Now we see ERROR. It's something that actually fails the command. Please read the error message carefully when you see it. Note that project configuration is not updated also.

Before moving to next topic, one small but important tip. How do we clear MCB
from a project? Set the value of project.mcb.cpus to "none" and remove
project.mcb.flags if there is.

root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.cpus=none
project.pool=pool_default
root@sqkx4450-1:~# projmod -A test-project
root@sqkx4450-1:~# pbind -q -i projid 100
root@sqkx4450-1:~#

Let's move on to a little bit of advanced usage. In Oracle Solaris systems, as
well as other systems, CPUs are grouped in certain units. Currently there are
'cores', 'sockets', 'processor-groups' and 'lgroups'. Utilizing these units can
improve performance aided by hardware design. (I have less familiarity with
those topics, so look at the following post about lgroups: Locality Group Observability on Solaris.) MCB for projects supports all of these CPU structures. The usage is simple. Just change "project.mcb.cpus" to "project.mcb.cores", "project.mcb.sockets", "project.mcb.pgs", or "project.mcb.lgroups".

Note: To get information about CPU structures on a given system, use following commands. "psrinfo -t" gives information about cpu/core/socket structure, "pginfo" gives information about processor groups, and "lgrpinfo -c" gives information about lgroups.

root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.sockets=1
project.pool=pool_default
root@sqkx4450-1:~# projmod -A test-project
root@sqkx4450-1:~# pbind -q -i projid 100
pbind(1M): pid 4156 strongly bound to processor(s) 1 5 9 13.
pbind(1M): pid 4170 strongly bound to processor(s) 1 5 9 13.
pbind(1M): pid 4184 strongly bound to processor(s) 1 5 9 13.

These examples explain the basics of MCB for projects. For more details, you can refer to the appropriate man pages. But, let me briefly summarize some features we didn't explain here. And, final warning: Many features we used in this post are not supported on Oracle Solaris 11.2, even those not related to MCB directly.

1. newtask(1) also utilizes projects. When we set MCB for a project in the
project configuration file, an unprivileged user that is a member of project can
use newtask(1) to put new or his/her existing processes in it.

2. For Solaris projects APIs, look at libproject(3LIB). Warning: some features work only for 64-bit version of the library for now.

3. There are many other existing attributes of project. Combining them with MCB usually causes no problems, but there is one exception: project.pool. Ignoring all the detail, there's only one important guideline when using both project.pool and project.mcb.(cpus|cores|sockets|pgs|lgroups): all the CPUs in project.mcb.(cpus|cores|sockets|pgs|lgroups) should reside in the project.pool.

When we don't specifiy project.pool and use project.mcb.(cpus|cores|sockets|pgs|lgroups), the system ASSUMES that project.pool is the default pool of the current zone. In this case, when we try to apply the project's attributes to processes, we'll see following warning message.

root@sqkx4450-1:~# projects -l test-project
test-project
projid : 100
comment: ""
users : (none)
groups : (none)
attribs: project.mcb.cpus=0,3-5,9-11
project.mcb.flags=weak
root@sqkx4450-1:~# projmod -A test-project
projmod: Updating project test-project succeeded with following warning message.
WARNING: We bind the target project to the default pool of the zone because an multi-CPU binding (MCB) entry exists.

Man page references.

    General information:

        Project file configuration: project(4)

        How to manage resource control by project: resource_controls(5)

    Project utilization:

        Get information of projects: projects(1)

        Manage projects: projadd(1M) / projdel(1M) / projmod(1M)

        Assign a process to project: newtask(1)

        project control APIs: libproject(3LIB)

    Existing interfaces dealing MCB:

        command line interface: pbind(1M)

        system call interface: processor_affinity(2)

    Processor information:

        psrinfo(1M) / pginfo(1M) / lgrpinfo(1M)

Join the discussion

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
 

Visit the Oracle Blog

 

Contact Us

Oracle

Integrated Cloud Applications & Platform Services