Oracle Processes and System Semaphore

Have you noticed that installing an Oracle database on a scale-up server might require some adjustment?

Adding sockets and cores in a box and let Oracle run is not enough. No, we know that, some specific settings are needed.

For instance, one may meet the following error when creating a new database with "dbca" (or without dbca by the way).

"No space left on device", this is a quite misleading error. Actually, the root cause is the semaphores setting as already documented in some other articles. But what is behind?

Actually, this is a consequence of the "processes" initialization parameter value. The "processes" default value is derived, and it typically depends on the number of cores reported in the alert log.

Basically, in a Linux x86_64 server, Oracle 12c will set processes to "logical processes x 80".

So in a server having 8 sockets with 28 core per socket and the hyper-threading enabled, one will get:

[root@server ~]# lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                448

On-line CPU(s) list:   0-447

Thread(s) per core:    2

Core(s) per socket:    28

Socket(s):             8

NUMA node(s):          8

448 threads x 80 = 35840

In dbca, this value was suggested in the "sizing" tab of the configuration options.

How to fix this?

1/ Well, the first option is just to decrease the number of processes. Do you really need that much processes? 

Quick and dirty.

2/ Adjust the semaphore parameter.

Some background information first. 

Semaphores are a system resource that Oracle utilizes for interprocess communication and they occupy a relatively small memory space. 

Since Oracle 10g, it is admitted that The number of semaphores required by an instance is equal to 2 times the setting of the 'processes' parameter. Keep in mind, however, that Oracle only momentarily grabs 2 X 'processes' then releases half at instance startup.

Applying this rule, the Oracle instance will require more than 70,000 semaphores at a point in time.

Following the Oracle recommendation, the values in the /etc/sysctl.conf should look like:

[root@server ~]# cat /proc/sys/kernel/sem

250     32000   100     256

These values represent SEMMSL, SEMMNS, SEMOPM, and SEMMNI.

Because of the semaphore structure, the maximum number of semaphores that can be allocated on a system will be lesser of: (semmsl * semmni) or semmns.
Thus, both semmns and (semmsl * semmni) need to be improved to a value higher than 35,840 x 2. So let's say 80,000.
The experience shows that one need to carefully balance the values between semmsl and semmni. A quick reminder first:

• semmsl: maximum number of semaphore per array
• semmni: maximum arrays

In the past we have seen some contention in environment having a limited number of array set. The recommendation will be then to increase the semmni value while semmsl remain unchanged.

In the example above, the system parameter was updated as per below: 

[root@server ~]# sysctl -w "kernel.sem=250 80000 200 320"

kernel.sem = 250 80000 200 320

The new setup can be double-checked using ipcs or looking at the /proc/sys value

[root@server ~]# cat /proc/sys/kernel/sem

250     80000   200     320

[root@server ~]# ipcs -la

------ Messages Limits --------

max queues system wide = 32768

max size of message (bytes) = 8192

default max size of queue (bytes) = 16384

------ Shared Memory Limits --------

max number of segments = 4096

max seg size (kbytes) = 2964122308

max total shared memory (kbytes) = 3717247308

min seg size (bytes) = 1

------ Semaphore Limits --------

max number of arrays = 320

max semaphores per array = 256

max semaphores system wide = 80000

max ops per semop call = 200

semaphore max value = 32767

Once updated, the database was created successfully.

more on this topic in My Oracle Support:

Semaphores and Shared Memory - An Overview (Doc ID 153961.1)