Have you noticed that installing an Oracle database on a scale-up server might require some adjustment?
Adding sockets and cores in a box and let Oracle run is not enough. No, we know that, some specific settings are needed.
For instance, one may meet the following error when creating a new database with "dbca" (or without dbca by the way).
"No space left on device", this is a quite misleading error. Actually, the root cause is the semaphores setting as already documented in some other articles. But what is behind?
Actually, this is a consequence of the "processes" initialization parameter value. The "processes" default value is derived, and it typically depends on the number of cores reported in the alert
log.
Basically, in a Linux x86_64 server, Oracle 12c will set processes to "logical processes x 80".
So in a server having 8 sockets with 28 core per socket and the hyper-threading enabled, one will get:
[root@server ~]# lscpu
Architecture: x86_64
CPU
op-mode(s): 32-bit, 64-bit
Byte
Order: Little Endian
CPU(s): 448
On-line
CPU(s) list: 0-447
Thread(s)
per core: 2
Core(s) per
socket: 28
Socket(s): 8
NUMA
node(s): 8
448 threads x 80 = 35840
In dbca, this value was suggested in the "sizing" tab of the configuration options.
How to fix this?
1/ Well, the first option is just to decrease the number of processes. Do you really need that much processes?
Quick and dirty.
2/ Adjust the semaphore parameter.
Some background information first.
Semaphores are a system resource that Oracle utilizes for interprocess communication and they occupy a relatively small memory space.
Since Oracle 10g, it is admitted that The number of semaphores required by an instance is equal to 2 times the setting of the 'processes' parameter. Keep in mind, however, that Oracle only momentarily grabs 2 X 'processes' then releases half at instance startup.
Applying this rule, the Oracle instance will require more than 70,000 semaphores at a point in time.
Following the Oracle recommendation, the values in the /etc/sysctl.conf should look like:
[root@server ~]# cat /proc/sys/kernel/sem
250 32000
100 256
These values represent SEMMSL, SEMMNS, SEMOPM, and SEMMNI.
Because of the semaphore structure, the maximum number of semaphores that can be allocated on a system will be lesser of: (semmsl * semmni) or semmns.
Thus, both semmns and (semmsl * semmni) need to be improved to a value higher than 35,840 x 2. So let's say 80,000.
The experience shows that one need to carefully balance the values between semmsl and semmni. A quick reminder first:
- semmsl: maximum number of semaphore per array
- semmni: maximum arrays
In the past we have seen some contention in environment having a limited number of array set. The recommendation will be then to increase the semmni value while semmsl remain unchanged.
In the example above, the system parameter was updated as per below:
[root@server ~]# sysctl -w "kernel.sem=250 80000 200 320"
kernel.sem = 250 80000 200 320
The new setup can be double-checked using ipcs or looking at the /proc/sys value
[root@server ~]# cat /proc/sys/kernel/sem
250 80000
200 320
[root@server ~]# ipcs -la
------
Messages Limits --------
max queues
system wide = 32768
max size of
message (bytes) = 8192
default max
size of queue (bytes) = 16384
------
Shared Memory Limits --------
max number
of segments = 4096
max seg size
(kbytes) = 2964122308
max total
shared memory (kbytes) = 3717247308
min seg size
(bytes) = 1
------
Semaphore Limits --------
max number
of arrays = 320
max
semaphores per array = 256
max
semaphores system wide = 80000
max ops per
semop call = 200
semaphore
max value = 32767
Once updated, the database was created successfully.
more on this topic in My Oracle Support:
Semaphores and Shared Memory - An Overview (Doc ID 153961.1)
