Have you noticed that installing an Oracle database on a scale-up server might require some adjustment?
Adding sockets and cores in a box and let Oracle run is not enough. No, we know that, some specific settings are needed.
For instance, one may meet the following error when creating a new database with "dbca" (or without dbca by the way).
"No space left on device", this is a quite misleading error. Actually, the root cause is the semaphores setting as already documented in some other articles. But what is behind?
Actually, this is a consequence of the "processes" initialization parameter value. The "processes" default value is derived, and it typically depends on the number of cores reported in the alert
log.
Basically, in a Linux x86_64 server, Oracle 12c will set processes to "logical processes x 80".So in a server having 8 sockets with 28 core per socket and the hyper-threading enabled, one will get:
[root@server ~]# lscpu
Architecture: x86_64
CPU
op-mode(s): 32-bit, 64-bit
Byte
Order: Little Endian
CPU(s): 448
On-line
CPU(s) list: 0-447
Thread(s)
per core: 2
Core(s) per
socket: 28
Socket(s): 8
NUMA
node(s): 8
448 threads x 80 = 35840
In dbca, this value was suggested in the "sizing" tab of the configuration options.
How to fix this?
1/ Well, the first option is just to decrease the number of processes. Do you really need that much processes?
Quick and dirty.
2/ Adjust the semaphore parameter.
Some background information first.
Semaphores are a system resource that Oracle utilizes for interprocess communication and they occupy a relatively small memory space.
Since Oracle 10g, it is admitted that The number of semaphores required by an instance is equal to 2 times the setting of the 'processes' parameter. Keep in mind, however, that Oracle only momentarily grabs 2 X 'processes' then releases half at instance startup.
Applying this rule, the Oracle instance will require more than 70,000 semaphores at a point in time.
Following the Oracle recommendation, the values in the /etc/sysctl.conf should look like:
[root@server ~]# cat /proc/sys/kernel/sem
250 32000
100 256
These values represent SEMMSL, SEMMNS, SEMOPM, and SEMMNI.Because of the semaphore structure, the maximum number of semaphores that can be allocated on a system will be lesser of: (semmsl * semmni) or semmns.
Thus, both semmns and (semmsl * semmni) need to be improved to a value higher than 35,840 x 2. So let's say 80,000.
The experience shows that one need to carefully balance the values between semmsl and semmni. A quick reminder first:
- semmsl: maximum number of semaphore per array
- semmni: maximum arrays
In the example above, the system parameter was updated as per below:
[root@server ~]# sysctl -w "kernel.sem=250 80000 200 320"
kernel.sem = 250 80000 200 320
The new setup can be double-checked using ipcs or looking at the /proc/sys value
[root@server ~]# cat /proc/sys/kernel/sem
250 80000
200 320
[root@server ~]# ipcs -la
------
Messages Limits --------
max queues
system wide = 32768
max size of
message (bytes) = 8192
default max
size of queue (bytes) = 16384
------
Shared Memory Limits --------
max number
of segments = 4096
max seg size
(kbytes) = 2964122308
max total
shared memory (kbytes) = 3717247308
min seg size
(bytes) = 1
------
Semaphore Limits --------
max number
of arrays = 320
max
semaphores per array = 256
max
semaphores system wide = 80000
max ops per
semop call = 200
semaphore
max value = 32767
Once updated, the database was created successfully.
more on this topic in My Oracle Support:
Semaphores and Shared Memory - An Overview (Doc ID 153961.1)
