Summary

Orbix 6.3 Java server hangs when used up threads in the thread-pool

Environment

Orbix 6.3

All supported platforms

Question/Problem Description

The Java version of Orbix uses threads in the thread-pool not just to service incoming requests
but also for Orbix-internal purposes, such as reading incoming messages from socket connections.
This means that if you put a limit on the size of the thread pool in an Orbix Java server then the server may hang under some circumstances.
This is because it may use up all the threads in the thread pool for monitoring socket connections and not have any threads left over to service
incoming requests or monitor a newly opened socket connection.


Cause

There are three variables  which related to Active Connection Manager(ACM 😞
1 - plugins:iiop:pool:java_max_threads
The variable specifies the maximum number of threads reserved from the WorkQueue to support tasks working on behalf of the Java ATLI transport.
This variable defaults to 512. The number of threads in IIOP plugin does not exceeds this value.

2 - plugins:iiop:incoming_connections:hard_limit
Specifies maximum number of TCP connections between server and client. This variable defaults to -1 (disabled).
When threads in the thread-pool, set in (1) are used up, no new connections will be accepted. New requests through existing connections will continue to be processed.

3 - plugins:iiop:incoming_connections:soft_limit
Specifies number of connections to start disconnect oldest connection.This variable defaults to -1 (disabled).
This makes Orbix server to accept a new connection within this limit.

The cause is java_max_threads defaults 512, and hard_limit and soft_limit are disabled by default.

Resolution

Setting plugins:iiop:pool:java_max_threads to -1(infinite) will resolve the issue, or enable ACM.
To enable ACM, set each value as:

(1) > (2) > (3)

The (2) should be less than (1) so the server can accept new connection before reaching (1).
The (3) should be less than (2)
ACM keeps maxmum number of connection within (3) so server is able to accept new connections.

Support Incidents

2856492, 2864238

Notes

thread_pool:high_water_mark specifies the maximum number of threads in the pool. If the java_max_threads is set to larger than the high_water_mark, the number of threads in the pool
will never exceed the high_water_mark's value.