QoS Relative Round Trip Timeout

QoS Relative Round Trip Timeout

[[Other Connection Tuning|Back]]

Have you ever seen a Client method invocation in your application being blocked for an unreasonably long time waiting for it's reply, especially in an untuned, highly stressed or unstable system? The reply may finally arrive after some delay, but if you are running a time-sensitive application (such as a telecommunication system), then any delay in response that exceeds a certain tolerance level may lead to some undesirable consequences.

Is it possible to specify this tolerance level in VisiBroker? For example, if a Client method invocation does not return within a specified time, then an exception should be thrown. The application logic should handle this exception by notifying the user or system, and perform the necessary remedial actions.

If this behavior is exactly what you want, then setting the Quality of Service (QoS) Relative Round Trip Timeout Policy at the Client side can solve your problem.

The "echo_service_cpp" [[Explanation of Example|example]] is used in this demonstration.

Scenario

  • The Client creates 15 threads to call 1 Server concurrently.
  • Each client thread calls the Server once.
  • Each invocation blocks for 10 seconds at the Server side.
  • Set Server Engine "threadMax=10" to simulate slow/busy Server.
  • Monitor the Client's resource consumption and invocation performance with the default configuration.
  • Observe the effect of setting "vbroker.qos.defaultRRTTimeout=12000" (i.e. tolerance is only 12 sec) at the Client side.

How long do you think each thread will take to return from the invocation?

Preparation

Configure the Client by modifying c.sh:

  • Set the following properties:
    • server_sleep_time 10
    • vbroker.agent.enableLocator=false
  • Disable all the other properties.

Configure the Server by modifying s.sh:

  • Set the following properties:
    • vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=10
    • vbroker.agent.enableLocator=false
    • vbroker.se.default.local.manager.enabled=false

  • Disable all the other properties.

Execution

Make sure you have set up the necessary VisiBroker environment and build the example before running this demonstration.

  • Start the Client Monitor script:
    • mon_c.sh
  • Start the Server:
    • s.sh 1
  • Start the Client:
    • c.sh 1 1 15
  • Monitor the Client's resource consumption (printed by Client Monitor). E.g.:

MEMORY(KB) THREADS SOCKETS
12192      16      1

  • Take note of the time taken for each invocation to return (printed by Client). E.g.:

. . . . . .
Reply of Client Thread Id P14295_T7_S1 is P14295_T7_S1, time taken is 10.0179
Reply of Client Thread Id P14295_T6_S1 is P14295_T6_S1, time taken is 10.0183
Reply of Client Thread Id P14295_T13_S1 is P14295_T13_S1, time taken is 20.023
Reply of Client Thread Id P14295_T12_S1 is P14295_T12_S1, time taken is 20.0234
. . . . . .

  • Stop the Client and Server.
  • Set the following property at the Client side by modifying c.sh:

vbroker.qos.defaultRRTTimeout=12000

  • Re-start the Server:
    • s.sh 1
  • Re-start the Client:
    • c.sh 1 1 15
  • Monitor the Client's resource consumption (printed by Client Monitor). E.g.:

MEMORY(KB) THREADS SOCKETS
12200      16      1

  • Take note of the time taken for each invocation to return (printed by Client). E.g.:

. . . . . .
Reply of Client Thread Id P15753_T10_S1 is P15753_T10_S1, time taken is 10.0173
Reply of Client Thread Id P15753_T6_S1 is P15753_T6_S1, time taken is 10.0175
Client Thread Id P15753_T11_S1 caught Exception: CORBA::TIMEOUT
Minor: 1447165955
Completion Status: NO
, time taken is 12.0125
Client Thread Id P15753_T12_S1 caught Exception: CORBA::TIMEOUT
Minor: 1447165955
Completion Status: NO
, time taken is 12.0131
. . . . . .

Observations

Compare the time taken by each invocation before and after tuning the property “vbroker.qos.defaultRRTTimeout=12000” at the Client side.

No. invocations returned within 12 secs No. invocations returned after 12 secs
Before Tuning 10 5
After Tuning 15  (5 TIMEOUT exceptions) 0

Key observations after tuning:

  • All the 15 invocations return within 12 seconds (i.e. the tolerance level set by the client application) compared to only 10 invocations before tuning.
  • 5 out of the 15 invocations which return within 12 seconds throw TIMEOUT exceptions.

Explanation

In this demonstration, each Servant method implementation takes about 10 seconds to complete the work. But the Client application's QoS requirement expects every invocation to return within 12 seconds (i.e. with 2 seconds buffer). To simulate a busy/slow Server, we intentionally set the Server Engine property "threadMax=10" so that the Server can only service 10 requests concurrently, and the subsequent 5 requests are serviced later. Consequently, not all the 15 concurrent requests received by the Server can be serviced within 10 seconds (time taken to complete work). That is the reason why before setting the QoS Relative Round Trip Timeout Policy, 5 out of the 15 Client invocations return after 20 seconds, which does not meet the Client application QoS requirement of 12 seconds.

To meet the application QoS requirement, we set the property "vbroker.qos.defaultRRTTimeout=12000" at the VBC++ Client side. The timing specified is the round trip time for the invocation, i.e. total time taken to send the Request Message and receive the Reply Message. The unit of measurement is milliseconds. If the invocation is not completed within 12 seconds, a TIMEOUT exception is thrown. With this timeout setting, 5 out of the 15 Client invocations return by throwing TIMEOUT exceptions within 12 seconds, which meet the application QoS requirement now. The Client application implementation logic should handle the TIMEOUT exception by notifying the user or system about the timeout event, and then perform the necessary remedial actions. The exception must be caught at the application level, otherwise the Client application will crash.

The default value of the "defaultRRTTimeout" property is 0, which allows all the method invocations in the ORB to take unlimited amount of time to return. The behavior you have observed (i.e. some invocations return after 20 seconds) before tuning is due to this setting.

If you analyse the Server side logs carefully, you should notice that the Server continues to service the 5 requests that have already timeouted at the Client side. However, their corresponding Reply Messages received by the Client side ORB are discarded internally. You must be aware of this behavior when designing your application so that the timeout scenarios are handled properly.

Sometimes, despite your best effort to design and tune the whole system properly, method invocation delays can still happen in a live production environment due to unexpected high load, network disruption or congestion, undiscovered bugs, etc. So it is a good practice to set timeout policy to unblock hanging Client invocations. The suitable timeout value depends on your application's QoS requirement and the worst case scenario which your system can tolerate when the method invocation does not return on time.

The timeout specified with this property sets the RelativeRoundTripTimeoutPolicy at the ORB Level policy. This means that all the invocations in the same ORB have the same round trip timeout. If some invocations in your Client application need different round trip timeout values, you must programmatically set the RelativeRoundTripTimeoutPolicy at the Object or Thread Level policy using the appropriate CORBA APIs. The Object Level timeout setting has the highest precedence, followed by the Thread Level, the ORB Level and  finally the property level setting has the lowest precedence.

The equivalent property for VBJ is "vbroker.orb.qos.relativeRTT", which has the same behavior as VBC++.

[[Other Connection Tuning|Back]]  |  [[QoS tcpTimeout|Next]]

Labels (1)

DISCLAIMER:

Some content on Community Tips & Information pages is not officially supported by Micro Focus. Please refer to our Terms of Use for more detail.
Top Contributors
Version history
Revision #:
2 of 2
Last update:
‎2020-03-13 21:05
Updated by:
 
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.