SM getting down at least once a day. Please suggest. Logs mentioned in Description

Hi Experts,

I am using SM 9.6 version. SM is getting down at least once a day and for yesterday, I got the following error in log file. No such details apart from this.

JRTE E sendSoapFaultResponse() failed - java.io.IOException: An established connection was aborted by the software in your host machine
JRTE E sendResponse() execute failed - java.io.IOException: An established connection was aborted by the software in your host machine
JRTE E sendResponse() failed!
JRTE E sendResponse() failed!
JRTE E sendResponse() execute failed - java.io.IOException: An established connection was aborted by the software in your host machine
JRTE E sendResponse()
java.lang.Exception: WorkerThead died unexpectedly. 3128

 

Please let me know your valuable inputs regarding the same.

  • I have some problem. Were you able to solve it?

    5300( 19792) 10/05/2020 10:10:29 JRTE W Send error response: java.io.IOException: An established connection was aborted by the software in your host machine
    5300( 19792) 10/05/2020 10:10:29 JRTE E sendSoapFaultResponse() failed - java.io.IOException: An established connection was aborted by the software in your host machine
    5300( 19792) 10/05/2020 10:10:29 JRTE E sendResponse() execute failed - java.io.IOException: An established connection was aborted by the software in your host machine
    5300( 6772) 10/05/2020 10:10:29 JRTE I Termination signal: 0
    5300( 6772) 10/05/2020 10:10:29 RTE I -Memory : S(63747752) O(51058900) MAX(117432844) - MALLOC's Total(7979746)
    5300( 6772) 10/05/2020 10:10:29 RTE I Thread termination in progress
    5300( 6772) 10/05/2020 10:10:29 RTE A SUMMARY-1 The following event has been reported 1 times in the last 11 minute(s)
    5300( 6772) 10/05/2020 10:10:29 RTE A Performance-7-$G.timezone.java.ids, Глобальный список $G.timezone.java.ids содержит слишком много элементов! число=600
    5300( 6772) 10/05/2020 10:10:29 RTE A SUMMARY-1 The following event has been reported 1 times in the last 11 minute(s)
    5300( 6772) 10/05/2020 10:10:29 RTE A Performance-7-$G.requestModels.local, Глобальный список $G.requestModels.local содержит слишком много элементов! число=1449
    5300( 6772) 10/05/2020 10:10:29 RTE A SUMMARY-1 The following event has been reported 1 times in the last 11 minute(s)
    5300( 6772) 10/05/2020 10:10:29 RTE A Performance-7-$G.files, Глобальный список $G.files содержит слишком много элементов! число=904
    5300( 6772) 10/05/2020 10:10:29 RTE A SUMMARY-1 The following event has been reported 1 times in the last 11 minute(s)
    5300( 6772) 10/05/2020 10:10:29 RTE A Performance-7-$G.tables.dsp, Глобальный список $G.tables.dsp содержит слишком много элементов! число=750
    5300( 6772) 10/05/2020 10:10:29 RTE A SUMMARY-1 The following event has been reported 1 times in the last 11 minute(s)
    5300( 6772) 10/05/2020 10:10:29 RTE A Performance-7-$G.approvalDef.local, Глобальный список $G.approvalDef.local содержит слишком много элементов! число=619
    5300( 6772) 10/05/2020 10:10:31 RTE I User 8_Yagovdik_383 has logged out and freed a Limited license ( 337 out of unlimited )
    5300( 6772) 10/05/2020 10:10:31 RTE I sql total 333.605 call#:865430 -->sqociDisconnect:OCISessionEnd
    5300( 6772) 10/05/2020 10:10:31 RTE I sql total 333.605 call#:865431 -->sqociDisconnect:OCIServerDetach
    5300( 6772) 10/05/2020 10:10:31 RTE I Thread termination cleanup complete. Thread was active for 0 days,00:11:40. Threads remaining: 32 of 50
    5300( 5252) 10/05/2020 10:10:32 JRTE E sendResponse() stop failed - WorkerThead died unexpectedly. 6772
    5300( 5252) 10/05/2020 10:10:32 JRTE E sendReponse()
    java.lang.Exception: WorkerThead died unexpectedly. 6772
  • Hi,

    Your logs don't suggest anything, just the symptoms but kindly advice you to try RTE upgrade  to latest 9.64 patch (just binary not application patch) and also to check sm servlet memory and shared memory usage with following commands.

    sm -reportlbstatus:1 (Shows sm servlet memory usages on all SM application servers)

    sm -reportshm (Shows shared memory usage which can be configured in sm.ini file)

     

  • May be not the right issue but I think there is a job that refreshes or
    manages global lists daily. If global lists are really out of bounds or
    corrupted may be some issue there.


  • We monitor memory usage and it seems that it does not exceed the norm. The screenshot shows the memory allocation when the node hangs.

    Application Server version is 9.64. RAD version - 9.64.

  • I also noticed that there are errors in the logs at the time the nodes hang. OutOfMemory:

    3604( 19332) 10/07/2020 09:26:45 RTE E Error extending record list to hold 524288 entries (old size was 524288)
    3604( 1868) 10/07/2020 09:26:55 RTE W Exception occurred for method execute and XML request <?xml version="1.0" encoding="utf-8"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><execute><thread>1</thread><event>2000</event><transaction>3</transaction><type>detail</type><formname>wizard-newApprovalsForm</formname><clientWidth>1161</clientWidth><modelChanges formid="wizard-newApprovalsForm4183typecheck10"><multiselect/></modelChanges></execute></SOAP-ENV:Body></SOAP-ENV:Envelope>
    3604( 1868) 10/07/2020 09:26:55 RTE E CTopaz::process(): Caught OutOfMemoryException code:266, msg:OutOfMemory
    3604( 1868) 10/07/2020 09:26:55 RTE E Caught XML API exception scxmlapi(23) - Получено исключение XML DOM – код 266 сообщения OutOfMemory.
    3604( 9236) 10/07/2020 09:26:55 JRTE W Send error response: A CXmlApiException was raised in native code : error 23 : scxmlapi(23) - Получено исключение XML DOM – код 266 сообщения OutOfMemory.
    3604( 1868) 10/07/2020 09:26:55 JRTE I Termination signal: 0
    3604( 1868) 10/07/2020 09:26:55 RTE I -Memory : S(13398400) O(2026876) MAX(19557564) - MALLOC's Total(6635710)
    3604( 1868) 10/07/2020 09:26:55 RTE I Thread termination in progress
  • Then you may have a shared memory issue if the servlet memory  utilization seem ok. The log clearly says there is a memory issue which could either be shared memory related or java heap of SM servlets.

  • Thanks to all. It looks like the problem was solved by decreasing the value of the sharedmemory parameter. It is set to a value when the unused memory is 40%. Command to check the status of shared memory: sm –reportshm