Regular Contributor.. Dan_Ellis Regular Contributor..
Regular Contributor..
143 views

Intermittent slow run times in central

Hi ,

 

We currently have the following tickets logged with HP support for slow performance issues 10.02

4650373331

4650361412

 

We had 6 flow that ran in a loop for 4 days over christamas period, which generated 9gb ( doubled database size) SQL . in the following tables

 

OO_EXECUTION_EVENTS_1 where EXECUTION_ID in ('368298075', '366551292', '365134916', '364060511', '363321274', '354293755')  --3483291 records

OO_EXECUTION_EVENTS_3 where EXECUTION_ID in ('368298075', '366551292', '365134916', '364060511', '363321274', '354293755')  --2488135 records

OO_EXECUTION_EVENTS_4 where EXECUTION_ID in ('368298075', '366551292', '365134916', '364060511', '363321274', '354293755')  --2998487 records

 

The data from the erroneous records was truncated and the database size is now back to normal. Serveral database maintenace scripts have been run on the recommendation of HP support.

 

The perfomance issues are still ongoing. We are seeing at times runs are processing normally and at other times the system slows down and a run may take up to 40 times longer to action than normal.

 

We  are seeing lots of errors in our logs relating to memory leaks and we are also getting big java dump files generated.

 

Following are some examples from our logs this morning.

 

Is there someone around that could possibly contact us offline to help us resolve the two open cases?

 

    WRAPPER

INFO   | jvm 1    | 2015/01/22 10:19:33 | java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  | 2015/01/22 10:19:33 | The JVM has run out of memory.  Restarting JVM.
INFO   | jvm 1    | 2015/01/22 10:19:33 | Dumping heap to java_pid8436.hprof ...

    GENERAL

2015-01-22 11:20:35,204 [http-nio-8443-exec-10] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:35,405 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:37,063 [http-nio-8443-exec-8] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:38,713 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:40,378 [http-nio-8443-exec-8] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:42,042 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:43,735 [http-nio-8443-exec-8] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:45,390 [http-nio-8443-exec-7] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:47,054 [http-nio-8443-exec-1] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:48,730 [http-nio-8443-exec-7] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:50,379 [http-nio-8443-exec-9] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:51,922 [http-nio-8443-exec-8] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:53,595 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:54,485 [http-nio-8443-exec-5] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:55,446 [http-nio-8443-exec-6] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:55,692 [http-nio-8443-exec-1] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:55,904 [http-nio-8443-exec-10] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:56,157 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:56,423 [http-nio-8443-exec-8] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:56,576 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:20:56,720 [http-nio-8443-exec-8] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:21:33,148 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:14,691 [http-nio-8443-exec-6] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:21,017 [http-nio-8443-exec-6] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:24,774 [http-nio-8443-exec-6] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:28,744 [http-nio-8443-exec-2] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:34,703 [http-nio-8443-exec-9] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:40,388 [http-nio-8443-exec-9] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}
2015-01-22 11:26:47,890 [http-nio-8443-exec-1] (HttpMethodDirector.java:697) WARN  - Unable to respond to any of these challenges: {xbasic=xBasic realm="Operation Orchestration"}

    SERVER

2015-01-22 11:26:37,387 [http-nio-8443-exec-4] (PASRestController.java:781) ERROR - Could not access HTTP invoker remote service at [http://masupappp06.aklc.govt.nz:8080/oo/backwards-compatibility/triggeringService]; nested exception is java.net.SocketTimeoutException: Read timed out
org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://masupappp06.aklc.govt.nz:8080/oo/backwards-compatibility/triggeringService]; nested exception is java.net.SocketTimeoutException: Read timed out
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy327.executeFlow(Unknown Source)
    at com.hp.oo.ws.services.WSCentralServiceImpl.runFlowEx(WSCentralServiceImpl.java:372)

 

 

 

 

Labels (1)
0 Likes
1 Reply
Alex_Eskin Absent Member.
Absent Member.

Re: Intermittent slow run times in central

Hello Dan,

1. What is the maximum heap size currently configured for the Central? It is set by wrapper.java.maxmemory property in central_wrapper.conf. I recommend to set it to at least 3072.

2. How many concurrent runs can you see in the run explorer, and do these runs have parallel/multi-instance steps?

3. How many workers (both internal and RASes) are used?

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.