web scheduler dissappear

[kofop]

I have  ~15pcs supported instance with 10.30 cell manager infrasturure. today I found 2-3 cell manager in a very strange state: 3 days ago the cell manager not running any scheduled backups. Every function are OK, I can I login, start backup etc... just if I check the web scheduler - it's empty. (see attachment). After I restarted those cell managers, this function are come back. have you any idea what is that? The server.log attached

[kofop]

Thank you so much!

[Sebastian.Koehler]

Hi @kofop,

Seems the issue mentioned (QCCR2A83377) was fixed in A.10.40. Support confirmed it and my own tests confirm it. So upgrading to the new version is considered safe.

I asked them to review/update the documentation accordingly.

Regards,
Sebastian Koehler

[kofop]

Hi,

It's a strange news, because the patch what you sent  me is not solve the problem on 10.30 (but ok, it's solve the high open file number problem). I found a workarond what looks like changeing something (4 days ago): I installed the patch, and I changed the xmx/xms from 2048m to 4096m in /opt/omni/AppServer/bin/standalone.conf, and restarted the processes:

JAVA_OPTS="-Xmx4096m -Xms4096m -Xss228k -XX:NewRatio=2 -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true"

The old logs:

"04:37:58,116 WARN [org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorTrackingImpl] (Transaction Reaper Worker 2) HHH000451: Transaction afterCompletion called by a background threa
05:00:10,110 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 2) ARJUNA012113: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 2,5,main] missed interrupt when cancelling TX 0:ffff0aa40433:-3fdee811:5cf1
05:22:50,382 ERROR [org.keycloak.services] (Timer-2) KC-SERVICES0089: Failed to run scheduled task ClearExpiredEvents: java.lang.RuntimeException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:77)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:126)
at org.keycloak.services.scheduled.ClusterAwareScheduledTaskRunner.runTask(ClusterAwareScheduledTaskRunner.java:62)
at org.keycloak.services.scheduled.ScheduledTaskRunner.run(ScheduledTaskRunner.java:45)
at org.keycloak.timer.basic.BasicTimerProvider$1.run(BasicTimerProvider.java:51)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 6 more

06:11:46,266 ERROR [stderr] (JCA PoolFiller) Exception in thread "JCA PoolFiller" java.lang.OutOfMemoryError: Java heap space
06:21:15,174 ERROR [org.xnio.listener] (default I/O-19) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

05:56:29,088 ERROR [org.xnio.listener] (default I/O-41) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:24:12,233 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:19:52,703 ERROR [org.xnio.listener] (default I/O-35) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:30:29,492 ERROR [org.xnio.listener] (default I/O-31) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:16:18,497 ERROR [org.xnio.listener] (default I/O-13) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:47:54,744 ERROR [stderr] (EJB default - 9) Exception in thread "ServerService Thread Pool -- 32" java.lang.OutOfMemoryError: Java heap space
06:44:36,218 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

"

[Shishir Misra · Accepted solution]

Hello @kofop.

    If the larger memory sizes fixed your problem, then 10.40 and later releases should not have this issue since the default memory values have now been increased.

Thanks,

Shishir

[kofop]

yes, the 10.40 fixed my problems.