Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..
804 views

web scheduler dissappear

Jump to solution

I have  ~15pcs supported instance with 10.30 cell manager infrasturure. today I found 2-3 cell manager in a very strange state: 3 days ago the cell manager not running any scheduled backups. Every function are OK, I can I login, start backup etc... just if I check the web scheduler - it's empty. (see attachment). After I restarted those cell managers, this function are come back. have you any idea what is that? The server.log attached

0 Likes
1 Solution

Accepted Solutions
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hello @kofop.

    If the larger memory sizes fixed your problem, then 10.40 and later releases should not have this issue since the default memory values have now been increased.

Thanks,

Shishir

16 Replies
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

It is hard to tell. Are the Cell Managers running HP-UX, Linux or Windows?

The following seems to be the issue. This could be caused by a dead process or unreachable PostgreSQL database.

13:08:22,233 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-433) RESTEASY002005: Failed executing GET /v1/quartz/status: org.jboss.resteasy.spi.ReaderException: java.lang.NullPointerException

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hello @kofop

I completely agree with @Sebastian.Koehler. I have seen this kind of situations when some process, mainly the AS, hangs. This could happen for high load of resources on the CM and then the AS fails. In the server.log I can see this: 

03:19:52,426 ERROR [stderr] (Brute Force Protector) Exception in thread "EJB default - 8" Exception in thread "Brute Force Protector" Exception in thread "Timer-14" Exception in thread "EJB default - 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,426 ERROR [stderr] (Transaction Reaper Worker 0) java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,428 ERROR [stderr] (IdleRemover) java.lang.OutOfMemoryError: GC overhead limit exceeded

So I think that we are in the correct idea. 

Regards, 

Andres Fallas Salazar
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a LIKE by clicking on the bottom at the left of the post and show your appreciation.
0 Likes
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

Please share the operating system used on the Cell Manager. Can you please confirm if you have configured Active Directory/LDAP with Data Protector? This seems to be a problem with KeyCloak, which is running on the AppServer, too.


03:19:52,426 ERROR [stderr] (Brute Force Protector) Exception in thread "EJB default - 8" Exception in thread "Brute Force Protector" Exception in thread "Timer-14" Exception in thread "EJB default - 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,426 ERROR [stderr] (Transaction Reaper Worker 0) java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,428 ERROR [stderr] (IdleRemover) java.lang.OutOfMemoryError: GC overhead limit exceeded


Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..

Re: web scheduler dissappear

Jump to solution

It's on suse linux 12. 16G RAM 24 Core CPU physical server. There are no LDAP integration with DP. I have 20 cell managers in different network with same configuration, and it's starts 4-5 days ago server-by-server. We upgraded to 10.30 1-2 months ago all from10.04~10.20. There are no big load or big backups. (just 1-2 jobs/day at night).

I think it's a starting point of any backup software: you have scheduler, please start the backup. If it's not working I searching another software what can do this easy 1st step. -  and we unable to monitor this situation: every hearth-check is giving OK, just not running the backups!

I think it's looks like a big 10.30 bug guys!

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

This is something known. You need to ask the Micro Focus support for QCCR2A83377. Restarting the AppServer from time to time is a valid workaround (that I don't like).

To give you some more context. With A.10.30 the AppServer does not properly close file handles part of regular license validation. If you see increasing numbers in the following output (after running omnicc or backups) the fix should cure the issue and most likely the unavailabilty of the Web-based Scheduler. I have seen this only on Linux-based Cell Managers.

lsof -p `ps faxu | grep java | grep standalone | awk ' { print $2 } '` | grep etc | wc -l 

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..

Re: web scheduler dissappear

Jump to solution

Thanks, that was my problem a months ago: too many open files and the OS limits are stopped the DP/DB and fulled the /var disks with db logs. After I increased the OS open files limits, the app server gets out of memory now.   The  QCCR2A83377 page is really un-informative for changes: the 10.40 may resolved this bug?

0 Likes
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..

Re: web scheduler dissappear

Jump to solution
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

I'm currently checking if it was fixed differently in A.10.40. Will revert back soon.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

QCCR2A83377 is not in A.10.40. I'm currently requesting a port of the fix to the new version for a customer that I upgraded recently. If you request it for A.10.30 it should be faster. because it was already build.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..

Re: web scheduler dissappear

Jump to solution

Dear Sebastian Koehler,

Thank you so much! If the 10.30 patch is the faster way that will be better for me. Can you send me a link if it's finished?

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

Please check your private messages.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..

Re: web scheduler dissappear

Jump to solution

Thank you so much!

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: web scheduler dissappear

Jump to solution

Hi @kofop,

Seems the issue mentioned (QCCR2A83377) was fixed in A.10.40. Support confirmed it and my own tests confirm it. So upgrading to the new version is considered safe.

I asked them to review/update the documentation accordingly.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Trusted Contributor.. kofop Trusted Contributor..
Trusted Contributor..

Re: web scheduler dissappear

Jump to solution

Hi,

It's a strange news, because the patch what you sent  me is not solve the problem on 10.30 (but ok, it's solve the high open file number problem). I found a workarond what looks like changeing something (4 days ago): I installed the patch, and I changed the xmx/xms from 2048m to 4096m in /opt/omni/AppServer/bin/standalone.conf, and restarted the processes:

JAVA_OPTS="-Xmx4096m -Xms4096m -Xss228k -XX:NewRatio=2 -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true"

The old logs:

"04:37:58,116 WARN [org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorTrackingImpl] (Transaction Reaper Worker 2) HHH000451: Transaction afterCompletion called by a background threa
05:00:10,110 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 2) ARJUNA012113: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 2,5,main] missed interrupt when cancelling TX 0:ffff0aa40433:-3fdee811:5cf1
05:22:50,382 ERROR [org.keycloak.services] (Timer-2) KC-SERVICES0089: Failed to run scheduled task ClearExpiredEvents: java.lang.RuntimeException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at org.keycloak.transaction.JtaTransactionWrapper.handleException(JtaTransactionWrapper.java:77)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:94)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:126)
at org.keycloak.services.scheduled.ClusterAwareScheduledTaskRunner.runTask(ClusterAwareScheduledTaskRunner.java:62)
at org.keycloak.services.scheduled.ScheduledTaskRunner.run(ScheduledTaskRunner.java:45)
at org.keycloak.timer.basic.BasicTimerProvider$1.run(BasicTimerProvider.java:51)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:92)
... 6 more

06:11:46,266 ERROR [stderr] (JCA PoolFiller) Exception in thread "JCA PoolFiller" java.lang.OutOfMemoryError: Java heap space
06:21:15,174 ERROR [org.xnio.listener] (default I/O-19) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

05:56:29,088 ERROR [org.xnio.listener] (default I/O-41) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:24:12,233 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:19:52,703 ERROR [org.xnio.listener] (default I/O-35) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:30:29,492 ERROR [org.xnio.listener] (default I/O-31) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:16:18,497 ERROR [org.xnio.listener] (default I/O-13) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

06:47:54,744 ERROR [stderr] (EJB default - 9) Exception in thread "ServerService Thread Pool -- 32" java.lang.OutOfMemoryError: Java heap space
06:44:36,218 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

"

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.