New Ranks & Badges For The Community!
Notice something different? The ranks and associated badges have gone "Star Fleet". See what they all mean HERE
Highlighted
Captain Captain
Captain
1325 views

web scheduler dissappear

Jump to solution

I have  ~15pcs supported instance with 10.30 cell manager infrasturure. today I found 2-3 cell manager in a very strange state: 3 days ago the cell manager not running any scheduled backups. Every function are OK, I can I login, start backup etc... just if I check the web scheduler - it's empty. (see attachment). After I restarted those cell managers, this function are come back. have you any idea what is that? The server.log attached

0 Likes
1 Solution

Accepted Solutions
Highlighted
Micro Focus Expert
Micro Focus Expert

Hello @kofop.

    If the larger memory sizes fixed your problem, then 10.40 and later releases should not have this issue since the default memory values have now been increased.

Thanks,

Shishir

View solution in original post

16 Replies
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi @kofop,

It is hard to tell. Are the Cell Managers running HP-UX, Linux or Windows?

The following seems to be the issue. This could be caused by a dead process or unreachable PostgreSQL database.

13:08:22,233 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-433) RESTEASY002005: Failed executing GET /v1/quartz/status: org.jboss.resteasy.spi.ReaderException: java.lang.NullPointerException

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hello @kofop

I completely agree with @Sebastian.Koehler. I have seen this kind of situations when some process, mainly the AS, hangs. This could happen for high load of resources on the CM and then the AS fails. In the server.log I can see this: 

03:19:52,426 ERROR [stderr] (Brute Force Protector) Exception in thread "EJB default - 8" Exception in thread "Brute Force Protector" Exception in thread "Timer-14" Exception in thread "EJB default - 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,426 ERROR [stderr] (Transaction Reaper Worker 0) java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,428 ERROR [stderr] (IdleRemover) java.lang.OutOfMemoryError: GC overhead limit exceeded

So I think that we are in the correct idea. 

Regards, 

Andres Fallas Salazar
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a LIKE by clicking on the bottom at the left of the post and show your appreciation.
0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi @kofop,

Please share the operating system used on the Cell Manager. Can you please confirm if you have configured Active Directory/LDAP with Data Protector? This seems to be a problem with KeyCloak, which is running on the AppServer, too.


03:19:52,426 ERROR [stderr] (Brute Force Protector) Exception in thread "EJB default - 8" Exception in thread "Brute Force Protector" Exception in thread "Timer-14" Exception in thread "EJB default - 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,426 ERROR [stderr] (Transaction Reaper Worker 0) java.lang.OutOfMemoryError: GC overhead limit exceeded
03:19:52,428 ERROR [stderr] (IdleRemover) java.lang.OutOfMemoryError: GC overhead limit exceeded


Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Highlighted
Captain Captain
Captain

It's on suse linux 12. 16G RAM 24 Core CPU physical server. There are no LDAP integration with DP. I have 20 cell managers in different network with same configuration, and it's starts 4-5 days ago server-by-server. We upgraded to 10.30 1-2 months ago all from10.04~10.20. There are no big load or big backups. (just 1-2 jobs/day at night).

I think it's a starting point of any backup software: you have scheduler, please start the backup. If it's not working I searching another software what can do this easy 1st step. -  and we unable to monitor this situation: every hearth-check is giving OK, just not running the backups!

I think it's looks like a big 10.30 bug guys!

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi @kofop,

This is something known. You need to ask the Micro Focus support for QCCR2A83377. Restarting the AppServer from time to time is a valid workaround (that I don't like).

To give you some more context. With A.10.30 the AppServer does not properly close file handles part of regular license validation. If you see increasing numbers in the following output (after running omnicc or backups) the fix should cure the issue and most likely the unavailabilty of the Web-based Scheduler. I have seen this only on Linux-based Cell Managers.

lsof -p `ps faxu | grep java | grep standalone | awk ' { print $2 } '` | grep etc | wc -l 

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Highlighted
Captain Captain
Captain

Thanks, that was my problem a months ago: too many open files and the OS limits are stopped the DP/DB and fulled the /var disks with db logs. After I increased the OS open files limits, the app server gets out of memory now.   The  QCCR2A83377 page is really un-informative for changes: the 10.40 may resolved this bug?

0 Likes
Highlighted
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi @kofop,

I'm currently checking if it was fixed differently in A.10.40. Will revert back soon.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi @kofop,

QCCR2A83377 is not in A.10.40. I'm currently requesting a port of the fix to the new version for a customer that I upgraded recently. If you request it for A.10.30 it should be faster. because it was already build.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
0 Likes
Highlighted
Captain Captain
Captain

Dear Sebastian Koehler,

Thank you so much! If the 10.30 patch is the faster way that will be better for me. Can you send me a link if it's finished?

 

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

Hi @kofop,

Please check your private messages.

Regards,
Sebastian Koehler

---
Please use the Like button below, if you find this post useful.
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.