web scheduler dissappear

I have  ~15pcs supported instance with 10.30 cell manager infrasturure. today I found 2-3 cell manager in a very strange state: 3 days ago the cell manager not running any scheduled backups. Every function are OK, I can I login, start backup etc... just if I check the web scheduler - it's empty. (see attachment). After I restarted those cell managers, this function are come back. have you any idea what is that? The server.log attached

  • Hi ,

    It is hard to tell. Are the Cell Managers running HP-UX, Linux or Windows?

    The following seems to be the issue. This could be caused by a dead process or unreachable PostgreSQL database.

    13:08:22,233 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-433) RESTEASY002005: Failed executing GET /v1/quartz/status: org.jboss.resteasy.spi.ReaderException: java.lang.NullPointerException

    Regards,
    Sebastian Koehler

  • Hi ,

    It is hard to tell. Are the Cell Managers running HP-UX, Linux or Windows?

    The following seems to be the issue. This could be caused by a dead process or unreachable PostgreSQL database.

    13:08:22,233 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-433) RESTEASY002005: Failed executing GET /v1/quartz/status: org.jboss.resteasy.spi.ReaderException: java.lang.NullPointerException

    Regards,
    Sebastian Koehler

  • Hello , 

    I completely agree with . I have seen this kind of situations when some process, mainly the AS, hangs. This could happen for high load of resources on the CM and then the AS fails. In the server.log I can see this: 

    03:19:52,426 ERROR [stderr] (Brute Force Protector) Exception in thread "EJB default - 8" Exception in thread "Brute Force Protector" Exception in thread "Timer-14" Exception in thread "EJB default - 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
    03:19:52,426 ERROR [stderr] (Transaction Reaper Worker 0) java.lang.OutOfMemoryError: GC overhead limit exceeded
    03:19:52,428 ERROR [stderr] (IdleRemover) java.lang.OutOfMemoryError: GC overhead limit exceeded

    So I think that we are in the correct idea. 

    Regards, 

  • Hi ,

    Please share the operating system used on the Cell Manager. Can you please confirm if you have configured Active Directory/LDAP with Data Protector? This seems to be a problem with KeyCloak, which is running on the AppServer, too.


    03:19:52,426 ERROR [stderr] (Brute Force Protector) Exception in thread "EJB default - 8" Exception in thread "Brute Force Protector" Exception in thread "Timer-14" Exception in thread "EJB default - 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
    03:19:52,426 ERROR [stderr] (Transaction Reaper Worker 0) java.lang.OutOfMemoryError: GC overhead limit exceeded
    03:19:52,428 ERROR [stderr] (IdleRemover) java.lang.OutOfMemoryError: GC overhead limit exceeded


    Regards,
    Sebastian Koehler

  • It's on suse linux 12. 16G RAM 24 Core CPU physical server. There are no LDAP integration with DP. I have 20 cell managers in different network with same configuration, and it's starts 4-5 days ago server-by-server. We upgraded to 10.30 1-2 months ago all from10.04~10.20. There are no big load or big backups. (just 1-2 jobs/day at night).

    I think it's a starting point of any backup software: you have scheduler, please start the backup. If it's not working I searching another software what can do this easy 1st step. -  and we unable to monitor this situation: every hearth-check is giving OK, just not running the backups!

    I think it's looks like a big 10.30 bug guys!

  • Hi ,

    This is something known. You need to ask the Micro Focus support for QCCR2A83377. Restarting the AppServer from time to time is a valid workaround (that I don't like).

    To give you some more context. With A.10.30 the AppServer does not properly close file handles part of regular license validation. If you see increasing numbers in the following output (after running omnicc or backups) the fix should cure the issue and most likely the unavailabilty of the Web-based Scheduler. I have seen this only on Linux-based Cell Managers.

    lsof -p `ps faxu | grep java | grep standalone | awk ' { print $2 } '` | grep etc | wc -l 

    Regards,
    Sebastian Koehler

  • Thanks, that was my problem a months ago: too many open files and the OS limits are stopped the DP/DB and fulled the /var disks with db logs. After I increased the OS open files limits, the app server gets out of memory now.   The  QCCR2A83377 page is really un-informative for changes: the 10.40 may resolved this bug?

  • Hi ,

    I'm currently checking if it was fixed differently in A.10.40. Will revert back soon.

    Regards,
    Sebastian Koehler

  • Hi ,

    QCCR2A83377 is not in A.10.40. I'm currently requesting a port of the fix to the new version for a customer that I upgraded recently. If you request it for A.10.30 it should be faster. because it was already build.

    Regards,
    Sebastian Koehler