Highlighted
Commander
Commander
1805 views

Error - Database Failure Manager for Event Broker

Hi All,

I am experiencing the following error message in my environment.

INFO | jvm 1 | 2011/06/07 12:58:31 | == SUBSYSTEM STATUS CHANGED ====================================================
INFO | jvm 1 | 2011/06/07 12:58:31 | Error - Event Receiver
INFO | jvm 1 | 2011/06/07 12:58:31 | == ORIGIN OF CHANGE ============================================================
INFO | jvm 1 | 2011/06/07 12:58:31 | Error - Database Failure Manager for Event Broker
INFO | jvm 1 | 2011/06/07 12:58:31 | -- DESCRIPTION -----------------------------------------------------------------
INFO | jvm 1 | 2011/06/07 12:58:31 | [Database Failure Manager for Event Broker: It appears the database is hung. There are insertion threads that have not returned from a database insert call for more than 120 seconds.]
I've 2 ESM managers running in my environment and getting the same error frequently. This is happening for the last few months and several tickets are opened with ArcSight.
But the point is, ArcSight tech support is not at able to identify the issue and provide a proper resolution.
If anybody gone through similar situation, please update me.
Regards,
Sudesh
Labels (2)
0 Likes
8 Replies
Highlighted
Admiral
Admiral

Hi There!

This is the most common error I've seen in ArcSight so far. It says that you have a bottleneck within your Database. Normally caused by high loading oracle threads - in reaction to a online DB backup, running Partition manager in the background, a significant increase of EPS, sheduled reports/trends over a long time etc.

You have to identify which Oracle Process stresses the Database so much that the response time is over the limit. There is no standard process for that, but many hints in the KB or even in this forum (like the Protect slides).

BTW: You can identify long running oracle threads by using this SQL command as sysdba - of cause during the outage:

SELECT opname, username, to_char(start_time,'DD-MON-YYYY HH24:MI:SS'), (sofar/totalwork)*100 prozent, time_remaining FROM v$session_longops WHERE time_remaining>0;

For more granular help you have to tell us what has changed so you get the outage, your environment details and what you have done so far to fix it. Otherwise the answers will be to generic.

br Tobias

0 Likes
Highlighted
Commander
Commander

Hi,

Thanks for your response.

SELECT opname, username, to_char(start_time,'DD-MON-YYYY HH24:MI:SS'), (sof                                                                                                 ar/totalwork)*100 prozent, time_remaining FROM v$session_longops WHERE time_rema                                                                                                 ining>0;
OPNAME
----------------------------------------------------------------
USERNAME                       TO_CHAR(START_TIME,'    PROZENT TIME_REMAINING
------------------------------ -------------------- ---------- --------------
Table Scan
ARCSIGHT                       04-NOV-2011 13:53:13 20.4993498           3739
Table Scan
ARCSIGHT                       04-NOV-2011 13:53:13 20.5006534           3738
Table Scan
ARCSIGHT                       04-NOV-2011 13:53:27 20.5006534           3684
OPNAME
----------------------------------------------------------------
USERNAME                       TO_CHAR(START_TIME,'    PROZENT TIME_REMAINING
------------------------------ -------------------- ---------- --------------
Table Scan
ARCSIGHT                       04-NOV-2011 14:04:23  8.5734084           2719
RMAN: incremental datafile backup
SYS                            04-NOV-2011 12:31:43 72.7373321           2194
RMAN: aggregate input
SYS                            04-NOV-2011 12:25:42 31.5359481          13308
OPNAME
----------------------------------------------------------------
USERNAME                       TO_CHAR(START_TIME,'    PROZENT TIME_REMAINING
------------------------------ -------------------- ---------- --------------
Table Scan
SYS                            04-NOV-2011 14:01:41 59.9637028            304
7 rows selected.

This is the result of given command.

Out of the possible reasons mentioned, I am sure that its not because of the Backup or Scheduled reports/trends. I am experiencing this issue for more than 5 months and getting the error event if backup not running.

One thing, I've noticed is, report generation seems to be very slow during the issue. For the last 3 days, we are not at all able to generate any single report. Not at all getting any reports.

-Sudesh

0 Likes
Highlighted

Did you namage to solve this issue.. now I am getting same error.

0 Likes
Highlighted
Admiral
Admiral

Do you have sheduled rules instead of real time rules enabled?

We had them and noticed a very high load on the DB caused by the scheduler that runs again through all events in the DB for rule matches.

This is reflected in the job name "Table Scan ARCSIGHT" on the DB. (But not only..)

We solved this by moving the rules back to real time.

br Tobias

0 Likes
Highlighted
Fleet Admiral Fleet Admiral
Fleet Admiral

Hello Tobias,

how do you find rules that are scheduled?

Volker

PS:


OPNAME
----------------------------------------------------------------
USERNAME
------------------------------
TO_CHAR(START_TIME,'DD-MON-YYY
---------------------------------------------------------------------------
   PROZENT TIME_REMAINING
---------- --------------
Table Scan
ARCSIGHT
11-JAN-2012 15:07:05
74.6485219            533


OPNAME
----------------------------------------------------------------
USERNAME
------------------------------
TO_CHAR(START_TIME,'DD-MON-YYY
---------------------------------------------------------------------------
   PROZENT TIME_REMAINING
---------- --------------
Table Scan
ARCSIGHT
11-JAN-2012 15:24:07
65.7421388            285


SQL>

0 Likes
Highlighted
Absent Member.
Absent Member.

Servus Volker....

You can schedule rules by right clicking on a group/folder in the rules ressource. All rules in that group are than scheduled.

You can find scheduled rules by open the scheduled jobs list (clock icon in the ArcSight Console)

Andreas

0 Likes
Highlighted
Vice Admiral
Vice Admiral

Hi all,

I have the same question. How can I find the rules that are scheduled in ESM manager?

Thanks

Biswa

0 Likes
Highlighted
Absent Member.
Absent Member.

Hi I am also facing the same issue. I am new to Arcsight as well.

Where to run the sql command - in the logger, ESM or Console...Also how do we disable rules and where to find the scheduled rules.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.