Database Failure Manager for Event Broker

[shawn.wickizer]

Need help thanks

[Mon May 11 14:56:27 MDT 2015] == SUBSYSTEM STATUS CHANGED
====================================================

   Error - Event
Receiver

== ORIGIN OF CHANGE
============================================================

   Error - Database
Failure Manager for Event Broker

-- DESCRIPTION
-----------------------------------------------------------------

[Database Failure Manager for Event Broker: It appears the
database is hung. There are insertion threads that have not returned from a
database insert call for more than 120 seconds.]

================================================================================

[mschleich]

Dear Shawn,

This kind of ERROR is difficult to troubleshoot because it could be many reasons where this error can occur.

I advice you to create a ticket to the HP ArcSight Support.

In the same time, could you please check if you have memory issue in the server.log (red zone)

and also check here:     https://<your manager>:8443/arcsight/web/manage.jsp

then go DBSecurityEventBroker to verify device descriptor value is not over max value.

Could you also check in server.log information before the ORA- Error to understand what is happen?

Could you please check the dashboard - Database Performance Statistics? Maybe you have a tablespace full!

I need more information on your infra to help you. Which ESM version are using? with ORACLE?

Thanks

Regards

Michael

[shawn.wickizer]

Michael,

I'm using 5.2.0.6847.0 right now. for the Database Performance Statistics

ACR-event data 72% free

ACR-event index 71% free

ACR System data 99% free

ACR System index 99% free

Now looking at the MaxMemory and the UsedMemory I'm just about max out and at the time of the event we are running 99% to 100% I'm looking in to getting more ram for now hope this helps.

Thanks

Shawn

[mschleich]

Dear Shawn,

I am using ESM v5.2 too but with patch 2 - v5.2.0.6964.2 to be precise.

Normally I should help you, we have approximatively the same ESM version.

With the information you show me, it is clearly not a disk space issue.

Could you please check memory and CPU value in the web link I gave you.

For CPU, click on HostSystemInfo, I need value below User

For RAM, click on MemoryMonitor, I need CurrentUsage and CurrentArea

last point, regarding DBSecurityEventBroker, could you please check in SideObjectCacheStatistics

the difference between Capacity (max value) and Size (current value) to be sure that you have not a Descriptor issue!

Could you also check these values at the same time you have the DB Issue.

Could you please tell me your results. Then if normal, you will analyse ESM log file.

To be sure if you have to increase JVM, you need to search in the ESM logs, words as 'Red Zone' or 'Yellow Zone' which is not good!!! (related to CurrentArea value = real-time)

Currently how is the Global Connector cache? Empty or Not?

Thanks

Kind Regards

Michael

[shawn.wickizer]

Michael,

MemoryMonitor  CurrentUsage 85 CurrentArea Green

DeviceDescriptor

900,000

900,000

1,040,503

2,283,268,803

282,022

2,283,506,207

3

11,557

0

0

ok a newbee Qustion how do i fix this

[mschleich]

Dear Shawn,

If you have 85% of memory used now!I It is not bad depends of your current ESM total EPS?

You should increase your JVM a bit to be sure that when there are peaks or a lots of rules triggered or system overloaded you will not enter in yellow zone or more critical in red zone!

How much is your current value for JVM?

Regarding Device Descriptor, it is a bit more complicated.

I will try to explain it tomorrow but could you please confirm the value you sent me, it is very high!

You have 900 000 in Capacity and 900 000 in Size?

How many devices send logs to your SIEM?

We have for device Descriptor 70000   for capacity and 63 000 in size.

According HP Support, we have already a huge infra 10000 EPS Avg. and 650000000 EPD for more or less 100 connectors and 200 devices!

Normally default value for this parameter is 50000!

Do you have already increased to 900 000! it is huge! Are-you sure about the line?

If it is correct you have an issue with a syslog connector for sure!

Regards

Michael

[josh_tonak]

We had the same issue and it turned out to be the side table cache values as well.  We increased the DeviceDescriptor to 1,500,000 and it bought us about 90 days before we started getting significant misses again and insertion hangs.  We were restarting the manager service every 20 days before we made the change.  Thankfully only a few more months before we go to CORR.  I thought I would share our current numbers for your amusement.  Collecting with about 100 connectors from 13,000 devices.

DeviceDescriptor

1,500,000  1,500,000  1,500,486  1,479,953  997  1,485,279  3  6,293  15  100

[mschleich]

Hi Josh,

It is not normal you should increase device descriptor to 1500000.

You should identify which connector generated the most device descriptor, I am sure it is a syslog connector or a flex parser not properly built.

The problem with that is the impact on performance and correlation engine.

I advice you to find the root cause and to fix it.

We have done the same, after finding the root cause and fixing it, we have finally been able to decrease the device descriptor to 70000 and now this value is stable.

Thanks

Kind Regards

Michael

[amane]

Hi Michael,

Could you please tell us what was the root cause in your case?

We are facing same database issue. Also now we are not able to open the URL: https://<your manager>:8443/arcsight/web/manage.jsp

Previously the URL was working fine.

Regards,

Ameer Mane

[devasismahato]

How to change the device descriptor value?

MY ESM is getting hung sometimes and restarting sometimes. ESM 6.8

[mschleich]

Hi Devasis Mahato,

In my ESM version v5.X, it was in the server.properties file (not the default - never change server.default.properties).

I do not think this has changed in ESM v6.x

Make a backup of your file for roll back and also for change management and troubleshooting.

Check in knowledge base if the parameters are still the same but for my ESM version, I have added the following lines:

persist.securityevent.stcache.GeoDescriptor=120000

persist.securityevent.stcache.AgentDescriptor=20000

persist.securityevent.stcache.DeviceDescriptor=120000

persist.securityevent.stcache.CategoryDescriptor=20000

persist.securityevent.stcache.LabelsDescriptor=10000

persist.securityevent.stcache.ResourceRef=100000

I hope I have answered to your question.

Thanks

Kind Regards

Michael