Database Failure Manager for Event Broker
Need help thanks
[Mon May 11 14:56:27 MDT 2015] == SUBSYSTEM STATUS CHANGED
Error - Event
== ORIGIN OF CHANGE
Error - Database
Failure Manager for Event Broker
[Database Failure Manager for Event Broker: It appears the
database is hung. There are insertion threads that have not returned from a
database insert call for more than 120 seconds.]
This kind of ERROR is difficult to troubleshoot because it could be many reasons where this error can occur.
I advice you to create a ticket to the HP ArcSight Support.
In the same time, could you please check if you have memory issue in the server.log (red zone)
and also check here: https://<your manager>:8443/arcsight/web/manage.jsp
then go DBSecurityEventBroker to verify device descriptor value is not over max value.
Could you also check in server.log information before the ORA- Error to understand what is happen?
Could you please check the dashboard - Database Performance Statistics? Maybe you have a tablespace full!
I need more information on your infra to help you. Which ESM version are using? with ORACLE?
I'm using 18.104.22.16847.0 right now. for the Database Performance Statistics
ACR-event data 72% free
ACR-event index 71% free
ACR System data 99% free
ACR System index 99% free
Now looking at the MaxMemory and the UsedMemory I'm just about max out and at the time of the event we are running 99% to 100% I'm looking in to getting more ram for now hope this helps.
I am using ESM v5.2 too but with patch 2 - v22.214.171.12464.2 to be precise.
Normally I should help you, we have approximatively the same ESM version.
With the information you show me, it is clearly not a disk space issue.
Could you please check memory and CPU value in the web link I gave you.
For CPU, click on HostSystemInfo, I need value below User
For RAM, click on MemoryMonitor, I need CurrentUsage and CurrentArea
last point, regarding DBSecurityEventBroker, could you please check in SideObjectCacheStatistics
the difference between Capacity (max value) and Size (current value) to be sure that you have not a Descriptor issue!
Could you also check these values at the same time you have the DB Issue.
Could you please tell me your results. Then if normal, you will analyse ESM log file.
To be sure if you have to increase JVM, you need to search in the ESM logs, words as 'Red Zone' or 'Yellow Zone' which is not good!!! (related to CurrentArea value = real-time)
Currently how is the Global Connector cache? Empty or Not?
MemoryMonitor CurrentUsage 85 CurrentArea Green
ok a newbee Qustion how do i fix this
If you have 85% of memory used now!I It is not bad depends of your current ESM total EPS?
You should increase your JVM a bit to be sure that when there are peaks or a lots of rules triggered or system overloaded you will not enter in yellow zone or more critical in red zone!
How much is your current value for JVM?
Regarding Device Descriptor, it is a bit more complicated.
I will try to explain it tomorrow but could you please confirm the value you sent me, it is very high!
You have 900 000 in Capacity and 900 000 in Size?
How many devices send logs to your SIEM?
We have for device Descriptor 70000 for capacity and 63 000 in size.
According HP Support, we have already a huge infra 10000 EPS Avg. and 650000000 EPD for more or less 100 connectors and 200 devices!
Normally default value for this parameter is 50000!
Do you have already increased to 900 000! it is huge! Are-you sure about the line?
If it is correct you have an issue with a syslog connector for sure!
We had the same issue and it turned out to be the side table cache values as well. We increased the DeviceDescriptor to 1,500,000 and it bought us about 90 days before we started getting significant misses again and insertion hangs. We were restarting the manager service every 20 days before we made the change. Thankfully only a few more months before we go to CORR. I thought I would share our current numbers for your amusement. Collecting with about 100 connectors from 13,000 devices.
1,500,000 1,500,000 1,500,486 1,479,953 997 1,485,279 3 6,293 15 100
It is not normal you should increase device descriptor to 1500000.
You should identify which connector generated the most device descriptor, I am sure it is a syslog connector or a flex parser not properly built.
The problem with that is the impact on performance and correlation engine.
I advice you to find the root cause and to fix it.
We have done the same, after finding the root cause and fixing it, we have finally been able to decrease the device descriptor to 70000 and now this value is stable.
Could you please tell us what was the root cause in your case?
We are facing same database issue. Also now we are not able to open the URL: https://<your manager>:8443/arcsight/web/manage.jsp
Previously the URL was working fine.
Hi Devasis Mahato,
In my ESM version v5.X, it was in the server.properties file (not the default - never change server.default.properties).
I do not think this has changed in ESM v6.x
Make a backup of your file for roll back and also for change management and troubleshooting.
Check in knowledge base if the parameters are still the same but for my ESM version, I have added the following lines:
I hope I have answered to your question.