[ERROR][default.com.arcsight.common.persist.DBSecurityEventBrokerBase$HungDatabaseCheck][runCheck] Database appears hung. 24 hung threads: 02-Pre-SecurityEventPersistor0, 02-Pre-SecurityEventPersistor8, 02-Pre-SecurityEventPersistor4, 02-Pre-SecurityEventPersistor11, 02-Pre-SecurityEventPersistor10, 02-Pre-SecurityEventPersistor17, 02-Pre-SecurityEventPersistor9, 02-Pre-SecurityEventPersistor16, 02-Pre-SecurityEventPersistor15, 02-Pre-SecurityEventPersistor19, 02-Pre-SecurityEventPersistor7, 02-Pre-SecurityEventPersistor18, 02-Pre-SecurityEventPersistor14, 02-Pre-SecurityEventPersistor6, 02-Pre-SecurityEventPersistor1, 02-Pre-SecurityEventPersistor5, 02-Pre-SecurityEventPersistor20, 02-Pre-SecurityEventPersistor23, 02-Pre-SecurityEventPersistor2, 02-Pre-SecurityEventPersistor22, 02-Pre-SecurityEventPersistor13, 02-Pre-SecurityEventPersistor21, 02-Pre-SecurityEventPersistor12, 02-Pre-SecurityEventPersistor3
My ESM: 5.2
I see that ESM cannot try to write events to DB. If I restart manager all stay ok, but after 5-6 hours error start again. No any ORA-Errors in oracle log. Direct link from ESM to DB.
Average EPS = 5000.
NEARLY EXACT SYMPTOMS
server.log:[2017-02-01 14:15:37,470][ERROR][default.com.arcsight.common.persist.DBSecurityEventBrokerBase$HungDatabaseCheck] Database appears hung; a thread dump has been requested. 6 hung thread(s): 02-Pre-SecurityEventPersistor206, 02-Pre-SecurityEventPersistor204, 02-Pre-SecurityEventPersistor207, 02-Pre-SecurityEventPersistor205, 02-Pre-SecurityEventPersistor200, 02-Pre-SecurityEventPersistor199
I can restart the manager and it will run for 4~6 hours before I get messages like the above or the following:
Database Failure Manager for Event Broker:
It appears the database is hung. There are insertion threads that have not returned from a database insert call for more than 120 seconds.
At this point, events no longer persist and all inbound events from all connectors stop.