On my side, I see none of these events, except when the manager is restarted, I get a manager:100
Thanks! Is anyone seeing "manager:" or "database:" events when the ingest problem occurs?
Joe, manager: and other internal monitor: events do not appear when manager is overloaded with events or having an issue processing events. It seems that at some point manager "decides" that these are of a lower priority ...
From your other post:
"22-30K eps for 30 days+ ; no reboot or manager restart ; no issues"
Does that mean you see the issue on your system too?
FYI - Sometimes you'll see the following when the manager is having issues with ingest. (But apparently not for what we're trying to fix in this thread)
We have zero manager:200, manager:201, and database:102 events. This occurred last night and there were none.
It triggered none of my 'critical system failures' which has a whole series of event ID (including manager 200/201, although I don't have database 102 - have 100/101) in it...
I am using Kaminario SSD and 125 connectors into ESM and I also have the issue with high CPU and no persisitance. I am also suffering from the IPV6 error mentioned above. Support tried to blame the name of a connector which is clearly not the issue. I am suprised that support was not aware of this asset creation problem as it seems to have to documented tickets.
Thanks- How did you find this out? I was having an issue with esm and supported pointed out this error and claimed it was my content.
Weird update, support just had us restart the rules engine (not the entire manager service) via the new version of manage.jsp. Cleared up our GC issues, caches, and memory real quick... (strikethrough'd the rule engine restart because it causes some major irrecoverable problems)
Going to monitor for a couple weeks to see if this maintains the same as a manager service restart...
***This is not a command to be used lightly!!! I would highly suggest troubleshooting this through support.
Login to the ESM 6 advanced admin web console, system management, rules engine, and then restart at the bottom. I also didn't know of this capability before today.
(strikethrough'd the rule engine restart because it causes some major irrecoverable problems)
Well, when I opened my console this morning, I actually had the problem occuring. EPS were at 6k (normally 10-12k) and cache was pilling up. So I did try to restart the rules engine, and to my surprise, it actually worked!!! EPS went to 20k almost instantly and cache cleared up.
One thing I did not check was the CPU load, but, lately, I had the problem every 4-5 days, so I'll have plenty of occasions to check it out!
Thanks a lot for that info Ray, everyone will stop complaining about the frequent manager restarts!
This is awesome Ray, thanks for sharing!
So instead of restarting the manager every week, we just need to restart the rules engine?
I'm assuming any events the rule was accumulating towards a threshold would get dumped.
One thing I just noticed, I got couple dashboards that are not working after the Rules Engine restart. Less of an impact on the analysts logged in, but bad if the dashboards stop working... (although I seen this behavior with 1 or 2 dashboards after a manager restart)