Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Highlighted
vdor Absent Member.
Absent Member.
1052 views

Active Channels (incorrectly?) showing empty despite constant event flow?

I'm having the weirdest problem. I have a recent ESM installation (5.0.0.6521.1) on a RHEL appliance that has 8 Cores and 16GBs of Ram. I'm dumping about 7 Million events into the system/day max.

All of my connectors are dual streamed into a logger and ESM. I use data monitors on my ESM and see a fairly consistant event rate between that and the logger.

Yet, somehow when I open an active channel, whether it be filtered on something other than Agent ID or Agent ID itself, the channel typically comes up completely empty with the message "This channel is active but temporarily quiet" or "No data matches this query, please select a different query and try again". Each of my channels is going back between 1-4 hours, and there should be hundreds of events from almost all 6 of my connectors. If I alter the channel while it's open and update to "continuously evaluate", then often times it will refresh and display all the events it should have. In one instance, it seems to only display events going through since the channel was opened.

While refreshing by updating a recently created channel to continuously evaluate seems to bring focus and attention from the ESM and allow it to populate correctly, the channel will sputter after that point. It basically runs for a little while, and then slowly begins to not populate until the channel is completely empty. I have a ticket open with Arcsight, but they don't seem to know what's going on, so I'm hoping someone else has experienced something similar.

The ESM instance is new, has 4Gbs of ram allocated, and 95% database free space. Everything is in FIPs mode.

Labels (4)
0 Likes
Reply
10 Replies
a2g Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

Do basic Reports produce results?  "List all events for the past 2 hours", that sort of thing?

Also, have you tried running your channels on both End Time and Mgr Receipt Time?

0 Likes
Reply
vdor Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

I've looked into the time issues, but everything is synced to GMT, so that hasn't changed anything. I'm having a separate issue getting reports to run at all due to  a resource error. I've opened a ticket with Arcsight for that one as well. However, I do have data monitors that show me a graph of the event flow from each connector over a period of 12 hours, and things are consistent with what we see on the logger and would expect in ESM. I think I may try a reboot of the ESM appliance (which means the DB, web and manager) to see if maybe there's anything hung, but I would prefer to avoid that.

0 Likes
Reply
Acclaimed Contributor.. Volker Michels Acclaimed Contributor..
Acclaimed Contributor..

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

Hello,

how do you open the channels?

Do you right click on the connector / logger and "Create Channel with Filter..."?

Do you have access to the connector / logger and are you able to check the event flow there?

You can also use the "Send Command" option to check you connetors and loggers, Status --> Get Device Status.

Volker

0 Likes
Reply
vdor Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

I've created channels multiple ways. I've right-clicked the connector and chosen to "Create channel with filter", I've created filters based off of Device Vendor, agent id, etc.

I have data monitors and dashboards setup to monitor connector event flow, and can see that I'm not losing events. I also have a logger where the events are coming in steadily. The events are in the database, it's just as if the channel either stops querying (even when set to continuously evaluate), or the query returns nothing the first time its sent. Often, if I close the channel and reopen it, it will populate with all the missing data that has dropped off the table while it was previously open. Sometimes, the channel will return no results, and then a few minutes later it will begin to magically appear.

I've also created a continuously evaluating channel that shows nothing for the previous 2 hours (the timeframe i've set), but will begin to populate events starting at the moment I've created the channel. So it brings in new events but doesn't grab the previous 2 hours of events.

It's pretty frustrating, as the system is at 5.0, on an Arcsight ESM appliance, with the latest java and release patch. Couple that with the problem that it's not consistent. Sometimes it'll work fine and I think I'm out of the woods, other times it just reverts back to spotty performance. I have another ESM on a different network at 4.5 patch 2 running in windows, and have never experienced this problem.

0 Likes
Reply
deathbywedgie1 Frequent Contributor.
Frequent Contributor.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

Forgive me if I missed it, but I didn't see a full answer per se to the previous question about using MRT and ET both. I wonder as well, could all your channels be based on End Time instead of Manager Receipt Time? Logger always uses its own timestamp, so if the agents are having time issues it would not affect Logger, but it would absolutely affect the End Time of events in ESM.

0 Likes
Reply
vdor Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

All of my channels are based off of MRT, and I actually have all of my connectors configured within the same time zone for ease of use. Whenever I have time issues, I do a test by setting the End time to equal $Now + 12h , which will actually work despite it not being an option.

0 Likes
Reply
hendersonc Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

this really sounds like a timestamp issue.   are you overwriting the event time with the connector time by chance?  is your manager in the same timezone as your console?  are you trying an active channel with no filter to ensure that events are coming in and its not just that your filter is wrong?

0 Likes
Reply
vdor Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

Unfortunately, all connectors and the ESM are all using GMT, and are all within a couple minutes of each other. If a channel stops populating data, and then I close and restart it, it works again (still set with the same time settings and to continuously evaluate). I'm going to be upgrading to SP1 on Monday.. we'll see if that takes care of it..  Arcsight support has basically been ignoring this one for me since it isn't obvious.

0 Likes
Reply
vdor Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

Things appear to be working better after upgrading to SP1. I'm crossing my fingers and am cautiously optimistic.

0 Likes
Reply
vivekvenu188 Absent Member.
Absent Member.

Re: Active Channels (incorrectly?) showing empty despite constant event flow?

Check the Active Channel using Manager Receipt time.

0 Likes
Reply
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.