Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
dzuperku1 Absent Member.
Absent Member.
373 views

Log management solution(s)?

Does anyone have experience using a log management solution other than ArcSight Logger (log logic, log rhythm, etc) that they forward to ESM?

Labels (2)
Tags (1)
0 Likes
10 Replies
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: Log management solution(s)?

I am aware of several customers who take this approach, all with limited results.

There are a few discussions around this with regards to Splunk (which is one that comes up fairly often), but its not without complexity or issues. In general, there is some perceived value in this, but you must take into account issues such as parsing and enrichment. Also, the network model (assuming you have a network model defined) is going to be a lot more complex when using a single forwarding system rather than dispersed SmartConnectors from the log source. One of the biggest challenges here is with overlapping network ranges for example.

That said though, it does work within certain bounds. The simplest and easiest thing to use here is syslog data, but if there is any other log data, you need to carefully work out what is needed. For example, collecting from a DB system from the log source is fine into say LR or LL, but the data is no longer SQL data and something else. Therefore forwarding the data via the system that they have (usually Syslog) means that you will have to use a FlexConnector to read the data (since its not in a native log format we can process).

If you have 90% Syslog data and a few other things in there, its going to be pretty simple to do, but if you have a bunch load of custom log sources or non-Syslog data, this is going to get pretty complicated. I have found that a few customers have gone with Splunk or LL for log storage but then double collected the data - so sending from the log source to both Splunk / LL and to ESM. That way there is a consistent set of data going into both solutions and its easier to manage. Sounds bizarre and complex, but works out easier in the long run.

One final point though - have you taken a look at Logger 6.0? Its a lot quicker, faster, easier and simpler and we have a lot more functionality added. If you have seen Logger 5.3 or lower, you will know the performance - but think at least 4 times faster and in some cases 10 or even 100 times faster!!! Take a look at some recording that were done on peered loggers:

And some stuff on Logger 6.0:

https://protect724.hp.com/message/50614#50614

Hope this helps.

0 Likes
jtsapos1 Absent Member.
Absent Member.

Re: Log management solution(s)?

Paul, have you had a chance to look over the SPLUNK App for CEF to see what it's worth?

From what I've seen it looks like it will allow you to translate the SPLUNK output into a CEF format that works easily with ArcSight.

The drawback though is that you have to take each event coming out of SPLUNK one-by-one and do the translation via the SPLUNK app, you cannot just stream the data straight from SPLUNK into ArcSight.

Look like it's very time consuming plus I'm not sure if it will work for log formats other than Syslog.

Any thoughts on that?


0 Likes
pbrettle Acclaimed Contributor.
Acclaimed Contributor.

Re: Log management solution(s)?

I have struggled with time to get to look at this. I really want to see the Enterprise Security module, but since this is a purchased product, its a bit of problem to procure internally! That said, I have tried to get involved with some partner systems and see what we can do.

It should be better than the old transformation system they used to use, but so far I don't have any feedback as yet. Sorry, but no news from me.

0 Likes
justin.gruwell@ Absent Member.
Absent Member.

Re: Log management solution(s)?

I came to the conclusion that it is better to just fork the events at the connector like you said and send them to both. I have tried many times at several companies to get Splunk to play nice forwarding the events and every time it is more trouble than it is worth. This battle always goes hand in hand with the WUC vs Snare battle because whenever Splunk is involved Snare is sure to follow. My most recent battle was convincing the company to switch from Snare to the Windows Unified Connector and that was hard won but they did eventually come around and we are now in the process of eliminating Snare from our environment.

Lessons I have learned over the years...

Collect the events with an ArcSight connector whenever possible. Not Snare

Send the events directly to your ESM or logger. Don't relay through another product like Splunk.

0 Likes
Highlighted
dzuperku1 Absent Member.
Absent Member.

Re: Log management solution(s)?

Justin/ community,

Thanks for the replies.

In your experience collecting windows logs via WUC are you using the connector appliances?  if so, did you setup container add the WUC connector and pull the logs that way?  I've pulled windows logs in the past that way but found the limitation that Connector Appliances only allow for 6-7 container(maybe this has changed)? and each container should only have about 20-30 servers for each WUC connector (depending on how noisy the systems are). doing the math, one connector appliance should be able to collect about 210 windows servers?

I'm looking at pulling in about 4000 window servers, would Windows Event Forwarding (WEF) be a better option? It's my understanding that you can forward the event logs from the Win servers(via GPO) to one Win server and than have WUC pull the events from that server, it seems like you could have a single point of failure here.

Using WEF how do you ensure windows servers are constantly sending you logs in a ArcSight logger environment?

0 Likes
jtsapos1 Absent Member.
Absent Member.

Re: Log management solution(s)?

Justin, have you used the SPLUNK app for CEF?

I wanted to get some feedback if possible as we are going to be visiting that option in the very near future and I would love to get some feedback as to whether it's even worth it to attempt.

I agree that trying to integrate SPLUNK into ArcSight is probably more trouble than it's worth but some additional feedback on this would be really appreciated.

I could use the feedback to support the idea of using ArcSight by itself without all the extra headaches involved in the SPLUNK integration. We don't need to add this into the mix.

ArcSight alone is enough to keep us busy for crying out loud!

0 Likes
justin.gruwell@ Absent Member.
Absent Member.

Re: Log management solution(s)?

Putting the connectors directly on the appliance isn't a very good use of the appliances resources so I always recommend that customers use them to remotely manage their connectors and use a virtual environment for example to host the connectors. To get the best performance out of your connectors you need to deploy them as close to the event sources as possible. Using the WEC configuration allows you to have less integration targets but adds some additional layers of complexity that must be considered and these are outlined on page 21 of MicrosoftWindowsEventLogUnifiedConfig.pdf. We just purchased ArcMC so we will be using that to manage our connectors going forward.

0 Likes
justin.gruwell@ Absent Member.
Absent Member.

Re: Log management solution(s)?

I have not used the Splunk app for CEF because it wasn't around when our Splunk instance was. Now that Splunk is gone from our environment I sort of lost interest in it. I have done a bit of reading on it and a two things it doesn't seem to address is the timing issues and health monitoring. It's aim is to fix the formatting issues so you can use a standard ArcSight connector in a standard configuration but it doesn't seem to address any other the other issues with relaying events through Splunk. Two of the biggest problems with relaying your events through Splunk is the timing issues it creates with late events and the lack of device/health monitoring that you would normally have if you were getting it straight from a WUC. I can elaborate further if needed but it is my understanding that the Splunk CEF app only addresses formatting it doesn't address delivery, event time or health monitoring issues which plague that configuration.

0 Likes
michael.loy@hpe Respected Contributor.
Respected Contributor.

Re: Log management solution(s)?

The best strategy for leveraging WUC with WEC  is to install the connector directly on the WEC  aka subscription server.  this will reduce network latency, encrypt at the source and improve overall performance.  Also if you anticipate high EPS consider installing one connector dedicated to the Security logs and the other to the Application and System logs.  I have seen in some environments splitting it a bit further and daisy chaining one connector pulling the logs from a windows source such as Security logs then forwarding the to an upstream syslog connector where the network model is applied.  This may sound complicated but it works really well.  The purpose of this is to split the load between two connectors one is consuming the Security logs and normalizing and the upstream connector is apply the network model.  If you have a really large network model (above 1500 Zones) this should be considered.

0 Likes
jtsapos1 Absent Member.
Absent Member.

Re: Log management solution(s)?

Justin, thanks for this information.

Looks like there is potential for loss of Data Integrity (if not altogether loss of data) among other things such as performance monitoring of the system (Not Good).

If you come up with any more information to support this argument against sending SPLUNK data to ArcSight SEIM, please let me know. I would appreciate it.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.