Syslog Connector question - Two part question
This is a two part question:
Part 1 -
I need to send syslog output from SPLUNKs syslog version of 3164 (bsd syslog) to the ArcSight syslog connector so I need to find out the RFC Syslog format that the ArcSight Syslog connector is using and find out if there will be any compatibility issues with that?
If the ArcSight connector uses 5424 Syslog format, does it accept the 3164 bsd syslog format from another device ? Is there backwards compatibility or not? thanks
Part 2 -
If this is a viable solution, then I would like to know what would be the correct choice in this case:
What is better for the ArcSight syslog connector in this scenario, syslog pipe or a syslog file?
Does it matter which one you use? Any help would be really appreciated. Thanks
No, getting routers and switches direct.
Windows, yes, but it's a lot of work (or was a lot of work for a teammate).
Can you post a line or two of your syslog feed for the routers/switches? I can compare it to the raw syslog that is getting parsed.
Just a quick question, why do you think that the ArcSight SmartConnector should parse MS Windows feeds coming from Splunk? As far I know there is no syslog connector for MS Windows so it wont ever parse windows feeds as long you create your own FlexConnector.
For Cisco devices it should work, but before that you have to check the SmartConnector documentation for specific Cisco device whether it's supported or not...
And "RFC 3164 format" is the correct format accepted by syslog SmartConnectors::
Following is the example format which can be accepted by the syslog SmartConnector, so you can verify if your time stamp is showing up in same format, if not, then you will have to apply some custom template to change that into following format.
Apr 2 12:47:26 server-name snort: [137:1:2] (ssp_ssl) Invalid Client HELLO after Server***************
Feb 6 08:41:56 server-name
Mar 17 14:44:33 server-ip
Another thing is that you can use Splunk APP for CEF which is specially build for ArcSight.
If you use this Splunk APP then everything is going to parse by default.
The Splunk App for CEF launched last week! Learn all about it so you are prepared to discuss the new features with your customers and prospects. Below you find contact details to our Splunk Channel Sales Managers who are eager to support you with webcasts, presentation material, more information or join meetings.
The Splunk App for CEF deepens our ability to embrace and extend HP ArcSightenvironments. It provides security analysts with greater control over their tools, and helps them enrich data for more business-centric decisions. Security teams can now use Splunk’s powerful security intelligence features to improve the data that goes into ArcSight.
For more information please contact:- Anders Johansson (Channel Sales Manager Nordics & Baltics)firstname.lastname@example.org
- Claudia Crawfurd (Channel Sales Manager Benelux)
Use Data models to Aggregate, Enrich, and Filter
Splunk App for CEF uses Splunk Enterprise’s powerful data modeling features to aggregate disparate data sources, enrich events for better analytics, and select specific data to send forward to external SIEMs. This allows security analysts to take advantage of Splunk’s industry-leading analytics and data management from within familiar, legacy toolsets.
Easy to Use Search Builder
A new GUI allows administrators to easily map fields from Splunk’s data models to specific Common Event Format output streams with a simple, point-and-click web interface. Filtering searches are easily created, edited, tested, and managed with theSplunk App for CEF.
Based on an entirely new backend engine, the Splunk App for CEF handles over ten times as much data as earlier systems for sending data to ArcSight, while offering far more flexibility for data enrichment. Improve raw data management, correlation, enrichment, and aggregation withSplunk without disrupting sensitive managers, sunk cost projects, or change-averse end-users.
How you position this to your customers?
· You´re enabling integration with an existing ecosystem to make it easier to embrace Splunk, not competing with the legacy products.
· This is about augmenting data, not just passing it through. The point of the app is to enable Splunk value for your customers who have to use non-Splunk front-ends.
· Embrace and Extend because Splunk already winners at Rip and Replace. Customers who don’t have to use non-Splunk front-ends are dropping them to use Splunk instead.
If you have any technical questions please contact our Pre-Sales Team at:
"Just a quick question, why do you think that the ArcSight SmartConnector should parse MS Windows feeds coming from Splunk? As far I know there is no syslog connector for MS Windows so it wont ever parse windows feeds as long you create your own FlexConnector."
I don't exactly agree with this solution but the decision was made from higher up on sending the data from SPLUNK to ArcSight.
My manager likes SPLUNK for logging and it's ease of use and wants to send only critical data to ArcSight for security monitoring...he likes SPLUNK but he doesn't want to give up the real-time monitoring we get from ArcSight.
We don't have enough problems managing ArcSight by itself, so we need to add another system to the mix to make things more interesting...
SPLUNK app for CEF looks like a good option but we would like to try getting the feed to ArcSight in more of a real-time mode.
Splunk app for CEF will reduce the real-time capability. Events will probably not be received in a timely enough fashion to constitute "Real-Time" as it is when data is streamed directly to ArcSight from device logs.
There will be a significant delay from the time the event occurs to the time it reaches ArcSight for monitoring. We are trying to avoid that kind of situation if possible.
I suspect the issue with the Cisco devices is that Splunk isn't writing the syslog message the same way that Cisco writes them. It may be mucking about with the header (suspect this is the case), but without comparing the two message feeds directly, it's impossible to tell. In our installation, we send to syslog-ng, which then sends to Arcsight via TCP, and writes to file. The file is then read by Splunk (Splunks file reader is pretty fast).
I don't know if Splunk writing a CEF message is that much of a time difference, but I do know that we had to write custome sdk files to install in the syslog daemon smart connector to get it into Arcsight, since we didn't go the CEF route.
I've actually no experience with syslog pipe, we're using syslog-ng to send to syslog daemon.
As commented by a few people here, be VERY CAUTIOUS with the Splunk App.
Splunk is great at what they do, but they also have a nasty habit of explaining that things are far easier than they actually are. I take a good example here:
A new GUI allows administrators to easily map fields from Splunk’s data models to specific Common Event Format output streams with a simple, point-and-click web interface.
Ah, so actually its just a tool that hides the older functionality that they have had for years and puts it in a GUI? Great idea and I am NOT knocking it, but its certainly not going to solve everything here. What is being provided is a way to forward data from Splunk in a format of your choosing that is effectively CEF. Now, if this is then subsequently read and processed by ESM, great. But unfortunately there is no guarantee.
ArcSight SmartConnectors are far from being perfect, but they have a lot of work, time and testing to get to where they are today. I take Cisco as a great example though. They have changed their formatting something like 3 times in the 7+ years that I have been dealing with their logs, and they are some of the more consistent log sources! If you manage to map it out correctly, it will work for that version / log source / format. Should this change, it may result in the forwarding not working and needing changing, which will cause issues with older log sources.
Additionally, please note that anything that is received by Splunk from a structured log source is unlikely going to work. For example, anything that comes from a database or a proprietary format (such as Windows) gets processed and received by Splunk in a structured format to start with. Splunk then processes it and forwards it on in CEF. Unless Splunk has managed to retro-engineer all of the processing involved here, I seriously doubt that they will map the files automatically. This means that you need to process and map this yourself and that will take time and effort!
Also, one point on Windows for a second - Microsoft took the idea of what they call "code pages" for internationalization some time ago. A nice idea and in general it works pretty well. A particular message generated internally is then mapped to a code, which is then applied to a code page where the internationalization text is used and data is entered into particular fields. Clever, but I take French for a good example - US / UK English has the source and destination username in source -> destination order. For the French log message, its reversed in destination <- source. Makes more sense in French, but a massive headache for processing log data. Internationalization of structured data is a massive issue and another issue to be aware of when we are looking at forwarding from any log source.
Don't get me wrong, Splunk has this issue with every other SIEM vendor too. We all want to receive the data in as close to the native format as we can. There are options (such as Snare for Windows logs), but it adds a layer of complexity that is not always visible. Q1, McAfee and everyone else has this issue because we all need to have structured data for us to process in a consistent and reliable way. Mapping it manually is OK, but just be aware of what is required and the administration and management that this will need. The new Splunk App is a step forward and embraces CEF better than ever before. Just be aware thats all.
Saying "everything will parse by default" is technically probably accurate, but might make things sound more straightforward than they are, so I felt I had to comment.
It won't "just work". The first issue is that Splunk does not add any categorisation, so if you are using categorisation in your content (and most foundation content plus much customer/PS content does), then it will not fire. Some dashboards and most compliance reports will be empty. etc. The Syslog Connector receiving the CEF does not add any categorisation either, since this is meant to be done by the device-specific connector.
So - everything that you forward from the CEF app in Splunk, you will need to either categorise yourself again manually, or dig out and maintain the map files for that device type from the original ArcSight connector, to the receiving CEF connector, so that the categories are re-added.
Operationally, I see the risk as being that someone will install the Splunk forwarder for CEF, map fields from Splunk CIM to CEF, test that it works, and then let it run. At some point in the future, a format or mapping will change, and someone will probably check and maintain that in the Splunk indexer, but not update and test the Splunk to CEF app integration and mappings, nor verify that the categorisations are still OK. For each such change that takes place, ESM will receive less and less of the data it was expected to be receiving, with the result that any given correlation rule, dashboard or report will not contain what it's meant to.
Neither ArcSight nor Splunk will support the data mapping or mapping maintenance; the LM/SIEM administrator takes on this responsibility when they build the integration.
Having said this, if you feel more comfortable maintaining the integration and it's a manageable number of well-understood events, then it is a handy way of forwarding data from Splunk to ArcSight.
Thanks for the offer but I cannot post anything from our logs since we are in a classified environment. Is there something else that you're running from Splunk to ArcSight that I can look at? Windows for example?
Don have much experience with Splunk but I can tell you that ArcSight Syslog CEF smartconnector will accept syslog message in the 3164 bsd format. Anyway, the parser for CEF will not reference the timestamp and hostname inside the rfc3164 syslog header.
The answer from Damian above is full of insights. We have other technologies that send CEF syslog directly out of the box and we manually have to write categorization file inside Syslog CEF smartconnectors and maintain them. For us, its a mood point as theses technologies are not supported by arcsight, anyway we needed to write theses files.