Syslog Connector question - Two part question
This is a two part question:
Part 1 -
I need to send syslog output from SPLUNKs syslog version of 3164 (bsd syslog) to the ArcSight syslog connector so I need to find out the RFC Syslog format that the ArcSight Syslog connector is using and find out if there will be any compatibility issues with that?
If the ArcSight connector uses 5424 Syslog format, does it accept the 3164 bsd syslog format from another device ? Is there backwards compatibility or not? thanks
Part 2 -
If this is a viable solution, then I would like to know what would be the correct choice in this case:
What is better for the ArcSight syslog connector in this scenario, syslog pipe or a syslog file?
Does it matter which one you use? Any help would be really appreciated. Thanks
It's a viable solution for the most part. We're doing a bit of this, though I can't say that we concerned ourselves with the RFC, just had Splunk write out to syslog, then read the output (via syslog daemon, but also syslog file).
I hate to say it, but it more depends on what you're gathering logs from. Windows and Unix via the Splunk UF are going to be a little different than using Splunk to pull a DB and then sending syslog. Our team has deployed a lot of custom filereader to match the custom input, but we've been happy with the result so far.
We are using SPLUNK heavy forwarder to send the data from SPLUNK directly to the ArcSight syslog connector.
We are trying to convert all logs to Syslog format and then send it out through the Heavy Forwarder as one data stream.
Do you recommend syslog daemon or file in this case?
Is the syslog file primarily for pulling from a database?
Syslog daemon. I haven't had good luck with the syslog file reader at any kind of volume.
We also write to syslog file for default archive and to replay events, but not for any technical reason.
We wrote a custom connector to read it into Arcsight though. I'm not sure if the default windows connector will handle it; actually, I'm not sure the Windows connector logic is in the syslog default connector at all, but I didn't look/test.
I just setup the Syslog daemon connector but I'm getting "Connector parameters did not pass the verification with error [0:Unable to bind to port to 514]
The options I set for the install are:
Network Port: 514
IP Address: (ALL)
Protocol : Raw TCP or UDP
Anything incorrect in the setup? Any ideas on why I'm getting that error? Thanks
Also, I have the syslog daemon installed on the manager to listen on port 514. Is that causing a problem?
Is it better to install the daemon on another device and then forward to the manager?
Ok, forget all the above. I got the syslog daemon connector installed. so now I will be checking to see what kind of output we're getting from SPLUNK.
In order to bind to a non-ephemeral port, you must have root privileges. You need to install your agent as the root user:
./arcsight agentsvc -i -u root -sn syslog
Then restart the agent via the init script.
Thanks, I got the connector installed and a live feed coming in from SPLUNK, however now it's looking like a parsing problem in which case a parser will probably have to be created. Does anybody have anything they can share on this? I would appreciate the assistance. Thanks.
So far we are getting mainly firewall traffic and some Unix OS traffic, nothing else.
We don't have it set to "send cooked data". We are just sending raw syslog format from SPLUNK.
I don't know if setting it to "cook" the data is going to make a real improvement?
Another question on this is whether an outputs.conf file would be a good idea to create on the SPLUNK side?
I don't know if you guys have tried that option...
Geez, I guess I'd have to see the raw output in order to see how it compares to what we read. I don't think cook will make a difference in this regard.
We're actually using syslog-ng in the middle of it all, which grabs a copy for posterity, then forwards along to the arcsight smart connector, so we can see the messages in a file, which makes writing the syslog connector sdk a lot easier.
We tried sending the raw syslog from SPLUNK to the Syslog daemon connector and the connector cannot parse it properly. Splunk claims their syslog format is RFC 3164 compliant but we're still getting only the raw syslog message and it's all going into the Event Name field. By any chance do you have a copy of the sdk file you used that you could share? I would really appreciate it.
Btw, did you create a regex file and basically include it as part of the configuration of the Syslog daemon connector? There is a part in the connector configuration documentation which discusses this:
Custom Regex to Capture Source
Custom regular expression to capture source; the capturing group indicates the location of the source IP or host name. This regular expression needs to match the entire raw syslog event, and have at least one capturing group, which tells the connector how to find the source address. For example, this regular expression would find everything between the words “before” and “after:”
For the following event, that regular expression would capture the IP address 192.168.1.2:
Hello there before192.168.1.2after and goodbye
Is this what you did when you created the regex file? Did you include as part of the Connector configuration during the setup?
Well, now you're talking custom connectors, which is pretty situation independent. I actually am not sure what kind of events you're sending, so I couldn't just post a snippet of what we have.
For some connectors, we do use the regex connector, but each of the feeds may have their own sdk file. Most of the time, when we are getting the feed from Splunk, it requires us to customize it.
I realize that isn't a lot of help, but without knowing what your messages look like, I can't add much.