Anonymous_User Absent Member.
Absent Member.

How to set up automatic syslog configuration?

A colleague recently asked: "I am wondering whether I need to specify
the "appid" (something like "nqcg") when I connect the collector to the
syslog event source. If yes, do I specify it in the "Syslog Message
Filter" dialog? It's asking for a message filter pattern. Do I put in
something like nqcg:*? Or, do I specify my "appid" as the "Applications"
property value in the Syslog:Map Output(appid) connection mode XML node
in the "connectionMethods.xml" file?

The answer to this question lies in the Syslog Connector documentation,
mostly, and some samples are provided in the template, but it's left up
to the developer, somewhat, to figure out how to put all the pieces

DCorlette's Profile:
View this thread:

3 Replies
Anonymous_User Absent Member.
Absent Member.

Re: How to set up automatic syslog configuration?

OK, so first some background:

- The Syslog Connector is pretty sophisticated - it will automatically
apply a proper syslog header if the event source does not provide one,
and will also pre-parse out a number of key pieces of information from
syslog headers (generated and provided) to provide metadata about the
event records received.

- The Connector will, for example, extract the facility etc, date/time,
and the Observer information from the syslog header. It will also figure
out which system is sending it the data (which may or may not be the
same as the actual Observer) and report that as well. All this metadata
is attached to the actual received record in various record fields like
's_syslogRelayIP' etc (see the Connector doc for details, which vary
slightly between different versions of the Connector).

- In addition, the Syslog Connector will take a peek at the first chunk
of data in the actual record body (usually up to the first ':'
character, and based on that set the 's_AppId' field. This often
corresponds to the name of the application that generated the actual
event record.

- So, the Syslog Connector extracts all this useful stuff - what does
it do with it? Well, this is where the Collector comes in - essentially,
the Connector can "register" to handle specific sets of data. There are
three layers here:
1) The Collector can declare itself to be the
'UniversalSyslogCollector'. In this case, all event records that are
otherwise unallocated will come to this Collector. Obviously, this
should only be one Collector in the system at a time, and is typically
our provided Generic Event Collector.
2) The Collector can register to handle specific applications. In this
case, for each and every event received if the Syslog Connector detects
the associated application ID, it will route that data to the Collector
that knows how to handle that data. If necessary, it will auto-create a
new Event Source node in ESM to represent the single application running
on the single Observer source that is sending data to that Collector.
3) The Collector can supply a regular expression matching pattern to
help the Connector identify the Collector to which Observer data should
be sent. In this scenario, all input records are checked against the
regex until the match is found; then all the Observer's data is sent to
the associated Collector.

- The way that these nest can be a little confusing, so I'll explain a
bit further here in terms of the chain of actions that take place when a
new event source starts sending data to Sentinel:
a) When a new connection is detected from a new event source
(determined from the syslog header hostid, which is either an IP or
hostname), a new Event Source node is immediately created in ESM to
represent the new source (well, actually, this is determined by policy
on the Event Source Server). This node is attached to the Generic Event
Collector initially, since we don't yet know anything about the source.
b) Inbound records will be compared against all the available
registered regex matching patterns for all installed Collectors; if a
match is found, then the Event Source is determined to be the product
for which the associated Collector was written - so for example it might
figure out that the stream of data coming from a particular server is
SUSE Liinux data. The Syslog Connector will then instantiate a new
instance of the SUSE Linux Collector (if necessary), and *move* the
existing Event Source node to be handled by that Collector.
c) At the same time, all inbound records are compared against the set
of registered application IDs, and if a match is found, another Event
Source node is created and attached to a new instance of the associated

It might not be obvious why the system works this way, but the reason
is that we often have scenarios where a particular application can
actually run on several different platforms. So, for example, Apache Web
Server can run on both SUSE Linux and Solaris - we have Collectors for
all three. A new event source will start sending data such as logins,
account creates, etc and eventually the Syslog Connector will figure out
"hey, that's a Solaris box" and route the data appropriately. At the
same time, however, if events from 'httpd' (Apache) arrive intermingled
with the same event stream, then they will be routed to the Apache
Collector instead of the Solaris Collector.

DCorlette's Profile:
View this thread:

Anonymous_User Absent Member.
Absent Member.

Re: How to set up automatic syslog configuration?

Now, as a quick followup, let's say you're writing a Collector for a new
source. How do you write the Collector s.t. the Syslog Connector will
automatically configure your Collector for the source?

First off, when you create a new Collector, samples of all the possible
configurations are provided in the connectionMethods.xml file that is
copied into the dev directory. You will need to review and configure the
method you want, then *delete* the others.

In general, the UniversalSyslogCollector mode is out - that's used by
Generic Event, and unless you're planning to replace that Collector
you'll create a conflict.

You can use 'Applications' if your event source consistently attaches
one or a small set of appid's to the front of its event data. Most *nix
apps will do this, for example; you'll see:
httpd: blah blah blah
sshd: blah blah blah
(sometimes with PIDs and whatnot which are ignored for this purpose).
Even Cisco does this, something like:
%ASA-3-14525: blah blah blah
(the 'ASA' is the "appid")
If this appears to be the case, then specify the relevant appid(s) as
part of the 'Applications' mode and you'll be all set. You do need to
make sure these don't conflict with any existing Collectors' registered
appids, however - the Syslog Connector will warn you if this is the

If that doesn't look like it will work, then you'll have to use the
UniqueMatchingRule. Look through the set of events you're likely to
received and try to identify a pattern that you'll only ever see from
that particular type of source. Note that the pattern doesn't need to be
in every event - as long as it's in one event that's generated once in a
while, eventually the Connector will see it and allocate your Event
Source correctly. If you can't guarantee that it will be generated
fairly often, you can come up with a process to manually generate the
event, and have the user perform that procedure as part of the device
configuration. We do this, for example, with SUSE Linux because from
normal event data it's quite hard to tell the difference between SUSE,
Red Hat, Ubuntu, or whatever, so we have them install a local script
that reads from the /etc/SUSE-releases file. The script also captures
console logins, so they have more than one reason to install it.
Again, you need to be careful that your regexes aren't too "loose" e.g.
they'll match data from other systems, or too "tight" meaning they'll
rarely match anything. This is a judgement call, really.

DCorlette's Profile:
View this thread:

Anonymous_User Absent Member.
Absent Member.

Re: How to set up automatic syslog configuration?

By the way, if you are processing some existing logfile, writing your
own application or script, and so forth, you can use this great
technology to make it really easy to integrate your app with Sentinel.
Simply have your app/script/etc inject a unique appid string into the
beginning of the syslog message body, and then configure the associated
Collector to register for that appid.

We use this, for example, to read from httpd logfiles - we have a
little script that reads the local logfiles, as follows:
/usr/bin/tail -F */logs/access_log */logs/error_log | awk
'{print("httpd: ",$0); fflush()}' | /usr/bin/netcat $cmip $cmport

$cmip and $cmport are parameters that are set to the IP/port of the
Sentinel Collector Manager, and the 'awk' script just injects an 'httpd:
' header in front of every line. On the Sentinel side, the
connectionMethods.xml is configured with:
as a property of the syslog connection method/mode.

You'll note that this isn't real 'syslog' data - this is really just
arbitrary line-oriented data sent over TCP to the syslog port, but it
works just fine...

DCorlette's Profile:
View this thread:

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.