Sentinel Collector SDK 2014 Updates: Debugger

over 6 years ago
Recently I posted a previous article in this series about the new Sentinel 2014 Preview SDK and its availability in a standard archive or via the public SVN repository. Today I want to go in-depth on one of the great new features that will likely cause anybody who has ever used the SDK to jump a little in excitement: the debugger. If you have built collectors in the past the workflow was something like this (grossly simplified):

Use the SDK to create a new project.
Code code code
Import into Sentinel or Log Manager system
Fire up the debugger from Event Source Management (ESM)
Setup breakpoints, watches, etc.
Generate events
Debug: Watch the debugger throw an error, or after a few tries, see something needing tweaking, etc.

The distance between coding and seeing errors in the debugger was vast, and full of manual steps requiring a full Sentinel environment to do anything. Pain, suffering, etc. This is the case with the currently-shipping 2011.1 SDK, but the 2014 Preview has a new debugger which is built into the SDK, and launches directly from the Eclipse plugin build process. As a result, the new workflow is something like this:

Use the SDK to create a new project
Code code code
Setup breakpoints, watches, etc.

This is, as you can see, a vast improvement. It is now reasonable to think that anybody with a laptop of any note (vs. a beefy workhorse) can do this kind of development, where before virtualizing a full Sentinel system, even if a small one, was required just to do debugging. There are other advantages to the ability to run code this way, specifically around built-in test cases, and ones that can be added by us, but that's left for another day.

I'd like to prove how this works with some instructions to really get it setup. To start with, you need the SDK, which is covered a bit in the previous article in this series: Sentinel Collector SDK 2014 Updates: Public SVN Repository In summary: download the SDK (bit 200 MB archive), Eclipse (if you do not already have it), and the Sentinel SDK plugin for Eclipse (tiny) from the 2014 Preview page, which is here under the 'Developer Kit' tab: If you use one of the Eclipse downloads on that page, it comes with the Eclipse plugin, so that saves those who are stuck on windows.

When launching Eclipse with the Sentinel SDK Plugin in place go to the Window menu, then click on 'Open Perspective' and then 'Other...' and from the resulting dialog choose 'Sentinel' and click 'Ok'. This should change Eclipse to a perspective that is optimized for by NetIQ engineering for developing Sentinel plugins, whether collectors for parsing data form event sources, reports for viewing those data in interesting ways, or whatever else is interesting to you.

On the left-hand side is the 'Sentinel SDK' view (tab/section) which has a few expandable sections for the various types of plugins supported by the SDK.


Since we are covering the debugger in this article, that implies that we'll be using Collector plugins, so expand that area. Default options within include 'Generic', 'Generic Test Collector Vendor', and 'NetIQ' but we can create our own by right-clicking on 'Collectors', choosing 'New Collector Plug-in', and then stepping through the prompts. The Vendor Name is who owns the product for which this collector is being written, and examples are usually 'NetIQ', 'SUSE', 'Novell', 'Amazon', etc. 'Product Name', of course, refers to the application for which this collector plugin is being developed. Examples include things like eDirectory, bind, sshd, and EC2. Finally the Category drop-down lets you choose what kind of application this is, such as an operating system (OS), Database, E-Mail System, etc.


After finishing this step Eclipse should go away for about one second and build the directory structure for your project. Under 'Collectors' is now the new vendor, and within that is the product, and within that are the directories that will be used for storing collector, test, and sample data files for the rest of the lifetime of the collector.


The Console view at the bottom of the screen should show build process information which may be useful to identify where in the filesystem everything is, and how to use the build system manually, etc.:

Buildfile: /home/ab/code/sentinel-plugin-sdk/current/sdk/build/build-new.xml


[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[xmltask] Cannot append values to properties
[echo] Type in the vendor that sells this Event Source; follow the vendor's website name
[echo] as closely as possible. Avoid using '_' and other special characters if possible.
[echo] Examples: Cisco, Oracle, SecureComputing.
[input] skipping input as property vendor has already been set.
[echo] Type in the current product name using the format:
[echo] Product Name Ver (Ver is optional). Avoid special characters.
[echo] Examples: PIX, SiteProtector-FW, Oracle 10g
[input] skipping input as property product has already been set.
[echo] What Category of Event Source is this? Valid values are:
[input] skipping input as property pcat has already been set.
[echo] What License Type is this event source? Valid values are:
[echo] I, II, III, IV, V, NA
[input] skipping input as property type has already been set.


[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1
[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/sample
[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/archive
[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/research
[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/tests
[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/tests/data
[mkdir] Created dir: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/tests/results
[copy] Warning: /home/ab/code/sentinel-plugin-sdk/current/sdk-local/2011.1/Collector/var does not exist.
[copy] Copying 17 files to /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1
[copy] Warning: /home/ab/code/sentinel-plugin-sdk/current/sdk-local/2011.1/Collector/var/docs does not exist.
[copy] Copying 4 files to /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1/docs

[copy] Copying 3 files to /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1
[copy] Copying 1 file to /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1
[copy] Copied 1 empty directory to 1 empty directory under /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1/pack
[copy] Copying 1 file to /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1/pack

[propertyfile] Creating new property file: /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1/

Create New Collector Plug-in:
Total time: 1 second

Notice at the end that, in all caps, we are told the build was successful. This is normal for all types of builds within Sentinel, so you can know pretty quickly (within a few seconds) whether or not your build is valid. If not, then trying to import the plugin into Sentinel, or using the debugger, is a futile effort, and hopefully the failure message includes details about files and line numbers to check, but at least from the start you have knowledge of which way things went.

At this point we are technically ready to start debugging. Sure, we have a blank collector plugin that should really do very little, but at least we're that far and can test the debugger. There is technically one last thing that we must do to work around the default configuration of a collector plugin, and that is setup the connectionMethods.xml file properly. If you do not do this then you'll start the debugger (or try to), have your processor warm things up for a few seconds, and then nothing will happen. The build within Eclipse will give you a message telling you that the debugger "window will appear momentarily" so you may wait for a while, even though this should take seconds, not multiple minutes. If you run the debugger manually you find this error coming out, leading to the need to setup the allowed connection methods:

2014-12-24 04:45 INFO: esecurity.base.configuration.MetaDataManager init Initializing metadata, container name null
2014-12-24 04:45 SEVERE: Unknown unknown Error running collector in directory /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412231432-internal-test/plugin with input ../tests/data/variety.dump, output /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412231432-internal-test/, and output compare null.; Exception The collector package contains duplicated connection mode internal name 'SYSLOG' for method 'SYSLOG'.; esecurity.ccs.comp.evtsrcmgt.collector.util.testbed.CollectorPackageChecker$InvalidCollectorPackage;
2014-12-24 04:45 SEVERE: Unknown unknown esecurity.ccs.comp.evtsrcmgt.collector.util.testbed.CollectorPackageChecker$InvalidCollectorPackage: The collector package contains duplicated connection mode internal name 'SYSLOG' for method 'SYSLOG'.
at esecurity.ccs.comp.evtsrcmgt.collector.util.testbed.CollectorPackageChecker.checkConnectionMethods(
at esecurity.ccs.comp.evtsrcmgt.collector.util.testbed.CollectorPackageChecker.checkCollectorPackage(
at esecurity.ccs.comp.evtsrcmgt.collector.util.testbed.Main.main(

The important part is "collector package contains duplicated connection mode internal name 'SYSLOG' for method 'SYSLOG'" which basically means that for the SYSLOG connection method you can only have one type of SYSLOG connection supported. If you open the connectionMethods.xml file (under the 2011.1 section for this collector plugin within Eclipse) you'll see a lot of connection methods checked by default, even though it is likely you will only be supporting one of them. Syslog is a great option because it's simple to setup for applications, reliable when using TCP, able to be encrypted, and simple to setup within Sentinel. To make routing of events to the appropriate collector as easy as possible the connector will let the collector plugin creator (you) define either an AppId, or a pattern (regex) for the data to match appropriate events to the appropriate collector. This is where our problem shows up, as both of these are registering as the SYSLOG connection method. The simple fix for this is to uncheck the invalid options, or Syslog entirely, and then we're on our way. For my testing, I uncheck 'Line Output' and 'Matching Rule' since, ideally, using 'Application ID' is the best option.

Moving along, save any changes made to connectionMethods.xml and then right-click on '2011.1' and choose to 'Debug'. The Console output within Eclipse shows something like the following:


[xmltask] Setting standalone
[zip] Building zip: /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test/

[echo] =========Setting variables===========
[echo] cntroot = /home/ab/code/sentinel-plugin-sdk/content
[echo] sdkroot = /home/ab/code/sentinel-plugin-sdk/current/sdk
[echo] plugindir = /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test/plugin
[echo] outfilepath = /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test
[echo] tmpltver = 2011.1
[echo] outfilename = /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test/
[echo] infilename = ../tests/data/variety.dump
[echo] Initializing collector debugger...
[echo] classpaths used:
[echo] /home/ab/code/sentinel-plugin-sdk/current/sdk/2011.1/Collector/static-J
[echo] /home/ab/code/sentinel-plugin-sdk/current/sdk/build/lib/collectorutil-libs
[echo] Debugger started, window will appear momentarily.
Total time: 3 seconds

Notice that, again, we have BUILD SUCCESSFUL showing up in all capital letters to let us know that the build is okay. Also a message at the bottom just before that letting us know that the Debugger is started and will show up momentarily (for me, after what feels like ten seconds). Other notable lines include a file of events being used by the debugger, which I have not yet created but which is there, empty, by default as 'infilename' pointing to ../tests/data/variety.dump and as this is a relative path it is probably useful to know that the current working directory (CWD) is the '2011.1' directory of your collector plugin project; in my case, that would be /home/ab/code/sentinel-plugin-sdk/content/dev/collectors/ABSoftware/ABProduct/2011.1 which is likely something you do not need to know except to find that variety.dump file, and maybe for running the debugger over and over from the command line. Also notice that there is an 'outfilename' defined, which is where JSON-based event output from the collector is written; this does not exist until you actually send something there, but in my case it will be created, at that time, here: /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test/ Also showing up are some classpath statements from the SDK side of things, so you can see what is potentially being used by the collector plugin at runtime.

I mentioned above the possibility of running the debugger, once built, over and over for efficiency. It's not a big deal to just start the debugger again from Eclipse, but it does rebuild each time (using some disk space), and perhaps I want to avoid wasting those three seconds, or perhaps I want to be totally sure that nothing is changing between each invocation of the debugger, or perhaps I want to start it within another JRE, or perhaps I want to attach deeper Java troubleshooting tools (jstack, etc.) to see what is happening when things do not work correctly, or perhaps I just want to know more about the process when it is actually running. Using 'ps aux | grep java' to pull out the running command, I see this (along with other 'java' processes, so find the right one based on class or path) on my system:

ab 25390 1.8 8.7 6153352 1437324 ? Sl 04:47 0:16 /home/ab/apps/jdk1.7.0_67/jre/bin/java -Duser.home=/home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test -Djava.util.logging.config.file=/home/ab/code/sentinel-plugin-sdk/current/sdk/build/lib/collectorutil-libs/ -classpath /home/ab/code/sentinel-plugin-sdk/current/sdk/2011.1/Collector/static-J/collectorutil- esecurity.ccs.comp.evtsrcmgt.collector.util.testbed.Main -dir /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test/plugin -in ../tests/data/variety.dump -out /home/ab/code/sentinel-plugin-sdk/content/build/Collector/ABSoftware_ABProduct_2011.1r1-201412240446-internal-test/ -Debug

Remember that the CWD is the '2011.1' under the 'dev' directory structure, and so that is where you must put yourself if you want to re-call the debugger manually from the command line interface (CLI). Other than that, nothing else is needed to restart the debugger other than the command above. From this command you can see some other good details like the actual classpath showing all included JARs and directories, which is very useful for troubleshooting, a list of properties files, paths for everything related, the class used to invoke the debugger, the JRE used for that invocation, etc. Again, if you want to run it manually without Eclipse, be sure you're in the correct directory before you paste in the command appropriate to your system.

Finally we are here with the debugger running just like we would expect from within Event Source Management (ESM) except without any dependencies on a full, running Sentinel system.


As you would within ESM, the next step is to hit the Play (sideways, right-pointing triangle) button and then set any breakpoints, watches, etc. To find some bit of text quickly use Ctrl f as usual and type in the sought string to step through results. One breakpoints are in place, click the Play button one more time and you're off to the races. At this point maybe nothing will happen, and if that is the case a few things could be wrong:

  1. Most-likely: You did not setup your variety.dump file correctly in your collector plugin's 2011.1/tests/data directory; sure, one is there by default, but it is basically empty so this is akin to starting the debugger within ESM and then forgetting to turn on any event sources nested beneath that collector, leaving the collector dutifully checking for events regularly but never receiving anything to parse. It will be a long, as in, infinite, wait, so go ahead and shutdown the debugger and fix this.

  • You set a breakpoint somewhere that the execution logic never reaches, such as on a catch block that is never hit, or a line within a conditional that is never reached. As a result, the collector may have parsed thousands or millions of events while you were waiting for it to reach a line that, apparently, will never be reached. Logic errors, data errors, and syntax errors all contribute to this, and if you know your variety.dump (#1 above) is sending data, then this is the next-most-likely issue.

To fix #1, the empty variety.dump file, usually involves having Sentinel nearby, or at least some great sample data. The reason is that this file is made to simulate the connector, whatever that connector will eventually be, so the data should appear as if they came through the appropriate connector. The easiest way to start building up a collection of events in this form is with Sentinel or Log Manager using a Connector to create the Connector Dump, which is basically a file written by the connector as events are sent in; the file format is JSON and it looks very similar to a raw data export from Sentinel or Log Manager, but it is not exactly the same. Discussions of how to setup a Connector Dump are covered in the Sentinel SDK under 'Collector Development Topics' and the 'Collector Development Guide' which is good reading overall for those starting out. To summarize: Go to ESM, choose the appropriate connector node, right-click and Edit, then click on the General tab and check the checkbox next to 'Save raw data to file'; do not forget to fill in the corresponding field with the absolute path to an existing directory, plus a to-be-used file (created automatically if not there) for the dump; be sure that Sentinel has rights to write to this file in this directory. Send events to this connector and watch the resulting file grow on the Collector Manager (CM) system involved. When done, uncheck the checkbox next to the 'Save raw data to file' option so that you do not fill your hard drive accidentally.

With a connector dump generated, copy/move that file to the 2011.1/tests/data directory for your collector plugin. Restart the debugger (via Eclipse or the CLI), setup breakpoints and now you should see events being parsed; look in the 'outfile' as defined in the build and see the parsed results show up there too even if you did not set breakpoints in the debugger.

One neat feature of the Eclipse-started debugger is its performance; using a connector dump within ESM for event replay is relatively slow, apparently because the JSON is de-serialized one event at a time and this is a slow operation anyway. Using the Eclipse-started debugger on my laptop I can push through around 5,000 lines (events) every second with a production collector, which may not mean anything depending compared to other collectors depending on hardware and parsing requirements, but it's orders of magnitude faster than ESM's debugger, and not too bad for a single-threaded collector.

If you have been doing any collector development in the past for Sentinel I hope this shows you the huge benefit that comes from using the 2014 Preview version of the SDK with its new Eclipse plugin and functionality. This feature, alone, makes a transition worth it, even though technically the documentation around it says to be careful/wary/afraid near production systems. Having used this for production systems already, it seems to be fairly solid, other than a few little quirks covered above (SYSLOG method) and which will be covered in future articles in this series.

Happy computing!
Comment List
  • I do not see a big reason why not except I do not know how, with the SDK's debugger, you can put the collector into custom mode. I presume that's possible, maybe even just a change in the package.xml file, but I have never tried that. I'm assuming in this case that you'd use the ability to start the collector from the command line without rebuilding the collector in order to avoid splitting apart a finished collector into its base pieces and parts.

    If you get it working, please write back. I'd like to see that work, as it's a place where troubleshooting could still be painful.
  • Great article, I haven't had time to try out the new SDK yet.
    Do you know if you can debug the "custom.js" file the same way, i.e. the customization of an existing collector?
Related Discussions