Sentinel Collector SDK 2014 Updates: Public SVN Repository

Sentinel Collector SDK 2014 Updates: Public SVN Repository

Working as a consultant on things Identity, Access, and Security I get to meddle with some really great products, and Sentinel is among those. Recent work building some customized collectors (for custom applications) has helped me find a few tidbits that I wish would have been more-obvious. Perhaps others, via this series of articles, will find the shared information to be useful, but if not at least it provides a great confirmation of NetIQ's willingness and ability to enable customers and partners with the tools necessary to extend their investment as much as possible.

Today I am focusing on the public SVN repository for the Sentinel Plugin Software Development Kit (SDK). If you have not used this before, particularly with the public 2011.1 version, it's probably time to try. If you have used it before, with the public 2011.1 version, it's definitely time to at least start looking at the preview release from this year (unofficially referred-to as the 2014 Preview). This series will be covering this preview and some of the great improvements which have some to it, and how they can be used for serious work in parsing the security data we receive from critical applications.

First, a bit of background may be in order so that we all start near the same point. The Sentinel product, developed by eSecurity, was acquired by Novell back in 2006, and at that time the Sentinel 5.x product line was moving ahead. The collector language was a proprietary one and, while functional, it was not exactly wonderful. The tool to deploy collectors to Sentinel was called the 'Collector Builder' and was a crash-happy tool that relied on an inferior operating system just to run. There may have been tools to package collectors from their source files in the proprietary language to a plugin able to be used by Sentinel, but at the time I never saw one or documentation on how to use one. The Novell acquisition led to newer versions of Sentinel which ran on Linux fairly quickly, and before too long a new major release (6.0) came alone with a vastly-revamped infrastructure for collector and connector management, Event Source Management (ESM).

ESM brought in the ability to use plugins as they should be used, as something to be instantiated for a specific purpose, very much analogous to the class/object paradigm with which we are likely familiar from object-oriented programming. ESM provided simpler, cleaner, and faster access to multiple Collector Manager (CM) systems which were all part of a larger Sentinel environment designed to scale out to handle large loads from thousands of event sources. Accessing and modifying the properties of collectors, connectors, and the nodes representing individual event sources became trivial, and other new features such as the raw data tap and debugger became available. Also, to facilitate all of this, was a port to ECMAScript/JavaScript for the parsers which significantly decreased the slope of the learning curve for new development. We're still working in the SIEM space with a product like Sentinel, so there is a curve but it's much easier for mere mortals like myself to start climbing.

As often happens, time continued to rush on and the Sentinel product expanded to the Log Manager product for a focus on capturing logs with less of the fancy SIEM functionality, plus the addition of a limited-use, 25 Events per Second (EPS), free perpetual product for those inclined. Reworking of the backend away from proprietary technologies made this possible, so Sentinel now uses Jasper Server-based reports, PostgreSQL for configuration and some data storage if needed, Lucene for really fast searches, and ActiveMQ for the iScale message bus linking everything together. New attributes have been added to handle new data types and customer data in larger quantities, and the 7.x versions of Sentinel (along with Log Manager) has been able to also expand the event rates supported to (per the documentation) at least 13,000+ fully parsed/analyzed events per second, and 60,000+ netflows per minute, so a lot of work has been happening on both sides of the Sentinel environment, collection and backend data handling. The last year in particular has seen a lot of good growth here, which all leads up to the SDK changes that are coming in the 2014 Preview.

To make the SDK accessible and usable to as many people as possible it has been made available for many years as a resource for use within Eclipse. The current version includes an Eclipse plug-in which has been improved a couple of times over the years to simplify the process of getting into a new type of plugin (collector, report, solution pack, etc.) so what used to be pretty simple is now even easier, particularly for those who are not Eclipse experts. First, let's outline some URLs that will be used throughout this article (you may need a novell.com/netiq.com account to access some of these, but as that's a free account it should be no problem):

Public SVN SDK repository - https://svn.novell.com/svn/plugin-sdk/trunk/
Sentinel 2011.1 SDK documentation - https://www.novell.com/developer/develop_to_sentinel.html
2014 Preview SDK Download - https://www.netiq.com/support/sentinel/plugins/preview.html (under the 'Developer Kit' tab)
Rhino (JS) implementation homepage - https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Rhino
Eclipse IDE downloads - https://www.eclipse.org/downloads/
Sentinel Plugin Forum - https://forums.netiq.com/forumdisplay.php?75-DEVELOPERS-Plug-in-SDK

With the links above in mind, there are a couple of ways to get the PlugIin SDK to your computer for immediate use; first, download it from the 2014 Preview SDK Download page mentioned above under the 'Developer Kit' tab. This tab includes a nice Javadoc version of the API used by the SDK which is pretty handy when you're wondering what the new functions for the DateTime class are and whether or not they're static, take certain parameters, etc. If in doubt, go with this route and extract the resulting SDK to your system somewhere, for example to /home/you/code/sentinel-plugin-sdk/ so try to minimize the amount of work required to find it and its enclosed files. The URL directly to the SDK follows:

https://www.netiq.com/support/sentinel/plugins/pre/sdk/Plugin-SDK-2014.1r1-Beta-36.zip

Along with the SDK this page also includes, as mentioned, a javadoc, and other documentation to help get started working through the use of the SDK, including this pretty-large (132 pages) PDF:

https://www.netiq.com/support/sentinel/plugins/pre/sdk/sdk_2014.1-preview.pdf

What may not be entirely obvious is that this PDF is accessible from the same 'SDK' line as the actual SDK zip linked-to previously. This download page follows the same format as the other plugin pages where the documentation is simply the PDF link on the same line as the actual content rather than being on its own line. This document is worth it, though, as it has quite a bit of new information over the previous SDK documentation, which is still largely relevant. For that, we go to the 'Sentinel 2011.1 SDK documentation) here: https://www.novell.com/developer/develop_to_sentinel.html This is a set of pages designed to help work through development of various types of plugins, from collectors to reports. A lot of the nuances that come from working with Sentinel for years are explained in this reference and so it is worth a read to better understand some topics that may not otherwise be obvious (looking at you, Sentinel Taxonomy and other mapping functionality).

Another option made available by NetIQ is via SVN, which is a source code versioning tool familiar to most who have ever programmed anything in the last decade. The repository has a tendency for me to timeout, but this command should work pretty well and has worked in the past:
svn co https://svn.novell.com/svn/plugin-sdk/trunk/

The process of checking out this way takes a while (again, the main download above is probably a better option) but once done you'll have a some directories which are used by Eclipse and the Sentinel plugin for Eclipse. Those are also on the preview page above, and at this point there are probably two audiences to address.

Audience #1: Most of you have probably used Eclipse before, and may be using it now for development of some kind. To provide some great features there is a tiny plugin for Sentinel use within Eclipse at the 'Eclipse IDE Plug-in' link which can be trivially added to Eclipse (do not forget to restart Eclipse, if already loaded, to apply) adding a new Eclipse "Perspective" (technical term basically meaning look/feel/setup for Sentinel plugin development). If you do not have Eclipse installed but feel comfortable doing a simple install then grab it (I'm actually testing the latest 4.4 version, though 3.7 is currently recommended and shipped by NetIQ... I have not had problems with 4.4, but do whatever makes you feel best) from its website (https://www.eclipse.org/downloads/) and then install it somewhere like /home/youruser/apps/eclipse44/ ).

Audience #2: If you are not a developer but want to get into it, you likely do not have Eclipse and you may not feel comfortable adding a plug-in; you may just want to get into things Sentinel-related and skip over the environment setup as well, which is fine. For you, NetIQ has provided an Eclipse IDE installer, including the Sentinel plugin, assuming you are stuck on something like windows, linked-to from the preview pages above. Pick the 64-bit one (probably, unless you're really on a 32-bit box for some crazy reason) and run it. I believe you still need to download the actual SDK file as well and extract it somewhere, but I am not positive on that so feel free to post comments if you follow these steps and are a windows user.

Voila, the Sentinel SDK is now available. After installing Eclipse one way or another, and getting the plugin one way or another, you are on some version of Eclipse version 3.7 or later and should be able to, when using the Sentinel perspective, easily point to the extracted SDK on your hard drive. If you do not see the Sentinel perspective, go to Window: Open Perspective: Other and then click on 'Sentinel'. After doing so you will see that the left-hand 'Sentinel SDK' View has options grouped by type of plugin (Actions, Collectors, Feeds, Report Data Definitions, Reports, Solution Packs). Expanding these you can create new plugins and start working.

The next step is probably to create a new Collector plugin and then find an event source whose security data would be relevant to a Sentinel system. I saw somebody created a 'dnsmasq' collector a week or two ago, which may be a good read for those starting out: https://www.netiq.com/communities/cool-solutions/cool_tools/dnsmasq-sentinel-collector/ Do you not have a Sentinel system handy right now? One of the perks of this 2014 Preview is a debugger built into the SDK, so while you will not get to see events in the same form as you would in Sentinel's Web UI, you can still test parsing, see the raw (JSON) event output from a created collector, run built-in and custom tests, etc. all from your workstation as long as you can get event data in a form ready for Sentinel to parse. More on that later in another part of this series.

Working with the paths of the SDK can be a bit daunting as there are a lot of possibilities and, thus, directories of the SDK. For the most part, we'll spend our time initially playing under the 'content' subdirectory, as this is where created content will be stored (under 'dev'), built into plugins (under 'build'), and generally managed. The sibling to 'content' is 'current' which means the current version of the SDK and is generally something you should NOT change, though feel free to browse and become familiar with the various files in there as they eventually become good to know, at least at a high level. Changing the SDK files will impact all plugins built from the SDK, though, so this is generally something best left to NetIQ.... generally.

The 'content' side of things has subdirectories for 'dev' and 'build' as mentioned above, and 'build' is probably empty. 'dev' will include subdirectories for the various types of plugins and as you create new types of plugins in the SDK (even if you do not do anything with them) you'll see new structures created for vendors and products as you add those. For example:

./content/dev/
./content/dev/sdk
./content/dev/sdk/build
./content/dev/rdds
./content/dev/rdds/Example Event Source Starts And Stops
./content/dev/rdds/Inactive Users Sync
./content/dev/rdds/Anomaly Report Sync
./content/dev/rdds/Source Summary
./content/dev/rdds/Object Access Dashboard Sync
./content/dev/rdds/Password Resets Sync
./content/dev/rdds/Example Login Attempts
./content/dev/rdds/Example Authentication Attempts
./content/dev/rdds/Example Admin User Activity
./content/dev/rdds/Event Source Overview Sync
./content/dev/solutions
./content/dev/solutions/Sentinel Core
./content/dev/solutions/Collector Base
./content/dev/feeds
./content/dev/actions
./content/dev/actions/Disable LDAP User by CN
./content/dev/reports
./content/dev/reports/Sentinel Core
./content/dev/reports/Collector Base
./content/dev/collectors
./content/dev/collectors/NetIQ
./content/dev/collectors/Generic
./content/dev/collectors/Generic Test Collector Vendor
./content/dev/collectors/RadiantLogic


As you can see at the bottom I have four different vendors' collectors defined in my current system, and within those are directories for each type of plugin ('ICS' for RadiantLogic, for example). Within that location are all of the files specific to that type of plugin. Some are grouped by version (such as tests, documentation, the release.js file which is the starting point for most things in the collector, etc.) and some are not (sample input data, research notes, etc.). The structure is very logical and is used, I am told, by the developers of NetIQ's official collectors so it has been tested over the years; it does have a tendency to become rather deep just because of how much is contained and how organization tends to simplify things later on. As a result, I have found it useful to add links to various points within the structure when I'm working actively on a project, so that if I need to get into that part of the filesystem I'm saved about ten clicks and can get there directly from my desktop or home directory very quickly. Removing a symlink once done with a project is trivial and does not impact the SDK at all.

The sibling to 'dev' is 'build', and this also gets used every time a plugin's source code is merged into a plugin for use within Sentinel, whether or not that happens because of the debugger, running automated tests, creating a development build, or creating a release (production) build. Note that while collectors and other plugins are not huge, the build process puts some JAR files into these directories, uncompressed along with their compressed versions, so space is eventually taken, and currently not automatically cleaned/recovered; after a lot of work you may see hundreds of MBs consumed here. From my system currently:

./content/build/
./content/build/Report
./content/build/Report/Sentinel-Core_Event-Details_2011.1r3-201410151758-release
./content/build/Report/Sentinel-Core_Event-Details_2011.1r3-201410151759-release
./content/build/Report/Sentinel-Core_Event-Details_2011.1r3-201410151800-release
./content/build/Report/Sentinel-Core_Event-Details_2011.1r3-201410151803-release
./content/build/Report/Sentinel-Core_Event-Details_2011.1r3-201410161652-internal-test
./content/build/Action
./content/build/Action/Disable-LDAP-User-by-CN_2011.1r1-201411032451-internal-test
./content/build/Action/Disable-LDAP-User-by-CN_2011.1r1-201411030102-internal-test
./content/build/Action/Disable-LDAP-User-by-CN_2011.1r1-201411030104-release
./content/build/Collector
./content/build/Collector/RadiantLogic_ICS_2011.1r1-201412100652-internal-test
./content/build/Collector/RadiantLogic_ICS_2011.1r1-201412101344-internal-test


Note that each build includes a date/time stamp in the directory structure, so you can compare one with another. This may be useful if, for example, you get a bit over-excited with the 'delete' key at one point, or if for some reason you do not use version control for your source files and lose a change that you want back. All of the content can be seen, in some form, in the 'build' directory and while it may not be as nicely-structured as it was in 'dev' once "built", it is still there for comparison against other versions.

Also note the 'test' vs. 'release' distinctions. The SDK lets you build either 'development' builds or 'release' (production) builds. The biggest difference that I have found is whether or not the release ('r1' in '2011.1r1') increments permanently or not, though there may be other subtleties as well. When doing development and testing small changes incrementing the release version is usually not beneficial, and having something like r1234 will probably just cause confusion or raised eyebrows. As a result, incrementing the release number is usually only done when a version is complete and you ship it to the customer for production use.

Have any outstanding questions, comments, etc? If about this article, use the comments features here. If about the SDK, send those to the SDK forum (https://forums.netiq.com/forumdisplay.php?75-DEVELOPERS-Plug-in-SDK) and let's get started with extending your system.

DISCLAIMER:

Some content on Community Tips & Information pages is not officially supported by Micro Focus. Please refer to our Terms of Use for more detail.
Top Contributors
Version history
Revision #:
1 of 1
Last update:
‎2014-12-31 18:40
Updated by:
 
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.