Integrating ArcSight with other systems: getting events, sending events, syncing cases and more
ArcSight products form the core of the security operations center. As such they are designed to integrate with any technology one may use in the SOC or outside it such as security systems, IT systems and applications. While basic integration comprises sending events to ArcSight, many other level of integration are available including sending events from ArcSight, loading context information to ArcSight, executing external commands and automating the ArcSight products themselves.
This documents provides an overview of this techniques alongside references to further resources for implementing them. In addition, the guide provides a use case section that summarizes the integration methods needed to implement real world scenarios such as vulnerability management integration or a CMDB integration.
Sending events to ArcSight
While ArcSight provides a large number of connectors for common event sources our of the box and partners and the community event more (see ), there is nearly always a need to collect events from additional event sources. The following techniques enable collection of other event sources by ArcSight.
Using CEF (Common Event Format)
CEF is primarily intended for vendors who would like to enable their software or devices to send events to ArcSight (as well as other products which support CEF) out of the box. That said, end users can also implement CEF in internally build systems or using an integration script that read information from a source and convert it CEF.
To use CEF to send events to ArcSight:
- The source device or an integration script has to format the events in a CEF format following the
- The CEF format events can be ingested to ArcSight:
- By sending them to a the or using the Syslog protocol.
- By writing them to a file to be picked by the or .
- By writing them to files managed by Hadoop HDFS to be picked up by the .
- By creating a web service that will provide the CEF content as a RESTful to the CEF Rest Connector. For more information refer to the .
- The different connectors then send the events to any ArcSight recipient including ESM, ESM Express, Logger and UBA.
Vendors implementing CEF in their devices can certify the implementation with HPE, more details here.
Creating CEF sources
Creating Flex connectors
If no existing connector is close enough and there is a need to build a connector from Scratch, ArcSight Flex Framework enables creating a connector without programming:
- Writing a Flex Connector requires creating one or more configuration file.
- To use a Flex Connector, the use the regular SmartConnector install and select one of the Flex Connector types.
The different types of Flex Connectors address different access method to the event source. The Flex Connector configuration files further instruct the connector on how to parse and map the information received. An exception are JSON and XML formats for which specific Flex Connectors types exists.
Flex Connector types include:
- Log File FlexConnector for reading fixed-format, delimited log files.
- Regex Log File FlexConnector regular expression for reading variable-format log files.
- Regex Folder Follower FlexConnectors for recursively reading variable-format log files in a folder or multiple folders.
- Multiple Folder Follower FlexConnector for reading log files of multiple formats in different folders.
- XML FlexConnector for recursively reading events from XML-based files in a folder.
- JSON Folder Follower FlexConnector for recursively reading events from JSON based files in a folder
- Time-Based Database and ID-Based Database FlexConnectors for reading the latest security events from a database.
- Multi-Database FlexConnector for reading events from multiple databases.
- Push sources:
- Simple Network Management Protocol FlexConnector.
- Syslog FlexConnector for reading events from Syslog messages.
- Windows events (if not parsed by default). For more details see:
- Scanner FlexConnector to import the scan results from a scanner device in the following formats and sources: Normal text report, XML report, Database
- REST FlexConnector that uses REST API endpoints, JSON parser, and OAuth2. See details in
It is important to note that using "Extra Processors" (as documented in the FlexConnector developer's guide) enables mixing and matching event sources and formats. Extra processors include:
- Map file like processing (see below)
- Regular expression parsing
- Key-value parsing
- Delimited parsing
- Ntsubparser for supported Windows application parsers
- XML parsing
This enables for example parsing an XML format embedded in a database field even if there is no specific Flex type in the list above.
The following resources are useful for flex connector development:
- HP Flex Connector deep dive and best practices, a popular Protect conference presentation, the 2014 edition
- "ArcSight Flex Connector Training V2 with More About Flex", an enhanced community contributed guide/presentation
- Writing FlexConnectors Webinar with HPE's Aaron Kramer
- Creating a regex flex connector for HP ArcSight, a video by by Shane Lilley
- How to Configure: Oracle ID Based Flex Connector - a highly recommended tutorial on creating this specific type of flex connector.
- Specific tips and tricks:
Building on top of SmartConnectors
In many cases there is no need for a full fledged customer connector and small modifications to an existing connector may be enough. For example, adjusting an existing connector to a similar device or another version of a supported device. There are several features that enable modifying the behavior of a SmartConnector:
- Map files allow adding a mapping file that can modify an event prior to sending it from the connector to any destination.
- Map files can assign a value to any field in the event based on different conditions applied to the event fields, including range and regular expression conditions.
- In addition, the value assigned can be an expression allowing the map file to generate new values for assigning, rather than just assigning a constant value of the value of another field. In this way a map file can include significant amount of logic beyond just "mapping".
Event categorization can be modified by the user in the following manner:
- User override files on the connector
- User overrides generated through the Console categorization tool
Build a Flex Connector by modifying the original parser
Contact HPE support and provide the business case to get the unobfuscated parser.
Flex Like Frameworks
Build a helper parser for Sylog, SNMP and Windows as described in the Flex Connectors section above.
Loading context information to ArcSight
Loading ESM and ESM Express Active Lists
Active lists are used by ESM and ESM Express as a generic mechanism for storing context information such as threat intelligence and indicators of compromise (IOCs). To import or update information in an Active List from an external source use the following technique:
- Create a Flex connector (see above) to send the records to be imported as events. Use the Flex connector type based on the format in which the source records are stored.
- Create a light weight rule to update an Active list based on the received events information. Using a light weight rule ensures that no correlation event is generated and therefore reduces the resources needed for this process. For more information about pre-persistence rules, the , chapter 18: "Rules Authoring" (for version 6.9.1c. For other versions refer to the corresponding guide for the specific version).
Loading ESM and ESM Express Network and Asset information
ESM operates on a data model that enables building business-oriented views of data derived from the IT systems. Network modeling is done to keep track of the devices on the network involved for which events are received. The model helps ESM clearly identify the events in the network, and provide more layers of detail to ESM's correlation capabilities.
The Asset Model Import Connector, available as part of ESM and ESM Express release, enables automating the import of assets and other network model entities to ESM or ESM Express. The Connector reads information in a CSV file and supports both initial import and an ongoing update of asset information. The default parser adds or modifies asset information. Alternate parsers can add or modify zones and networks. Please contact ArcSight professional services for information about such alternate parsers.
Loading ESM Actor (User) information
The ESM actors feature enables building a model of the users in the network environment that helps ESM to assess and prioritize events. The information is also available to ESM's correlation capabilities such as rules.
The Active Directory Model import connector. available as part of ESM (but not ESM Express), enables automated import of user information from Active Directory to the ESM actor mode.
Loading Logger Lookup tables
Logger uses lookup tables to provide context within search. Lookup tables are used for example for threat intelligence and for GeoIP lookup. Logger enables schedule upload of those tables from CSV files. For more information refer to "Uploading Lookup Files" in the (for version 6.1. For other versions refer to the corresponding guide for the specific version).
An advanced method for importing and exporting information to and from ArcSight is using archives. The archive command ($ARCSIGHT_HOME\bin\arcsight archive) accepts specially formatted XML files and imports them as resources to ESM. It can also export ESM resources as an XML file. This enables importing and exporting any resources including rules, active lists and assets.
Few resources to get you started in using this advanced feature:
- and a nice doc for it
Getting events from ArcSight
Using a forwarding connector
Both Logger and ESM have forwarding connectors that enable sending events onward, to another ArcSight system or to an external system.
The ESM forwarding connector can forward events in CEF format to a syslog server or to a file. In addition events can be send to HPE OM and HPE OMi. The source ESM can be configured to forward a subset of the events using a filter as well as to automatically forward all the based events associated with a correlation event.
For further information:
- The forwarding connector documentation (for version 6.9.1c. For other versions refer to the corresponding guide for the specific version)
Vendors consuming CEF through the ESM forwarding connector can certify the implementation with HPE, more details here.
(To be completed)
Using multiple event feeds from a connector
- Smart Connectors can send the same events to multiple destinations, whether ArcSight systems or others.
- This has the advantage that it does not put load on an ArcSight product. While it does put load on the connector, connectors are generally easier and cheaper to replicate, especially using the connector .
- On the other hand, this method would not enable sending correlation events, the base events they refer to and any information enriched by the ArcSight products. It may also complicate deployment as there may be many connectors that require forwarding events to an external system.
- Forwarding from a connector is done using either syslog or a file and uses CEF format.
- Note that while technically possible, this method is allowed only if you have the new ArcSight Data Platform (ADP). Contact an ArcSight sales person or partner for more detailed.
Using the Event Data Transfer tool (ESM and ESM Express)
The Event Data Transfer Tool, available on the ArcSight Market Place here, enables batch fast event transfer in CEF format from ESM to files or to Hadoop HDFS.
Using the API
- Build a Query + QueryViewer and use the QueryViewerService->getMatrixData ESM API interface to retrieve the data from the query viewer.
- If you know the ID of an event you can retrieve the event directly using the ESM API SecurityEventService.
Logger Web Services API enables an external program to execute a search and retrieve the results.
The Logger API has three methods than enable retrieving event:
- Search using a SOAP API - runs a Logger search query and let the client iterate through the results.
- Search using a RESTful API - offers an alternative to the SOAP API using a RESTful API. The RESTful API allows better performance as up to 10,000 results can be fetched per call, which better suits extraction of a large amount of data. It also allows getting the histogram for the search as well as chart data and raw event information associated with search restuls.
- Running a report using a SOAP API - enables running a predefined report.
Useful resources on the APIs
Tools for using the API
Bi-Directional Ticketing System integration
ArcSight ESM and ESM Express can export events or cases to an external ticket system and be updated on the ticket status. When exporting events, a case is generated automatically which will accommodate the status update. The interface utilizes XML files: ESM creates XML files for each exported cases or event in a directory and polls another directory for responses. The integration program is expected to monitor the cases directory for new files and send them to the target ticketing system and write status information to the responded directory.
From a users' perspective, the export can be initiated:
- Manually by a right click command on an event or a case:
- For an event: "Export" / "External tracking system"
- For a case: "Export to external system"
- Through the rule action "export to external system", which will export the event generated by the rule
- Automated periodic case export using the "case search group" feature.
All those options utilize the same XML files based interface described above.
For more information:
- The provides documentation which is relevant to any use of this interface. Refer to the following sections for information
- Features and Functional Summary
- Using the ArcSight Console to Export
- Tracking Event and Case Exports
- Troubleshooting Using the Log Files and Product Log
- ESM DTDs Location and Example
- The following presentations are useful
- Tips: To preserve base events in exported cases add "perserve.baseevents=true" to the server.property file
Sending alerts from ArcSight
(To be completed)
- Send email notifications from a rule action
- Send SNMP traps from ESM Manager
- Use Logger scheduled alerts
Executing commands from ArcSight
Manual activation using Integration commands (ESM and ESM Express)
Integration commands are defined within the ESM or ESM Express console and enable the user to initiate an external commands. The integration command can:
- Be used in different contexts such as Active Channels, Active Lists and Query Viewers.
- Embed values from the entity they are to be used with such as the "right clicked" event.
- Be configured with setup values such as username and password required for running the external command.
- Prompt the user interactively for additional parameters
There are three types of commands:
- Executing a program or a script on the console's host.
- Opening a URL for browsing on the console's host.
- Executing a program or a script on the a connector (see "Action Connector" below)
Automated activation using rule actions (ESM and ESM Express)
When a rule triggers it can initiate an external command using either of those actions:
- "Execute Command" enables running an external command or script on the ESM Manager server.
- "Execute Connector Command" enables running a command on an action connector registered on ESM (see below).
Since executing commands locally on the ESM manager has performance impact on the manager the rate of commands activation is limited. Executing commands on an action connector enables a much higher rate of command activation and should be the preferred method for any system that requires a lot of external commands activation. Action connectors also enable running Windows commands and parsing the command response which a locally executed command cannot.
Action connectors (ESM and ESM Express)
Action connectors (sometimes called CounterAct connectors) are installed like regular SmartConnectors but serve to execute commands on behalf of ESM and ESM Express managers. Action connectors enable:
- A higher rate of external actions activation.
- Running eternal actions on Windows systems as connectors, unlike the ESM manager, can run on Windows.
- Parsing the results of the external command and sending it back to ESM.
The actual commands that an action connector can initiate are defined in the action connector configuration file and appear in the following locations in the ESM the connector is registered with:
- As an integration command action.
- As a rule action "Execute Connector Command" option.
- As a command for the specific connector resource.
Vendors implementing Action Connectors for their devices can certify the implementation with HPE, more details here.
Automating and extending ArcSight
Using the ESM and ESM Express API
The ESM API enables an external program to automate different tasks in ESM as well as to provide a mechanism for an alternative UI for several ESM features. For example, the following functionality is exposed by the ESM web services API:
- Reading and modifying cases
- Managing reports
- Getting information about resources.
- Getting information about an event using an event ID
The ESM API is described in the following documents (for version 6.9.1c. For other versions refer to the corresponding guide for the specific version):
- ESM Service Layer Developer's Guide
- ESM Service Layer API Reference, Vol. 1: Core-Client Services
- ESM Service Layer API Reference, Vol. 2: Manager-Client Services
Using the Logger API
As described above in the "Sending events from ArcSight" section, the Logger API enables performing search and running reports on Logger. See the for more details (for version 6.1. For other versions refer to the corresponding guide for the specific version).
From the command line
(To be completed)
Integrating a vulnerability scanner
Vulnerability scanners provide vulnerability information to ArcSight ESM which is used as part of the event prioritization formula. Scanner reports are also used to enrich the ESM asset model. Lastly, ESM can be used as a vulnerability manager reporting on vulnerability status and integrating remediation tools.
ArcSight provides a number of scanner connectors our of the box. To implement a custom scanner integration:
- The Flex framework supports creating customer scanner connectors for scanner reports stored in regular or XML text files or in a database.
- The REST CEF format defines special event types for delivering scan results and asset information to ESM. Information about these event types can be found in the CEF specifications.
Integrating intrusion prevention systems (IPS)
Intrusion prevention systems are a common event source for ArcSight. ArcSight includes out of the box connectors for a wide array of intrusion prevention systems. Connectors for additional intrusion prevention systems can be implemented using any of the methods described above for sending events to ArcSight.
A unique feature for intrusion prevention systems is the ability to fetch the payload associated with the event on demand without sending it as part of the event for every event. This feature is described in the "Working with Event Payloads" section in the (for version 6.9.1c. For other versions refer to the corresponding guide for the specific version).
To emulate payload retrieval for a flex connector use an action connector:
- Implement the connector action command to fetch the payload. Action connectors are described here.
- Define an integration command to utilize this action command and display the payload. Integration commands are described here.
Integrating STIX and other threat feeds
To import threat intelligence information for STIX or other sources use the instructions for loading information from an external source to an Active List here. The following are examples for loading threat intelligence into ESM:
Integrating a ticketing system
To integrate an external ticketing system refer to the section about bi-directional cases integration.
Implementation automated remediation
Using ArcSight for remediation requires activating an external command to perform the remediation. This is covered in "execute commands from ArcSight" section. An example of such an a remediation solution is .
Sounding an alarm when the a rule triggers
A common use case is to sound an alarm or another sound when a rule triggers. The challenge with this requirement is that the sound usually needs to be played on the console or another system in the SOC and not on the manager which may reside in a data center - sometimes event a remote one however the "Execute Command" rule action is performed on the manager.
The solution to this is to use the "Execute Connector Command" together with an action connector as described here. This way any script that plays a sound on the console or a dedicated system located in the SOC itself can be used.
HPE Security Technology Alliances Partner Program
As noted above vendors HPE certify the following integration types as part of our alliance program:
- Sending events to ArcSight in CEF format.
- Receiving events in CEF format from the ESM forwarding connector.
- Implementing an action connector
More details here.
) You are pretty much correct. Logger API you can query the events using logger search syntax. On the ESM you need to build a Query + QueryViewer and then use the QueryViewerService->getMatrixData to retrieve the data from the query viewer.
2) If you know the security event id's you can query them directly using the ESM API SecurityEventService.[pdf-att]/home/lithium/migration/hp_protect724/mnt/jive_persist/binstore/scan_jivesbs/a15526.bin[/pdf-att]
Dear Ofer Shezaf,
Very Interesting subject (but a bit old with remedy). Do you have any feedback or return about the Integration of ESM and HP Service Manager.
Because the Case management in ArcSight is not enough efficient and highly complex to custom.
We would like to decrease the time to create a case on ArcSight and if the integration with the HP ticketing system is good, efficient and easily manageable. I am interested but due to the difficulty to evaluate this process internally directly in production I would like to have comment from users who has already implemented this solution.
Since this is a user community, it might be better to seek advice from members rather than from me. Anyone out there wants to share feedback on ArcSight and Service Manager integration?
Hello, Actually I am facing some error while trying to integrate ESM 6.8c with HP SM 9.34. I installed the Enterprise connector and I can export the events/cases to HP SM but in ESM console, the "External ID" is not getting generated and no update coming from the HP SM side. I am looking for help here, would be nice if I get anything from the forum. Thanks in advance, Rakesh
Hi, actually I am going to integrate remedy with arcsight in a couple of days. Can anyone help me to share the information, is there any point which I have to take care about. Because ESM 6.8c is already working in a production environment with HA. What type of precaution I need to take care about.