New Ranks & Badges For The Community!
Notice something different? The ranks and associated badges have gone "Star Fleet". See what they all mean HERE
Highlighted
Fleet Admiral
Fleet Admiral
1658 views

What custom integrations or scripts would you like to see? Community feedback appreciated!

The upcoming weeks i will be releasing a lot of new things to help people use and understand the API's available for ArcSight ESM and ArcSight Logger, after that i would like to create some new custom functionality and scripts that the community can enjoy, the only issue is that it would not really be efficient to create something that no one would want to use.

Please let me know if there is any custom work that you would love to see when it comes to interaction with these two products. All will be released on github in case people would like to make their own changes and tweaks, together with proper documentation and commented code for learning purposes.

Any type of request is appreciated, and the only requirement is that it is not related to a product that is behind a license wall as i would not be able to test it or access API documentation to the product.

A few examples of requests that i can think off:

1. Slack or messaging notifications. Being able to create a rule action that notifies your slack channel when an alert happens.

2. Open Source threat intelligence framework, scripts that retrieve threat intelligence sources from a large amount of open source feeds, being able to choose which ones you want to use, and feeding it to ESM through a syslog connector.

3. Logger interaction script. Something that you can manually run to for example return the results of a query in CLI. Could also be scheduled to create scheduled exports of certain queries.

UPDATE:

First version of Request Tracker integration with ESM has been released: 

https://community.softwaregrp.com/t5/ArcSight-User-Discussions/Unofficial-ESM-Request-Tracker-RT-Ticketing-Integration-released/td-p/1672538

First version of the ArcSight Logger API documentation + examples:

https://community.softwaregrp.com/t5/ArcSight-User-Discussions/Unofficial-API-documentation-and-examples-Part-1-3-ArcSight/td-p/1674083

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
18 Replies
Highlighted
Fleet Admiral
Fleet Admiral

One last run to see if anyone more is interested! Got some PM's about it already, though better to put it in here.

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Highlighted
Knowledge Partner Knowledge Partner
Knowledge Partner

@Marius2,  Are you going to give any examples for ESM API? That would be great.

For Logger API, I think it's possible to do a simple device monitoring. Here is my scenario: pull device Address and device Hostname from a search( we can do this by using dedup operator) and compare it to a list in a file(csv or something like that). After comparing, export missing or extra device informaiton to a file or sent an email.
There are so many customers just having logger and not capable of monitor the status of devices. I believe this would be awesome.

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
Highlighted
Fleet Admiral
Fleet Admiral

@mr_ergene Yeah absolutely, any integration examples will be fully available and documented for learning purposes towards the community.

If you mean the examples i will be giving out, then yes, it will cover both Logger and ESM.

Logger is quite small, so i will cover all API calls there, on ESM i will deliver 2 versions, one for getting started with all the most used API calls, and then a bit later one with all API calls.

The last one covering all API calls takes a bit extra time, as there is certain API calls i have to keep out as they are dangerous, while i like to test all of them before releasing them (current count of API calls is 2000, though many are redundant of eachother, which is why i am stripping out certain ones).

All in a easy exportable format, that you can import with this tool that i use for all my API testing before adding them in whichever script or software i am developing with: https://www.getpostman.com/apps.

I will provide some quick examples on how to import and get started, but the rest is up to the developer 🙂 Though ofc any API specific questions can be posted, and i normally pick them up quite quickly.

It's all a part of a followup after the presentation i had at the Cyber Security Summit in Washington, no way to direct link it, but was 9:50 on Thursday: http://www.cvent.com/events/micro-focus-cybersecurity-summit-2018/agenda-f2b113e2a7234e0b8680e70404728f68.aspx

For your logger example i am trying to understand what you meant.

You want to run a search, that returns all unique device names/addresses from that search, and IF any of the devices from your local list is not in the search results, then create a notification by email etc?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Highlighted
Knowledge Partner Knowledge Partner
Knowledge Partner

Thanks for the details 🙂

For the logger example, yes I meant that. If there is an alternative way for device status monitoring using logger, that would be also OK.

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
Highlighted
Fleet Admiral
Fleet Admiral

Hmm, when enabling device health monitoring on a connector, i am quite sure it can also send this to the Logger, so i could just simulate the ESM implementation of that in a few lines of code i think. Let me add it to the list.

I am always open for some more challenging approaches as well, feel free to suggest anything! 🙂

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Highlighted
Knowledge Partner Knowledge Partner
Knowledge Partner

@Marius2,

Device monitoring doesn't work properly on smart connectors even the aggregation for internal events is disabled. That's why I mentioned about running a query with dedup operator on deviceIP and deviceHostname 🙂

------------------------------------
Please use the Like button below, if you find this post useful or mark it as an accepted solution if it resolves your issue.
0 Likes
Highlighted
Fleet Admiral
Fleet Admiral

Hmm, so connector monitoring would be a piece of cake, as Internal Event Storage includes minute by minute EPS count per Connector, but drilling it down to how many EPS per device is not really available at this point that i could find.

I also didn't find any Agent:043 events, as i was hoping it at least sent a copy of that to the Logger (might be missing a destination configuration parameter for that though.

The issue is really if you have 10k devices, these types of monitoring solutions would be limited to connector level, as it would be too much to do statistics on that many devices in each search (also because 10k is the max limit for one API call, though you can bypass that in certain ways).

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Highlighted
Captain
Captain

Hello Marius,

1.  Adding to "point 3" from your post. A script that could pull the dashboard logs/Job execution status of the Arcsight logger via API. My idea is to automate the healthcheck of the logger software without manual intervention. I am aware that reports and searches can be pulled using the REST/SOAP API but not sure about this.

2. This may seem farfetched. But what if the smart connectors had the option to fix log stoppages itself without manual intervention? Auto restart for starters(not from ArcMC). Or even a notification with a detailed analysis based on the error logs along with suggestions.

regards

Sharan Bhat

0 Likes
Highlighted
Fleet Admiral
Fleet Admiral

Hey @sharan Bhat

The Logger API is only limited to Search and Reports. Reports being SOAP only for some reason..

It is not all lost though, all depending on what you want to monitor. It would be quite straigth forward to create monitoring for these events, as they are internal events available from searches:

CPU:
/Monitor/CPU/Usage cpu:100

Disk:
/Monitor/Disk/Read disk:102 
/Monitor/Disk/Write disk:103

EPS:
/Monitor/Receiver/EPS/All eps:100
/Monitor/Receiver/EPS/Individual eps:102
/Monitor/Forwarder/EPS/All eps:101
/Monitor/Forwarder/EPS/Individual eps:103

Memory:
/Monitor/Memory/Usage/Platform memory:100

Network:
/Monitor/Network/Usage/In network:100
/Monitor/Network/Usage/Out network:101

Search:
/Monitor/Search/Performed search:100

Storage Group:
/Monitor/StorageGroup/Space/Used

Would these be sufficient? For example i could enable the user to fill in a template, of min and max values that any of these values is supposed to have, like:

Disk 10%-80%, CPU 0-90% etc etc. Then it will just notify you if any of the values are outside of your "boundaries". Sounds useful?

For connectors, this is normally done through basic system administration scripts, and outside of the product itself. Normally what you could do here, is have a cronjob that runs every 10-15 minutes, checks that logs are coming in (by checking the last time the queue file was edited, then also check if cache is building up and that the process is running.

Then if queue file is old but process is up, it should notify, if cache is building up it can notify, and restart process if down.

The only issue here, is if you want to automate this process, like removing big cache file, you are essentially losing data, so automatic intervention would not always be such a good idea.

Do you have any specific examples on things that happens, and what you want to automate in response on connectors?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
Highlighted
Lieutenant Commander
Lieutenant Commander

@Marius2

A couple of things I am about to start looking into for a customer, I am not sure if they are sutiable for your integrations but here they are:-

Right click funtionality in ESM to create a ticket in Request Tracker Ticketing system.

Basically an event comes in to an Active Channel and the customer wants the Analyst to Right Click on that event and have it generate a ticket in RT

https://bestpractical.com/request-tracker/

Also a quick and easy way to monitor current cache sizes of all connectors in the event of ESM not being available.  I was going to try to do this in stages with the first stage being to collate current cache sizes of all connectors.

Ideal features are:-

  • Current cache size
  • available cache remaining
  • free disk space remaining
  • based on the above, a prediction on how long the connector can continue to cache before events will be dropped
  • connectors that are dropping events, if possible include a count of dropped events.

How much of the above is poossible I do not know but having just suffered from an incident of ESM being down due to a hardware failure this is quite a hot topic.

 

 

0 Likes
Highlighted
Fleet Admiral
Fleet Admiral

@Frenjd

For RT i can create a simple script that creates a ticket in RT sure, as the API documentation is available.

For connectors, it seems that quite a few are requesting this, and i am a bit suprised as to why, there is a few things:

ArcMC is mandatory in later versions of ADP, utilized to monitor license usage etc. ArcMC is always the best place to monitor and manage your connectors, including cache size etc. Using a manual script to do this is possible for some of the usercases, but they really should implement ArcMC instead.

Predictions and statistics requires access to the history of the growth, since ArcMC does not have an API, it would have to retrieve it from ESM, or keep it's own history. 

From the requests i got in this thread, what i could see is:

@mr_ergene

Logger API script for device monitoring, not really possible i am afraid, as Logger only stores connector monitoring data. ESM API examples should be coming soon at some point though.

If i was to do device monitoring on logger, it would need to be done through the report API, with heavy queries that would break if you got lots of devices + large amounts of data.

@sharan Bhat

1. I checked the overview of all internal events that generates on the logger, and there is no job status or report status internal events that i can access unfortunately 😞 The API itself only focuses on search.

2. Detailed automatic log analysis is a bit out of scope, those really should be integrated into your central logging solution instead. A script could restart connectors etc, but i wonder how much time you would save.

@Frenjd

1. RT Ticket creation should be just fine, should it be able to click on a event, and update an existing ticket with new info as well?

2. See answer at the top.

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.