New Ranks & Badges For The Community!
Notice something different? The ranks and associated badges have gone "Star Fleet". See what they all mean HERE
Highlighted
Lieutenant
Lieutenant
4003 views

Understanding ESM API documentation

Jump to solution

Hi all,

I'm developing a new tool and I need to interact with ESM API. I want to craft url requests with Python instead of Java and I use both urls as documentation:

https://esm:8443/www/manager-service/services/listServices
https://esm:8443/www/core-service/services/listServices

Checking WSDL sections I don't see how could I know if the method requires POST or GET request, what kind of Content-Type should I set or what attributes are mandatory.

For example, testing one method (sorry, I don't remember which one was) API returns HTTP error and inside the trace I saw "<attribute name> is empty", so I understood that it was mandatory. After that, the method worked as expected. But now I'm working with getSecurityEvents from SecurityEventService and I only got 4xx errors.

Looking the WSDL I found next info:

<xs:element name="getSecurityEvents">
	<xs:complexType>
		<xs:sequence>
		<xs:element minOccurs="0" name="authToken" nillable="true" type="xs:string"/>
		<xs:element minOccurs="0" name="ids" nillable="true" type="xs:anyType"/>
		<xs:element minOccurs="0" name="timeField" nillable="true" type="ns1:ArcField"/>
		<xs:element minOccurs="0" name="startMillis" type="xs:long"/>
		<xs:element minOccurs="0" name="endMillis" type="xs:long"/>
		</xs:sequence>
	</xs:complexType>
</xs:element>

If I understand correctly, names are attributes. After some tries, my python code for this method looks like this:

def getSecurityEvents(authToken):
	endTime=int(round(time.time() * 1000))
	startTime=endTime-86400000	# 24h
	url=gvars.arcUrl+'/www/manager-service/rest/SecurityEventService/getSecurityEvents?alt=json'
	jsonData='{"sev.getSecurityEvents":{"sev.authToken":"'+ authToken +'","sev.startMillis":"'+ str(startTime) +'","sev.endMillis":"'+ str(endTime)+'"}}'
	headers = {'Content-Type':'application/json'}
	r = requests.post(url, verify=gvars.cert, data=jsonData, headers=headers)
	# Debug
	print jsonData
	print r
	values = r.json()
	print "-----"
	print values

The code doesn't work, so I asked for some help on https://community.softwaregrp.com/t5/ArcSight-User-Discussions/help-with-ESM-REST-API-and-getSecurityEvents/m-p/1657715 . @Marius2 typed some code where he uses the method and works, and I saw that it looks like "ids" parameter is mandatory.

So, my question is: how can I properly know what I need (request type, mandatory parameters, etc.) from WSDL instead of trying multiple requests and seeing what answers the ESM?

Regards.

Tags (3)
0 Likes
1 Solution

Accepted Solutions
Highlighted
Fleet Admiral
Fleet Admiral

Me and the API has a sort of love/hate relationship, as i spent countless of hours working towards it.

Unfortunately at this time of writing i do not have enough time to cover everything, so instead i leave with a few documentation links comments and an example script i wrote in python to show as an example.

First place for documentation should be the ServiceLayer API guides, these will cover most of your needs for the official Supported API calls, which calls that is, is mentioned in the documentation. Check the ESM section of this site on documentation, and search for ServiceLayer, there is one for core and one for manager.

For the non-supported API calls, that is still there and is there in your listService HTTP documentation you are currently using, you will have to either look at the javadocs (referenced in the official ServiceLayer docs), or you have to grab the *ServiceImpl.class files from the ESM (hint: try to run "find /opt/arcsight -name *ServiceImpl.class*") Open the class files in JD-GUI or something similar, and you will see what each request expects in return.

In terms of POST/GET, this API was originally made for internal application calls, so it is not 100% yet, almost all calls supports both POST and GET for some reason, and the format of params in GET, and body in POST is always the same format, check my example script for a few format examples.

Later today/tomorrow i can write some more info if you need it, gotta run!

"""Example script for ArcSight community

Example script to communicate and fetch an event in ArcSight ESM
"""
import json
import argparse
import requests

def parse_command_line():
    """Stores the values passed by the user when running the script"""
    parser = argparse.ArgumentParser(description='Script to convert subnets to full descriptive ranges')
    parser.add_argument('--username', required=True, help='ESM Username')
    parser.add_argument('--password', required=True, help='ESM Password')
    parser.add_argument('--host', required=True, help='IP or hostname of ESM')
    parser.add_argument('--port', required=False, help='Port to use for the requests', default='8443')
    parser.add_argument('--eventid', required=True, help='An existing EventID on ESM')
    parser.add_argument('--verifyssl', required=False, help='Verify SSL Certificate or not', default=False)

    args = parser.parse_args()

    return args

def api_request(url, data, requesttype, verifyssl):
    """Prepares all API requests

    Best to have one function handle all requests,
    in that way we can ensure all requests have the same
    way of catching both errors and formating responses.

    Arguments:
        url {string} -- Full url of the API call.
        data {array} -- An array of parameters, either sent as JSON or urlparams.
        requesttype {string} -- Either GET or POST.
        verifyssl {string} -- If we are verifying the SSL certificate or not.

    Returns:
        string -- Response of the API request in json format, no error control yet.
    """
    if not data:
        return "No input was given"

    headers = ({'Accept' : 'application/json', 'Content-Type': 'application/json'})

    if requesttype.lower() == "post":
        response = requests.post(url, json=data, verify=verifyssl, headers=headers)
        return response.json()

    #Todo: Add get, not enough time to finish
    return requests.get(url, data, verify=verifyssl, headers=headers)

def login(args):
    """Logs the user in

    Logs the user in with the supplied username and password, and returns a token

    Arguments:
        username {string} -- Username of the ESM API user
        password {string} -- Password of the ESM API user

    Returns:
        string -- Userid if the authentication is successful
    """
    url = 'https://' + args.host + ':' + args.port + '/www/core-service/rest/LoginService/login'
    data = {
        "log.login" : {
            "log.login" : args.username,
            "log.password" : args.password
        }
    }
    response = api_request(url, data, 'post', args.verifyssl)

    return response["log.loginResponse"]["log.return"]

def get_event(args, authtoken):
    """Returns detailed information about events

    Expects minimum one ID, or an array, then returns details about each ID

    Arguments:
        args {array} -- Array of arguments supplied from the commandline
        authtoken {string} -- The authentication token of the user

    Returns:
        array -- Returns an array of results, depending on how many id's was requested
    """
    url = 'https://' + args.host + ':' + args.port + '/www/manager-service/rest/SecurityEventService/getSecurityEvents'
    data = {
        "sev.getSecurityEvents" : {
            "sev.authToken" : authtoken,
            "sev.ids" : args.eventid,
            "sev.startMillis" : "-1",
            "sev.endMillis" : "-1"
        }
    }

    return api_request(url, data, 'post', args.verifyssl)

def main():
    """Lazy main to get rid of linting errors

    Some random docs
    """
    args = parse_command_line()
    authtoken = login(args)
    event_details = get_event(args, authtoken)
    try:
        print(json.dumps(event_details))
    except ValueError as error:
        print(error)

if __name__ == '__main__':
    main()

Ensure you have requests, argparse and json library available (can check with pip if you don't have them).

Script is run like so:

 

python test.py --username someuser --password "somepassword" --host 192.168.1.1 --eventid 6565274

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius

View solution in original post

3 Replies
Highlighted
Fleet Admiral
Fleet Admiral

Me and the API has a sort of love/hate relationship, as i spent countless of hours working towards it.

Unfortunately at this time of writing i do not have enough time to cover everything, so instead i leave with a few documentation links comments and an example script i wrote in python to show as an example.

First place for documentation should be the ServiceLayer API guides, these will cover most of your needs for the official Supported API calls, which calls that is, is mentioned in the documentation. Check the ESM section of this site on documentation, and search for ServiceLayer, there is one for core and one for manager.

For the non-supported API calls, that is still there and is there in your listService HTTP documentation you are currently using, you will have to either look at the javadocs (referenced in the official ServiceLayer docs), or you have to grab the *ServiceImpl.class files from the ESM (hint: try to run "find /opt/arcsight -name *ServiceImpl.class*") Open the class files in JD-GUI or something similar, and you will see what each request expects in return.

In terms of POST/GET, this API was originally made for internal application calls, so it is not 100% yet, almost all calls supports both POST and GET for some reason, and the format of params in GET, and body in POST is always the same format, check my example script for a few format examples.

Later today/tomorrow i can write some more info if you need it, gotta run!

"""Example script for ArcSight community

Example script to communicate and fetch an event in ArcSight ESM
"""
import json
import argparse
import requests

def parse_command_line():
    """Stores the values passed by the user when running the script"""
    parser = argparse.ArgumentParser(description='Script to convert subnets to full descriptive ranges')
    parser.add_argument('--username', required=True, help='ESM Username')
    parser.add_argument('--password', required=True, help='ESM Password')
    parser.add_argument('--host', required=True, help='IP or hostname of ESM')
    parser.add_argument('--port', required=False, help='Port to use for the requests', default='8443')
    parser.add_argument('--eventid', required=True, help='An existing EventID on ESM')
    parser.add_argument('--verifyssl', required=False, help='Verify SSL Certificate or not', default=False)

    args = parser.parse_args()

    return args

def api_request(url, data, requesttype, verifyssl):
    """Prepares all API requests

    Best to have one function handle all requests,
    in that way we can ensure all requests have the same
    way of catching both errors and formating responses.

    Arguments:
        url {string} -- Full url of the API call.
        data {array} -- An array of parameters, either sent as JSON or urlparams.
        requesttype {string} -- Either GET or POST.
        verifyssl {string} -- If we are verifying the SSL certificate or not.

    Returns:
        string -- Response of the API request in json format, no error control yet.
    """
    if not data:
        return "No input was given"

    headers = ({'Accept' : 'application/json', 'Content-Type': 'application/json'})

    if requesttype.lower() == "post":
        response = requests.post(url, json=data, verify=verifyssl, headers=headers)
        return response.json()

    #Todo: Add get, not enough time to finish
    return requests.get(url, data, verify=verifyssl, headers=headers)

def login(args):
    """Logs the user in

    Logs the user in with the supplied username and password, and returns a token

    Arguments:
        username {string} -- Username of the ESM API user
        password {string} -- Password of the ESM API user

    Returns:
        string -- Userid if the authentication is successful
    """
    url = 'https://' + args.host + ':' + args.port + '/www/core-service/rest/LoginService/login'
    data = {
        "log.login" : {
            "log.login" : args.username,
            "log.password" : args.password
        }
    }
    response = api_request(url, data, 'post', args.verifyssl)

    return response["log.loginResponse"]["log.return"]

def get_event(args, authtoken):
    """Returns detailed information about events

    Expects minimum one ID, or an array, then returns details about each ID

    Arguments:
        args {array} -- Array of arguments supplied from the commandline
        authtoken {string} -- The authentication token of the user

    Returns:
        array -- Returns an array of results, depending on how many id's was requested
    """
    url = 'https://' + args.host + ':' + args.port + '/www/manager-service/rest/SecurityEventService/getSecurityEvents'
    data = {
        "sev.getSecurityEvents" : {
            "sev.authToken" : authtoken,
            "sev.ids" : args.eventid,
            "sev.startMillis" : "-1",
            "sev.endMillis" : "-1"
        }
    }

    return api_request(url, data, 'post', args.verifyssl)

def main():
    """Lazy main to get rid of linting errors

    Some random docs
    """
    args = parse_command_line()
    authtoken = login(args)
    event_details = get_event(args, authtoken)
    try:
        print(json.dumps(event_details))
    except ValueError as error:
        print(error)

if __name__ == '__main__':
    main()

Ensure you have requests, argparse and json library available (can check with pip if you don't have them).

Script is run like so:

 

python test.py --username someuser --password "somepassword" --host 192.168.1.1 --eventid 6565274

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius

View solution in original post

Highlighted
Lieutenant
Lieutenant
Hi Marius, Thanks for your script and description! One thing I've observed is that methods like login work with get and post, so I'll try both requests for every methon in the future. Another question I have is if it's possible to get a list of security events created by rules in a period of time. I though I could do it with getSecurityEvents but I need all id's and I don't see how to get all of those id's. Any idea? Regards.
0 Likes
Highlighted
Fleet Admiral
Fleet Admiral

Ah yes, so unfortunately there is no stable API to gather data directly, but there is a stable API to gather Query data, so what we have done is created a large amount of queries, attached a queryviewer to all of them, and then we grab the queryviewer data. And in that sense, you can in theory gather any data you want, because the queries supports all types of conditions and local variables etc.

You might have noticed the pattern in request and response body, where in the request it always looks like this:

Random Requests

{
  "{{prefix}}.addCaseEvents":{
    "{{prefix}}.authToken": "{{authToken}}",
    "{{prefix}}.caseId": "{{caseid}}",
    "{{prefix}}.eventIds": "200165022"
  }
}

 

Random Response:

{
    "log.loginResponse": {
        "log.return": "R89tt0pTLXGguADK7-_uHF3s68n0MINSoI2GoHC8OxY."
    }
}

 

 So the buildup of any body when using POST is always going to be  

{
"PREFIX.NAMEOFSERVICE(Exameple "log.login": {
    "PREFIX.Value1": "Value"
    "PREFIX.Value2": "value"
}

 

Response will always be: 

{
"PREFIX.NAMEOFSERVICERESPONSE(like log.oginResponse) :  {
    "PREFIX.RETURNM": VALUE
}

 

So with that explained, let's say you create a 1 query, it's condition is 1 day of data, with condition Generator ID (ID of rule).

You then create 1 queryviewer, explain that the only field you want is the ID's, and you then query that queryviewer from the API like so:

GET:

https://localhost:8443/www/manager-service/rest/QueryViewerService/getMatrixData?authToken=SOMETOKEN&id=RESOURCEIDOFQUERYVIEWER.

POST, since we already know how GET looks, i can guess that the post will be this:

{
  "{{prefix}}.getMatrixData": {
    "{{prefix}}.authToken": "AUTHTOKEN"
    "{{prefix}}.id": "RESOURCEID"
  }
}

Hopefully this thought process makes sense?

You should then have an array of ID's returned, use this array in your getSecurityEvents API call, it supports an array of ID's, so you can call them all in one go, instead of looping through all of them.

For more detailed descriptions you will have to look in the javadocs, as this is not a officially available API call, though has appeared in many presentation slides at conferences etc.

For knowing with prefix to use, the easieast resource is the Impl.class file i talked about, here is how it looks, the 3-4 characters on the right side is the prefix.

ap<String, String> ns = new HashMap();
    ns.put("http://model.v1.service.manager.product.arcsight.com/model/", "mod");
    ns.put("http://www.w3.org/2001/XMLSchema-instance", "xsi");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/activeListService/", "act");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/archiveReportService/", "arc");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/caseService/", "cas");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/conAppService/", "cap");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/connectorService/", "con");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/dashboardService/", "das");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/dataMonitorQoSService/", "dmq");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/dataMonitorService/", "dat");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/drilldownListService/", "drl");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/drilldownService/", "dri");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/fieldSetService/", "fie");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/fileResourceService/", "fil");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/graphService/", "gra");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/groupService/", "gro");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/internalService/", "int");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/managerAuthenticationService/", "man");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/networkService/", "net");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/portletService/", "por");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/queryService/", "que");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/queryViewerService/", "qvs");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/reportService/", "rep");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/resourceService/", "res");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/securityEventIntrospectorService/", "sei");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/securityEventService/", "sev");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/serverConfigurationService/", "ser");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/userResourceService/", "use");
    ns.put("http://ws.v1.service.resource.manager.product.arcsight.com/viewerConfigurationService/", "vie");
    ns.put("http://ws.v1.service.manager.product.arcsight.com/infoService/", "inf");
    ns.put("http://ws.v1.service.manager.product.arcsight.com/managerSearchService/", "mss");

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.