Generic REST Collector supporting odata

I am considering to connect an application offering an odata compliant REST API to IG (3.7.3), and I am wondering if it is possible - and/or suggested - to modify the shippir REST collector for GiHub.

Unfortunately, without modification the REST GitHub Collectors cannot be used for this challenge, since they have some "hard" coded stuff in them.

After exporting the collector I found only the following two functions:

  • getRestResource(connectionParm, token, pageSize)
  • getNextPageURL(response) 

I believe, the responsibility of the first one is to implement the REST request and compose the JSON object representing the collection.

The code seams to utilize a paged search (token = '/users?per_page=' + pageSize;) which can not work for two reasons when working with odata:

1) the schema (/users) is not correct

2) odata does not support paged searches in this way.

The schema can be possibly simply set to the one used in the odata implementation, but the paged search has to be implemented differently.

 In addition to these findings, the first function uses some "HelpperFunctions" provided by com.netiq.daas.restconnector.internal.HelperFunctions!

Can anybody provide some information regarding the HelperFunctions class, and the structure of the final JSON object, IG is able to process?

In case a paged search can not be implemented for odata, is there a technical limitation on the data received in one REST response?

To replace a paged search odata offers the parameters &size and $skip - so a paged search could be replaced by a loop running several requests with in increasing value in the $skip parameter, if needed!

I am not sure, regarding the purpose of the function getNextPageURL(response)  - is this needed, or does this need to be adapted as well?

Kind regards

Thorsten

Parents
  • Hello Thorsten,

    1) What you downloaded was the RESTGitHub (Account or Permission) Collector Template

    2) At this time, we do not offer a Generic set of REST Collectors (Identity, Account, or Permission).  This is on our Roadmap and an aspect we are investigating.
    What we provide are specific collectors that utilize REST.

    Is it possible to utilize SCIM as we do provide a Generic set of SCIM collectors?

    I will review the information about OData as that might be an aspect to look at regarding the ability to have a Generic set of REST collectors.

    Sincerely,
    Steven Williams
    Principal Enterprise Architect
    OpenText Cybersecurity

  • Hello Steven,

    thanks for your fast answer!

    I modified the getRestResource() function in a way it does not tray to run a paged search, and I replaced the users schema with the one which would work in my case.

    token = '/opx2_user?$select=class,object_number,name,comment,list_of_groups'

    The $select limits the properties/attributes returned for each object!

    Running a collection I can see the REST Response code in catalina.out presented in the following way:

    {"@odata.context":"https://<serverFQFDN>/odata/$metadata#opx2_user(class,object_number,name,comment,list_of_groups)","value":[
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(448911)",
    "class" : "USER",
    "object_number" : 448911,
    "name" : "PROJEKTLEITER",
    "comment" : "Projektleiter",
    "list_of_groups" : "PROJECT_MANAGER,ALL,P_PREMIER"
    },
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(196911)",
    "class" : "USER",
    "object_number" : 196911,
    "name" : "SUPPORT",
    "comment" : "Support User",
    "list_of_groups" : "ALL,P_APP"
    },
    {
    "@odata.id":"https://<serverFQFDN>/odata/opx2_user(1669821)",
    "class" : "USER",
    "object_number" : 1669821,
    "name" : "N10005",
    "comment" : "Freier Dienstag TEst",
    "list_of_groups" : "ADMIN,ALL,P_FADM"
    }
    ]
    }

    But IG does not "translate" this to Accounts, and therefore the collection does not import any object.

    I still believe, there is a way by further manipulating the gerRestRessource() function - but I am not 100% sure how - and even if I get it to work, if I need to implement a replacement for the paged search!

    Kind regards

    Thorsten 

  • Hello Steven,

    I applied a few more changes to the getRestResource() function used in the GitHub REST collector and finally I got the users collected running a test collection!

    I changed the function in the way, it only sends one REST request, and iterates once through the JSON provided in the resulting response. In this case, there is a Content.value property providing an array in the JSON object of all users/accounts!

    I use the for-loop within the function to setup user properties of the result JSON object for each value found!

    As far as I interpret the original code of the function it used a link in the first REST response to run additional REST requests for each individual object, but I guess this is not needed, is it?

    In my PoC I am not using a paged search, but request all objects at once - I hope this would not be a problem, especially since I am requesting only a few properties of users/Accounts!

    I believe, with some more additional time spend on further changes of this function it would not be a problem to use the $size and &skip parameters to "simulate" a paged search, but still the question is, if this is mandatory?

    Kind regards

    Thorsten

  • Hello Steven,

    as written before, I modified the getRestResource() to request all users or groups in one response. I further modified the code to add an outer loop utilizing the $top and $skip odata parameters. The end of the loop is defined by a variable providing the maximum results provided by the REST request.- For this purpose, odata offers the $count parameter. 

    var objectsFound = ((helpperFunctions.httpClient(connectionParm, 'GET','/opx2_user?$count', '{}')));

    According to the debug data offered in catalina.out the REST request is working, but the function terminates because the response is not provided in JSON.

    [INFO] 2024-04-15 15:18:00.398 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Connection Framework execution info message: 'Executing the request to: sparkassen-versicherung-dev.planisware.live/.../opx2_user
    [INFO] 2024-04-15 15:18:00.398 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Connection Framework execution info message: 'Executing the request to: sparkassen-versicherung-dev.planisware.live/.../opx2_user
    [FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Response Body: 48
    [FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.BasicAuthentication] [DAAS] Response Body: 48
    [FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Http Response Code :200
    [FINE] 2024-04-15 15:18:00.505 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Http Response Code :200
    [FINE] 2024-04-15 15:18:00.506 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Response is a not a valid JSONObject or JSONArray
    [FINE] 2024-04-15 15:18:00.506 [com.netiq.daas.restconnector.restauthentication.AbstractHttpsOperation] [DAAS] Response is a not a valid JSONObject or JSONArray
    [FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Received request to unload service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7
    [FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Received request to unload service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7
    [FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Decremented load count on service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7, load count: null
    [FINEST] 2024-04-15 15:18:03.306 [com.netiq.daas.daaservice.ServiceMap] [DAAS] Decremented load count on service: RestPlaniswareAccountTemplate-18-72-7ad751b80b8d42c98598b75746a27ad7, load count: null
    [SEVERE] 2024-04-15 15:18:03.306 [com.netiq.iac.persistence.dcs.dce.thread.TestDataCollectionServiceThread] [IG-DTP] Data collection from DaaS failed

    is there any way to force the helpperFunction() to accept anything in this case? 

    The only other option would be to request all objects, or find a way to exit the outer loop in case no more objects are returned in one iteration ...

    Kind regards

    Thorsten

  • HI,

    since I cannot use the REST Request from the helpperFunction() to work with a string in the REST response, I changed the outer loop to while-loop like:

    while (responseArray.length > 0)

    Within this loop I run several times the REST Request with the URL parameters $top and $skip. top is set to the page size and skip is incremented in each loop by the page size.

    This is working as expected, and finally I end with a resource JSON object containing all users returend.

    {
    "resource":[
    {
    "id":123,
    "name":"INTRANET",
    "comment":"INTRANET ",
    "list_of_groups":"MODULES,ALL",
    "user":{
    "@odata.id":"FQDN/odata/opx2_user(123)",
    "class":"USER",
    "object_number":5625,
    "name":"INTRANET",
    "comment":"INTRANET ",
    "list_of_groups":"MODULES,ALL"
    }
    },
    {
    "id":1234,
    "name":"P_PLW_TEMPLATE_USER",
    "comment":"Template for profile P_PLW",
    "list_of_groups":"P_PLW",
    "user":{
    "@odata.id":"FQDN/alt/plw/localhost:8400/odata/opx2_user(1234)",
    "class":"USER",
    "object_number":196111,
    "name":"P_PLW_TEMPLATE_USER",
    "comment":"Template for profile P_PLW",
    "list_of_groups":"P_PLW"
    }
    }
    ]
    }

    This JSON code is simplified, but I double-checked the code on Catalina with the expected results, and found no problems.

    The code even matches my first attempt, requesting all users at once (no pagination with top & skip).

    But DAAS is not happy with the result in this case:

    [FINEST] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Execution of Javascript completed......
    [FINEST] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Execution of Javascript completed......
    [FINE] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Result returned by script must contain token
    [FINE] 2024-04-16 10:57:31.514 [com.netiq.daas.restconnector.collectorservice.RestResourceCollector] [DAAS] Result returned by script must contain token

    I suspect, this error is coming up, since during the last iteration of the while-loop the REST Response returned an empty Array! Or did I miss something else?

    I can see in catalina,  that the while loop is run several times with a new URL used in each iteration, and different objects contained in the response JSON.

    Due to the fact the JSON.stringify(result) I put to the debug code is as expected as well, I guess the logic of the new function is correct ....

    Can anybody provide some ideas, please?

    BTW: Is there any way to ease the process of testing a collector? In my case I have to delete the collector, import a new collector template and setup a new collector? This is quite hard, if I apply only some changes to the function ;-)

    Kind regards

    Thorsten

  • Finally, I found, the JSON response object needs a "token" element

    For my understanding this contains the URLs for the paged search  - in my case I believe this is not needed, therefore I chose no value for the token element.

    After adding this, the previous error is gone, but the import script is run infinitely! The only way to get further is to stop the collection!

    Can anybody tell me, what value shall be used for token to stop the looping?

    In the working collector the code reads:

    result['token'] = helpperFunctions.getNextPageURL(connectionParm, JSON.stringify(httpRes['Response_Headers']));

    But I guess there is a hard coded value to tell the collector to stop reading ....

    BTW: this is my current function:

    function getRestResource(connectionParm, token, pageSize) {

    var helpperFunctions = Java.type('com.netiq.daas.restconnector.internal.HelperFunctions');

    var skip = 0;

    var size=pageSize;

    var token ='/opx2_user?$select=class,object_number';

    var respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

    var httpRes = JSON.parse(respString);

    var responseArray = httpRes.Content.value;

    var maxObjects = responseArray.length;

    helpperFunctions.debug('--------------------------------------------------------------');

    helpperFunctions.debug('Page Size: ' + pageSize);

    helpperFunctions.debug('MaxObjects: ' + maxObjects);

    helpperFunctions.debug('--------------------------------------------------------------');

    token ='/opx2_user?$select=class,object_number,name,comment,list_of_groups&$top='+ size + '&$skip=' + skip;

    userResource = [];

    respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

    httpRes = JSON.parse(respString);

    responseArray = httpRes.Content.value;

    helpperFunctions.debug('--------------------------------------------------------------');

    helpperFunctions.debug('Response Array: ' + responseArray);

    helpperFunctions.debug('--------------------------------------------------------------');

    while (responseArray.length > 0){

      for (var i = 0; i < responseArray.length; i++) {

    if (responseArray[i]['class'] === 'USER') {

    var userObject = {};

    var id = responseArray[i]['id'];

    userObject['id'] = responseArray[i]['object_number'] ? responseArray[i]['object_number'] : '';

    userObject['name'] = responseArray[i]['name'] ? responseArray[i]['name'] : '';

    userObject['comment'] = responseArray[i]['comment'] ? responseArray[i]['comment'] : '';

    userObject['list_of_groups'] = responseArray[i]['list_of_groups'] ? responseArray[i]['list_of_groups'] : '';

    userObject['user'] = responseArray[i];

    helpperFunctions.debug('User Data ' + userObject);

    userResource.push(userObject);

    }

    }

    skip = skip + size;

    token ='/opx2_user?$select=class,object_number,name,comment,list_of_groups&$top='+ size + '&$skip=' + skip;

    respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

    httpRes = JSON.parse(respString);

    responseArray = httpRes.Content.value;

    helpperFunctions.debug('--------------------------------------------------------------');

    helpperFunctions.debug('Response Array: ' + responseArray);

    helpperFunctions.debug('UserResource Array: ' + userResource);

    helpperFunctions.debug('--------------------------------------------------------------');

    }

    var result = {};

    result['resource'] = userResource;

    helpperFunctions.debug('UserResource Array: ' + JSON.stringify(result));

    return JSON.stringify(result);

    }

Reply
  • Finally, I found, the JSON response object needs a "token" element

    For my understanding this contains the URLs for the paged search  - in my case I believe this is not needed, therefore I chose no value for the token element.

    After adding this, the previous error is gone, but the import script is run infinitely! The only way to get further is to stop the collection!

    Can anybody tell me, what value shall be used for token to stop the looping?

    In the working collector the code reads:

    result['token'] = helpperFunctions.getNextPageURL(connectionParm, JSON.stringify(httpRes['Response_Headers']));

    But I guess there is a hard coded value to tell the collector to stop reading ....

    BTW: this is my current function:

    function getRestResource(connectionParm, token, pageSize) {

    var helpperFunctions = Java.type('com.netiq.daas.restconnector.internal.HelperFunctions');

    var skip = 0;

    var size=pageSize;

    var token ='/opx2_user?$select=class,object_number';

    var respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

    var httpRes = JSON.parse(respString);

    var responseArray = httpRes.Content.value;

    var maxObjects = responseArray.length;

    helpperFunctions.debug('--------------------------------------------------------------');

    helpperFunctions.debug('Page Size: ' + pageSize);

    helpperFunctions.debug('MaxObjects: ' + maxObjects);

    helpperFunctions.debug('--------------------------------------------------------------');

    token ='/opx2_user?$select=class,object_number,name,comment,list_of_groups&$top='+ size + '&$skip=' + skip;

    userResource = [];

    respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

    httpRes = JSON.parse(respString);

    responseArray = httpRes.Content.value;

    helpperFunctions.debug('--------------------------------------------------------------');

    helpperFunctions.debug('Response Array: ' + responseArray);

    helpperFunctions.debug('--------------------------------------------------------------');

    while (responseArray.length > 0){

      for (var i = 0; i < responseArray.length; i++) {

    if (responseArray[i]['class'] === 'USER') {

    var userObject = {};

    var id = responseArray[i]['id'];

    userObject['id'] = responseArray[i]['object_number'] ? responseArray[i]['object_number'] : '';

    userObject['name'] = responseArray[i]['name'] ? responseArray[i]['name'] : '';

    userObject['comment'] = responseArray[i]['comment'] ? responseArray[i]['comment'] : '';

    userObject['list_of_groups'] = responseArray[i]['list_of_groups'] ? responseArray[i]['list_of_groups'] : '';

    userObject['user'] = responseArray[i];

    helpperFunctions.debug('User Data ' + userObject);

    userResource.push(userObject);

    }

    }

    skip = skip + size;

    token ='/opx2_user?$select=class,object_number,name,comment,list_of_groups&$top='+ size + '&$skip=' + skip;

    respString = ((helpperFunctions.httpClient(connectionParm, 'GET',token, '{}')));

    httpRes = JSON.parse(respString);

    responseArray = httpRes.Content.value;

    helpperFunctions.debug('--------------------------------------------------------------');

    helpperFunctions.debug('Response Array: ' + responseArray);

    helpperFunctions.debug('UserResource Array: ' + userResource);

    helpperFunctions.debug('--------------------------------------------------------------');

    }

    var result = {};

    result['resource'] = userResource;

    helpperFunctions.debug('UserResource Array: ' + JSON.stringify(result));

    return JSON.stringify(result);

    }

Children
  • Finally, I got it working!

    In my last attempt I set the "token" element in the result JSON the same way, the original script did:

    result['token'] =  helpperFunctions.getNextPageURL(connectionParm, JSON.stringify(httpRes['Response_Headers']));

    Now there is no error anymore, and IG reports the correct number of object (users) to be imported.

    Since there are only two functions in the original collector template and the undocumented two? functions of the helpperFunctions class, I am wondering, If OpenText could provide some details on the functions defined in the java-script and pagination-script of the template as well as the functions contained in the helpperFunctions class!

    This would make it more convenient to customize the existing (REST) collectors.

    Kind regards

    Thorsten