Highlighted
Honored Contributor.
Honored Contributor.
1044 views

REST Driver and Pagination (Link Header with rel="next")

Hi!

I´m working on an IDM integration of an existing REST API using IDM 4.8.1 and  the REST driver 1.1.1.0. The driver has to provide user and group entitlements with value from the application.

The API has an endpoint to get the list of available permissiongroups, but as result of a GET request the API returns only a limited number (currently 100) of objects plus a Link header with rel="next" to get the next.

Level 7 Trace shows the returned header (with anonymized URLs):

[09/15/20 14:27:48.668]:RESTDRIVER PT:RESTDRIVER: poll
[09/15/20 14:27:48.669]:RESTDRIVER PT:RESTDRIVER: Custom: preparing GET to https://apiserver.domain/api/v1/permissiongroups?param1=value1&param2=value2
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Resetting headers
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Setting the following HTTP request properties:
Authorization: <content suppressed>
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Accept:application/json
[09/15/20 14:27:48.671]:RESTDRIVER PT:RESTDRIVER: Did a HTTP GET with 0 bytes of data to https://apiserver.domain/api/v1/permissiongroups?param1=value1&param2=value2
[09/15/20 14:27:48.671]:RESTDRIVER PT:RESTDRIVER: *******************************************************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: **********************LOGGING REQUEST******************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: *******************************************************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: Request URL :https://apiserver.domain/api/v1/permissiongroups?param1=value1&param2=value2
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Http Method : GET
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Sending http request with below headers :-
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Authorization: <content suppressed>
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Accept: application/json
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: ***************************END**************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ********************************************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ***********************LOGGING RESPONSE*****************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ********************************************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: Http response code : 200
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: Http response status : HTTP/1.1 200 OK
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Getting http response with below headers :-
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Date: Tue, 15 Sep 2020 12:27:40 GMT
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Server: Apache
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Expires: 0
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Cache-Control: no-cache, no-store, must-revalidate
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Pragma: no-cache
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Content-Length: 11508
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Link: <https://apiserver.domain/api/v1/permissiongroups?param1=value1&param2=value2&offset=100>; rel="next"
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Keep-Alive: timeout=3, max=10000
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Connection: Keep-Alive
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Content-Type: application/json
[09/15/20 14:27:48.945]:RESTDRIVER PT:RESTDRIVER: Sending http response with body :-
[09/15/20 14:27:48.945]:RESTDRIVER PT:RESTDRIVER: {"liste":[{..... <the first 100 records> }]}

The API does not allow to force a complete result set, or a required number of results in the request, so I have to follow the Link.

I could not find any information on how to get the REST driver to follow the links to get the complete data.

Would be grateful for any ideas...

best regards,
Thomas

PS: This is also an issue from publisher poll request.

 

 

19 Replies
Highlighted
Micro Focus Expert
Micro Focus Expert

Currently the REST driver returns the request headers with the response instead the ones send by the server. (I opened an SR for this). So to get the value for Link respose header, you'll probably have to look at writing a DocumentModifier: https://www.netiq.com/documentation/identity-manager-48-drivers/generic_rest/data/bvpan6k.html

 

--
Norbert
Highlighted
Knowledge Partner
Knowledge Partner

While this seems like a daunting task, it turns out, with basic Java skills you can do this.

 

What happens is there is a skelton file in the Driver Dev kit, with the various classes you can add to the SOAP or Delim text drivers.

They are basically a class definition that takes the document as input, and then returns your modified version. (For the most part).  So your code goes inside and does what is needed.  Now offhand, I have no idea how to get access to the headers... Nor how to write to them, but seems doable.

0 Likes
Highlighted
Outstanding Contributor.
Outstanding Contributor.

You mean the "Document Handling" class stuff?

I am not sure that  it will be possible to get the http headers after the XDS document is recreated and send to your custom document handler.

Highlighted
Honored Contributor.
Honored Contributor.

Thank you for your answeres.

Currently I understand:

- the REST drriver shim itself does NOT handle paged results (thats what i initially expected)

- the Link header from the response is not passed to the event ans is therfore not available in policies

- it is not sure, if a Java extension can get to the Link header

So it looks like the REST driver possibly is not the way to go...

I could look into writing my complete own driver shim or look into the new Invoke REST Endpoint action (whitch lacks the PATCH method at least according to the documentation).

Any other ideas?

best regards,
Thomas

 

 

Highlighted
Honored Contributor.
Honored Contributor.

Found the "IDM connector for Text Protocols (universal)"  at http://www.opns.be/idm-connector-for-text-protocols-universal/ According to the documentation is seems to be the REST driver that should be in the product.

Has anyone already worked with this driver?

regards,
Thomas

Highlighted
Outstanding Contributor.
Outstanding Contributor.

I suspect that the Rest driver came to existence from the Groupwise Driver (which came first), instead the other way around - which would have been the right way to do it.

And as Norbert mentioned there is an request to get access to the (returned) http hearders, now we're waiting.

 

Highlighted
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor

Thomas, Norbert’s SR should allow handling pagination (albeit painfully) through Document Modifiers in the short term.

As for supporting pagination natively, I would recommend that you log an ideas exchange request. We are actively looking at enhancing the capabilities of the REST driver, and this should get addressed with the next release of the driver.

Regards,
Abhishek

Highlighted
Outstanding Contributor.
Outstanding Contributor.

Hi Abhishek,

Will you document how this is done (in details) ?

Regards,
Casper

Highlighted
Micro Focus Frequent Contributor
Micro Focus Frequent Contributor

Hi Casper,

With the enhancement in place with a forthcoming release, there wouldn't be a need for this less than optimum mechanism to be documented.


Let me see what can be done in the short term. Additionally, we already have experts on the community(including you!) who I hope would offer guidance!

Regards,
Abhishek

Highlighted
Knowledge Partner
Knowledge Partner

I think what Casper is asking you, is how to handle the pagination in a document modifier?

I think he is referring to your first sentence "Norbert’s SR should allow handling pagination (albeit painfully) through Document Modifiers in the short term."

Not the second one.

0 Likes
Highlighted
Honored Contributor.
Honored Contributor.

@Abhishek Prasad  abs.  the pagination functionality is must-have  built in netiq REST driver, i would vote for this for sure, mostly REST api today if done (professionally) does pagination, example Microsoft Graph and other oDATA apis, so yes, it would be still possible to do work-around with current version,  but NetIQ REST driver should be smart enough to discover REST api and support pagination out of the box. 

one of the work around could be:

 

1) Issue query from subscriber command transformation

2) Output transformation should implement command 

3) Publisher input return data from REST api , as result set with pagination back to subscriber command

4) Subscriber command transformation recieves result set, parse data, and  submit new command with new pagination link

 

this would work, if the response of first query is not so big enough, so smart idea would be to issue query with filters possible to reduce the result-set received from service.

 

but yes, it requires some experience and knowledge of netiq.

 

Regards

/Maqsood

 

 

 

 

 

 

 

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.