REST Driver and Pagination (Link Header with rel="next")

Hi!

I´m working on an IDM integration of an existing REST API using IDM 4.8.1 and  the REST driver 1.1.1.0. The driver has to provide user and group entitlements with value from the application.

The API has an endpoint to get the list of available permissiongroups, but as result of a GET request the API returns only a limited number (currently 100) of objects plus a Link header with rel="next" to get the next.

Level 7 Trace shows the returned header (with anonymized URLs):

[09/15/20 14:27:48.668]:RESTDRIVER PT:RESTDRIVER: poll
[09/15/20 14:27:48.669]:RESTDRIVER PT:RESTDRIVER: Custom: preparing GET to apiserver.domain/.../permissiongroups
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Resetting headers
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Setting the following HTTP request properties:
Authorization: <content suppressed>
[09/15/20 14:27:48.670]:RESTDRIVER PT:RESTDRIVER: Accept:application/json
[09/15/20 14:27:48.671]:RESTDRIVER PT:RESTDRIVER: Did a HTTP GET with 0 bytes of data to apiserver.domain/.../permissiongroups
[09/15/20 14:27:48.671]:RESTDRIVER PT:RESTDRIVER: *******************************************************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: **********************LOGGING REQUEST******************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: *******************************************************
[09/15/20 14:27:48.672]:RESTDRIVER PT:RESTDRIVER: Request URL :apiserver.domain/.../permissiongroups
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Http Method : GET
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Sending http request with below headers :-
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Authorization: <content suppressed>
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: Accept: application/json
[09/15/20 14:27:48.673]:RESTDRIVER PT:RESTDRIVER: ***************************END**************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ********************************************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ***********************LOGGING RESPONSE*****************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: ********************************************************
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: Http response code : 200
[09/15/20 14:27:48.943]:RESTDRIVER PT:RESTDRIVER: Http response status : HTTP/1.1 200 OK
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Getting http response with below headers :-
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Date: Tue, 15 Sep 2020 12:27:40 GMT
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Server: Apache
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Expires: 0
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Cache-Control: no-cache, no-store, must-revalidate
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Pragma: no-cache
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Content-Length: 11508
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Link: <apiserver.domain/.../permissiongroups rel="next"
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Keep-Alive: timeout=3, max=10000
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Connection: Keep-Alive
[09/15/20 14:27:48.944]:RESTDRIVER PT:RESTDRIVER: Content-Type: application/json
[09/15/20 14:27:48.945]:RESTDRIVER PT:RESTDRIVER: Sending http response with body :-
[09/15/20 14:27:48.945]:RESTDRIVER PT:RESTDRIVER: {"liste":[{..... <the first 100 records> }]}

The API does not allow to force a complete result set, or a required number of results in the request, so I have to follow the Link.

I could not find any information on how to get the REST driver to follow the links to get the complete data.

Would be grateful for any ideas...

best regards,
Thomas

PS: This is also an issue from publisher poll request.

 

 

Parents Reply
  • Hi Casper,

    With the enhancement in place with a forthcoming release, there wouldn't be a need for this less than optimum mechanism to be documented.


    Let me see what can be done in the short term. Additionally, we already have experts on the community(including you!) who I hope would offer guidance!

    Regards,
    Abhishek

Children