I get this error in catalina:
[https-openssl-nio-8443-exec-13] ERROR com.netiq.idm.auth.oauth.OAuthRestFilter - [RBPM] An error occurred while attempting to contact the authentication service.

And no error in the osp log (mode ALL).

Have you tried to reset the SAML configuration (via configupdate.sh) ?

Yes, I did.

Get this error from configupdate.bat , but I think it worked.

C:\NetIQ\IDM\apps\configupdate>configupdate.bat
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.netiq.internal.installer.idm.ldap.AppConfig$SSOConfiguration (file:/C:/NetIQ/IDM/apps/configupdate/lib/ldapconfig_support-5.0.0-346.jar) to method sun.security.tools.keytool.Main.main(java.lang.String[])
WARNING: Please consider reporting this to the maintainers of com.netiq.internal.installer.idm.ldap.AppConfig$SSOConfiguration
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

C:\NetIQ\IDM\apps\configupdate>

Also did not get any error in catalina (SAML gives typical error).

sma2006 said:

An illegal reflective access operation has occurred

That's just a warning (and not an error) that comes with the usage of Java 11.

Java 17 will enforce this for some packages: https://docs.oracle.com/en/java/javase/17/migrate/migrating-jdk-8-later-jdk-releases.html#GUID-7BB28E4D-99B3-4078-BDC4-FC24180CE82B

I normally only see this error: "ERROR com.netiq.idm.auth.oauth.OAuthRestFilter - [RBPM] An error occurred while attempting to contact the authentication service." when there is something wrong with SAML.

Otherwise I sadly must say that I do not know what it otherwise would be.

The signing cert for the IDP you are connecting to, do you have the public keys from its signing and other certs in your osp.jks keystore?  Usually it is a trust issue between hosts that cause this.  If the SAML cert from the IDP is signed by something your Java does not trust you get errors like this.

Hello Geoffrey, 
I'm back from vacation now and did a new try:
- Import edir CA + osp host cert in osp.jk.
- Import edir CA + osp host cert in idm.jks.
- Import edir CA + osp host cert in cacert keystore
- Add this line in setenv.bat: -Dcom.sun.security.enableCRLDP=true

And it's working now.

Thanks a lot for your help.

Best.

Sylvain