Cybersecurity
DevOps Cloud (ADM)
IT Operations Cloud
Ran into a very odd error in ID Apps. This was a IDM 4.6.4 box upgraded to 4.8 then 4.8.1.
Logins failed with a blank screen. Tested the usual culprits. Certificates, hostnames, ports, etc.
Everything was good. Compared to the Dev box and everything looked the same or good.
Catalina.out had nothing in the logs. Osp.log however had a crazy looking error.
Preamble: [OIDP] Priority Level: FINER Java: internal.osp.framework.UIResponder$Response.setResponse() [1699] thread=https-jsse-nio-8443-exec-10 Time: 2020-09-17T14:05:36.727-0400 Elapsed time: 5.720 milliseconds Log Data: Set response: Forwarding: Page: /idm/jsp/saml2post.jsp Error committing response for /osp/a/idm/auth/oauth2/grant: org.apache.jasper.JasperException: Unable to compile class for JSP =>javax.el.ELException: Unable to find ExpressionFactory of type: # Licensed to the Apache Software Foundation (ASF) under one or more =>java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more org.apache.jasper.JspCompilationContext: JspCompilationContext.java: compile: 610 org.apache.jasper.servlet.JspServletWrapper: JspServletWrapper.java: service: 368 org.apache.jasper.servlet.JspServlet: JspServlet.java: serviceJspFile: 386 org.apache.jasper.servlet.JspServlet: JspServlet.java: service: 330 javax.servlet.http.HttpServlet: HttpServlet.java: service: 742 org.apache.catalina.core.ApplicationFilterChain: ApplicationFilterChain.java: internalDoFilter: 231 org.apache.catalina.core.ApplicationFilterChain: ApplicationFilterChain.java: doFilter: 166 org.apache.tomcat.websocket.server.WsFilter: WsFilter.java: doFilter: 52 org.apache.catalina.core.ApplicationFilterChain: ApplicationFilterChain.java: internalDoFilter: 193 org.apache.catalina.core.ApplicationFilterChain: ApplicationFilterChain.java: doFilter: 166 org.apache.catalina.core.ApplicationDispatcher: ApplicationDispatcher.java: invoke: 728 org.apache.catalina.core.ApplicationDispatcher: ApplicationDispatcher.java: processRequest: 470 org.apache.catalina.core.ApplicationDispatcher: ApplicationDispatcher.java: doForward: 395 org.apache.catalina.core.ApplicationDispatcher: ApplicationDispatcher.java: forward: 316
(There is more I am snipping for brevity) and then the interesting Caused By: line.
Caused by: javax.el.ELException: Unable to find ExpressionFactory of type: # Licensed to the Apache Software Foundation (ASF) under one or more javax.el.ExpressionFactory: ExpressionFactory.java: newInstance: 154 javax.el.ExpressionFactory: ExpressionFactory.java: newInstance: 93 org.apache.jasper.compiler.PageInfo: PageInfo.java: <init>: 79 org.apache.jasper.compiler.Compiler: Compiler.java: generateJava: 116 org.apache.jasper.compiler.Compiler: Compiler.java: compile: 374 org.apache.jasper.compiler.Compiler: Compiler.java: compile: 351 org.apache.jasper.compiler.Compiler: Compiler.java: compile: 335 org.apache.jasper.JspCompilationContext: JspCompilationContext.java: compile: 595 org.apache.jasper.servlet.JspServletWrapper: JspServletWrapper.java: service: 368 org.apache.jasper.servlet.JspServlet: JspServlet.java: serviceJspFile: 386 org.apache.jasper.servlet.JspServlet: JspServlet.java: service: 330 javax.servlet.http.HttpServlet: HttpServlet.java: service: 742
If you read that, it seems like it is saying the saml2post.jsp file is failing to compile. This is an Apache Tomcat level problem not NAM/OSP. In fact we turned off SAML and did Name/Password and same problem.
Searching the inter-tubes, we found that there is a bug in the EL Loader in Tomcat versions, supposedly fixed in the version IDM installs. Basically the issue is that the broken EL Loader thinks that the commented text (# Apache Foundation..) is a class load call, and of course fails with the goofy error shown above.
The good news is I found that the JAR in question is named el-api.jar and resides in the lib directory under tomcat.
What was odd, and I cannot explain (If you can, please comment) is that there were two installs of Tomcat. (rpm -qa | grep tomcat showed us using 9.0.33 of Tomcat installed)
One is in:
/opt/netiq/idm/apps/tomcat (this is where conf, webapps all reside as expected)
/opt/netiq/idm/tomcat which is where the new version is installed. We deleted the el-api.jar from the /opt/netiq/idm/apps/tomcat/lib which is where we think it runs from and everything started working again,
But the /opt/netiq/idm/apps/tomcat/bin/setenv.sh defines the tomcat path to the /opt/netiq/idm/apps/tomcat path.
So what is the /opt/netiq/idm/tomcat install for? Also, the files there are newer, from 2020 whereas what I thought was the real directory are from 2018.
This seems to resolve this issue, but I suspect there is a bigger underlying issue here.