Install of Analytics Server 5.0 fails

Hi everyone

After a fresh install of Analytics Server
(AM_4531_50_AnalyticsDashboard.tar.gz) on Centos 7.9 I can see the
following in the catalina log file:
,----
| 03-Feb-2021 17:54:50.682 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8445]]
| org.apache.catalina.LifecycleException: Protocol handler initialization failed
| at org.apache.catalina.connector.Connector.initInternal(Connector.java:1042)
| at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
| at org.apache.catalina.core.StandardService.initInternal(StandardService.java:533)
| at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
| at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1057)
| at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
| at org.apache.catalina.startup.Catalina.load(Catalina.java:690)
| at org.apache.catalina.startup.Catalina.load(Catalina.java:712)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| at java.lang.reflect.Method.invoke(Method.java:498)
| at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
| at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
| Caused by: java.lang.IllegalArgumentException: /opt/novell/devman/jcc/certs/ra/connector.keystore (No such file or directory)
| at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99)
| at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
| at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216)
| at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1141)
| at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1154)
| at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
| at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
| at org.apache.catalina.connector.Connector.initInternal(Connector.java:1039)
| ... 13 more
| Caused by: java.io.FileNotFoundException: /opt/novell/devman/jcc/certs/ra/connector.keystore (No such file or directory)
| at java.io.FileInputStream.open0(Native Method)
| at java.io.FileInputStream.open(FileInputStream.java:195)
| at java.io.FileInputStream.<init>(FileInputStream.java:138)
| at java.io.FileInputStream.<init>(FileInputStream.java:93)
| at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
| at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
| at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:121)
| at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:197)
| at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:207)
| at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:282)
| at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
| at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97)
| ... 20 more
`----

It complains about the miissing file
`/opt/novell/devman/jcc/certs/ra/connector.keystore' which I can
confirm does not exist.

The install and configure log files show the following error:
,----
| # /tmp/novell_access_manager/ar_configure_XX
| Installing JCC rpm: Installing JCC rpm: Preparing... ########################################
| Updating / installing...
| novell-jcc-5.0.0.0-624 ########################################
| /var/tmp/rpm-tmp.b3VZGe: line 13: : No such file or directory
| cp: ‘/opt/novell/devman/jcc/bin/novell-jcc’ and ‘/etc/init.d/novell-jcc’ are the same file
`----

,----
| # /tmp/novell_access_manager/ar_install_XX
| Configuring Analytics Server $CATALINA_PID was set but the specified file does not exist. Is Tomcat running? Stop aborted.
`----

I'm guessing because it can't connect to the application server as it
never started in the first place. Other than that there is no
significant error.

I tried to search in the install scripts at which point should the
connector.keystore file be created, but I couldn't find anything.

Can anybody help me?

  • Make sure you are following the system requirement for Analytics Server.

    https://www.netiq.com/documentation/access-manager-45/system-requirements/data/system-requirements.html#sys-req-analytics

    analytics_spec.PNG

     

    If you are not installing the product on tested OS and not using correct RAM CPU you may get some issue.

    Make sure you are following the firewall requirements.

    https://www.netiq.com/documentation/access-manager-45/install_upgrade/data/b651hwh.html#ar-firewallports-table

    If above requirements are met and you still see the same issue, try doing reimporting the Analytics Server.

    https://www.netiq.com/documentation/access-manager-45/admin/data/analytics-server-manageconf.html#import-ar-script

     

  • Use /cyberres/accmgmt/accessmanager/f/namdashbeta/352987/integrate-nam-with-analytics-dashboard---easy-steps for easy install and setup.

  • Thank you for the response.

     

    I booted up a new centos 7.8 server with 10G RAM, firewalls disabled in both Aanlytics Server and NAM AIO and followed the guide you provided (I was taking the same steps) but the error persists:


    04-Feb-2021 15:41:49.380 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.librar
    y.path: [/usr/lib64]

    04-Feb-2021 15:41:50.103 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8445"]

    04-Feb-2021 15:41:50.345 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8445]]
           org.apache.catalina.LifecycleException: Protocol handler initialization failed
                   at org.apache.catalina.connector.Connector.initInternal(Connector.java:1042)
                   at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                   at org.apache.catalina.core.StandardService.initInternal(StandardService.java:533)
                   at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                   at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1057)
                   at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                   at org.apache.catalina.startup.Catalina.load(Catalina.java:690)
                   at org.apache.catalina.startup.Catalina.load(Catalina.java:712)
                   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                   at java.lang.reflect.Method.invoke(Method.java:498)
                   at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
                   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
           Caused by: java.lang.IllegalArgumentException: /opt/novell/devman/jcc/certs/ra/connector.keystore (No such file or directory)
                   at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99)
                   at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                   at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216)
                   at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1141)
                   at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1154)
                   at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
                   at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
                   at org.apache.catalina.connector.Connector.initInternal(Connector.java:1039)
                   ... 13 more
           Caused by: java.io.FileNotFoundException: /opt/novell/devman/jcc/certs/ra/connector.keystore (No such file or directory)
                   at java.io.FileInputStream.open0(Native Method)
                   at java.io.FileInputStream.open(FileInputStream.java:195)
                   at java.io.FileInputStream.<init>(FileInputStream.java:138)
                   at java.io.FileInputStream.<init>(FileInputStream.java:93)
                   at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)

    As this is a test environment I don't have a RHEL license. I will try to get a test RHEL server if I can.

  • I believe I discovered the problem (not yet the solution) the novell-jcc script relies heavily on functions that are available in SLES but not in RHEL:

    # Shell functions sourced from /etc/rc.status:
    #      rc_check         check and set local and overall rc status
    #      rc_status        check and set local and overall rc status
    #      rc_status -v     be verbose in local rc status and clear it afterwards
    #      rc_status -v -r  ditto and clear both the local and overall rc status
    #      rc_status -s     display "skipped" and exit with status 3
    #      rc_status -u     display "unused" and exit with status 3
    #      rc_failed        set local and overall rc status to failed
    #      rc_failed <num>  set local and overall rc status to <num>
    #      rc_reset         clear both the local and overall rc status
    #      rc_exit          exit appropriate to overall rc status
    #      rc_active        checks whether a service is activated by symlinks
    #      rc_splash arg    sets the boot splash screen to arg (if active) .

    . /etc/rc.status

    The script in facts complains that it can't source the /etc/rc.status file. And  the JCC service can't start.

  • Verified Answer

    Thanks for digging it. However, the document claims to support RHEL 7.7 and 7.8.

  • I believe some checks are made during the installation which prepares some of the file based on OS. If it is not RHEL it may be considered as SLES .

  • Finally I could try in RHEL 7.8.

     

    The same errors occur:

    catalina.out

    09-Feb-2021 10:08:45.187 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.librar
    y.path: [/usr/lib64]
    09-Feb-2021 10:08:45.777 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8445"]
    09-Feb-2021 10:08:46.000 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8445]]
           org.apache.catalina.LifecycleException: Protocol handler initialization failed
                   at org.apache.catalina.connector.Connector.initInternal(Connector.java:1042)
                   at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                   at org.apache.catalina.core.StandardService.initInternal(StandardService.java:533)
                   at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                   at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1057)
                   at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                   at org.apache.catalina.startup.Catalina.load(Catalina.java:690)
                   at org.apache.catalina.startup.Catalina.load(Catalina.java:712)
                   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                   at java.lang.reflect.Method.invoke(Method.java:498)
                   at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
                   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
           Caused by: java.lang.IllegalArgumentException: /opt/novell/devman/jcc/certs/ra/connector.keystore (No such file or directory)
                   at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99)
                   at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                   at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216)
                   at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1141)
                   at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1154)
                   at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
                   at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
                   at org.apache.catalina.connector.Connector.initInternal(Connector.java:1039)
                   ... 13 more
           Caused by: java.io.FileNotFoundException: /opt/novell/devman/jcc/certs/ra/connector.keystore (No such file or directory)

                  at java.io.FileInputStream.open0(Native Method)
                   at java.io.FileInputStream.open(FileInputStream.java:195)
                   at java.io.FileInputStream.<init>(FileInputStream.java:138)
                   at java.io.FileInputStream.<init>(FileInputStream.java:93)
                   at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
                   at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
                   at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:121)
                   at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:197)
                   at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:207)
                   at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:282)
                   at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
                   at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97)
                   ... 20 more

     

  • You guys are dancing around the issue with requirements.

     

    The keystore you need was not created.  There needs to be a trusted certificate in the keystore and it did not get created.  Why, could be Centos vs RHEL issue, but not clear.

     

    You have the path of the keystore.  Go make it. Odds are almost certain the password should be changeit or default then add in the public keys of every CA involved.  (In fact the way   could really help here would be to suggest what belongs in the keystore and what the password should be (if defaulted, if not, where the password should be set).

    I would look at the server.xml for the Tomcat instance and at the <Connector> line for port 8445 and see what cert it is using and where it gets it from.

     

    Just pointing requirements, while perhaps an approach, is often not that helpful.

  • Thank you geoff.

     

    Of course I created the keystore that was missing. Oddly the password in the server.xml file was getting populated with a pseudo random string and was a pointing to the rc.keystore that didn't exist.  That's when I realized that the problem with the dashboard wasn't really tomcat but rather the JCC service,that can't start because it relies on functions and programs (ej checkproc, startproc) that exist in SLES but not in RHEL. I guess I could alias the commands to suitable equivalents, but I'd rather this worked without that much hassle.

     

    I'm starting to wonder if this was even tested in red hat.

  • As per the documentation this is tested with RHEL. Creating keystore manually and changing scripts files may work for now but it can bring side effects. I would suggest to create a support ticket for this. Please share the version of Analytics Server from troubeshooting -> version.