# objdump -p /opt/novell/apache2/sbin/httpd

...

  RUNPATH              /opt/novell/pcre/lib:/opt/novell/apache2/lib

This means shared libraries by httpd and any module loaded by httpd (e.g. mod_ssl) will be searched for under /opt/novell/pcre/lib and /opt/novell/apache2/lib, respectively. Now, as to what shared libraries are being required by mod_ssl.so:

# objdump -p /opt/novell/apache2/libexec/mod_ssl.so

...

NEEDED libssl.so.1.0.0
NEEDED libcrypto.so.1.0.0

Do either of these exist in /usr/lib64 on RHEL 8?

achinayoung_wau said:

NEEDED libssl.so.1.0.0
NEEDED libcrypto.so.1.0.0

Do either of these exist in /usr/lib64 on RHEL 8?

Not supplied by RHEL, I searched and only found these:

/opt/netiq/common/openssl/lib64/libssl.so.1.0.0
/opt/novell/ssllib/libssl.so.1.0.0
/opt/novell/openssl/lib/libssl.so.1.0.0

and

/opt/netiq/common/openssl/lib64/libcrypto.so.1.0.0
/opt/novell/ssllib/libcrypto.so.1.0.0
/opt/novell/openssl/lib/libcrypto.so.1.0.0

Hmm.  In order to test my theory, I tried moving/renaming this:

/usr/lib64/libcrypto.so.1.1.1k

and tried to start the novell-apache2 service. When I did that, I got errors about libcrypto.so.1.1.1k missing.  So that is how I deduced it was trying to use the system provided crypto, but maybe that is not a correct assumption?

On RHEL 7.9 though, they're a libcrypto.so.1.0.0 in /usr/lib64 and a libssl.so.1.0.0.  They are NOT symlinks and they differ form what is in /opt/novell/ssllib.

Hmm, I wonder what would happen if I created a symlink on the RHEL 8.8 box in /usr/lib64 for the libcrypto and libssl in /opt/novell/ssllib?

Matt

If you look at the systemd unit file for novell-apache2, you find:

Environment="LD_LIBRARY_PATH=/opt/novell/ssllib:/opt/novell/openssl/lib"
ExecStart=/opt/novell/apache2/sbin/httpd $ARGL

I would modify ExecStart to run httpd under strace and take a look at the resulting output to determine which version of the libssl library is being used. You can possibly also try running httpd manually like so:

# LD_LIBRARY_PATH=/opt/novell/ssllib:/opt/novell/openssl/lib strace -s256 -f -o /tmp/apache.tr /opt/novell/apache2/sbin/httpd

then when it fails look at /opt/apache.tr and search for libssl or libcrypto.

If you cannot run httpd manually, modify ExecStart with the strace command then "systemctl daemon-reload" and restart the novell-apache2 service.

Ok, so I did that and I grep'd apache.tr for libcrypto.  It almost seems like it is loading/using both of them:

2561179 openat(AT_FDCWD, "/opt/novell/ssllib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 4
2561179 openat(AT_FDCWD, "/opt/novell/ssllib/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2561179 openat(AT_FDCWD, "/opt/novell/openssl/lib/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2561179 openat(AT_FDCWD, "/lib64/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 5
2561179 read(4, "1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 1"..., 1024) = 1024
2561179 read(4, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcryp"..., 1024) = 1024
2561179 openat(AT_FDCWD, "/opt/novell/ssllib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 7
2561179 read(7, "fe9c290e000 r--p 002b5000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c290e000-7fe9c2912000 rw-p 002e1000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-"..., 1024) = 1024
2561179 read(7, "7806 /opt/novell/nghttp2/lib/libnghttp2.so.14.22.0\n7fe9c7eb6000-7fe9c8127000 r-xp 00000000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /"..., 1024) = 1024
2561179 read(7, "fe9c290e000 r--p 002b5000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c290e000-7fe9c2912000 rw-p 002e1000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-"..., 1024) = 1024
2561179 read(7, "7806 /opt/novell/nghttp2/lib/libnghttp2.so.14.22.0\n7fe9c7eb6000-7fe9c8127000 r-xp 00000000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /"..., 1024) = 1024
2561179 read(7, "fe9c290e000 r--p 002b5000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c290e000-7fe9c2912000 rw-p 002e1000 fd:00 1443848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-"..., 1024) = 1024
2561179 read(7, "7806 /opt/novell/nghttp2/lib/libnghttp2.so.14.22.0\n7fe9c7eb6000-7fe9c8127000 r-xp 00000000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /"..., 1024) = 1024
2561179 read(15, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
2561179 read(15, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024
2561179 read(7, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
2561179 read(7, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024
2561179 read(7, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
2561179 read(7, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024
2561179 read(7, "848 /usr/lib64/libcrypto.so.1.1.1k\n7fe9c2912000-7fe9c2917000 rw-p 00000000 00:00 0 \n7fe9c2917000-7fe9c292b000 r-xp 00000000 fd:00 1439002 /usr/lib64/libresolv-2.28.so\n7fe9c292b000-7fe9c2b2b000 ---p 00014000 fd:00 14390"..., 1024) = 1024
2561179 read(7, " /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8127000-7fe9c8326000 ---p 00271000 fd:00 657027 /opt/novell/ssllib/libcrypto.so.1.0.0\n7fe9c8326000-7fe9c8352000 rw-p 00270000 fd:00 657027 /opt/novell/ssllib/libcrypto.s"..., 1024) = 1024

I see both the novell and system ones being read/referenced here.  What does that tell us?

Matt

Does running httpd on RHEL 7 also show more than one version of libssl being loaded?

Matt Weisberg  Check out https://support.microfocus.com/kb/doc.php?id=7024520.

Cheers,

D

I saw that TID too when I was first troubleshooting this. Tried some variations of that but it made no difference unfortunately.

Did you symlink the libraries in /opt/novell/ssllib?

Just tested on a RHEL 7.9 box that hasn't been updated and no, it only loads the libssl and libcrypto from /opt/novell/ssllibs.  As a matter a fact, I see much fewer entries with libcrypto (only two I see in apache.tr), nothing like on the RHEL 8.8 box.

Tried that, no difference.