TE Super Contributor.
Super Contributor.
231 views

Cannot send email from template

I am chasing down an issue with do-send-email-from-template. This is something that was working, post upgrade, but now has quit.

The error I get is this:

DirXML Log Event -------------------
Driver: \TREENAME\Services\DriverSetName\Company_AD_Driver_Stage
Channel: Subscriber
Object: \TREENAME\CompanyName\TestUser002
Status: Error
Message: Code(-9195) Error in vnd.nds.stream://TREENAME/Services/DriverSetName/Company_AD_Driver_Stage/itp-SendNewUserInfo#XmlData:336 : Couldn't send email: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[05/24/19 09:22:29.576]:MF_AD_STAGE ST: Query from policy
[05/24/19 09:22:29.576]:MF_AD_STAGE ST:

I found a related (somewhat) TID (forgot the number) that said to add the smtp server's cert to the cacert keystore. Did that. The Cert signers were already in the keystore. No joy.

I started checking access using nc, dig, host, and nslookup, just to verify settings and found an interesting problem. It seems this server has a mismatch in where it is getting it's library files.
This is SLES 12 SP4. eDir 9.1.3, IDM is 4.7.2, AD Driver is 4.1.1.0.

This environment has 3 virtually identical servers. I tried doing the send-email on another server, and it works fine. Digging deeper, I used the linux ldd command to trace some libraries, and found this.

Server 1 is the working server, Server 2 is the one that has the email problem. Server 1 did NOT get the smtp server cert added to cacerts, yet works fine.

Using ldd on openssl, here are the differences...

server1:~ # ldd -v /usr/bin/openssl
linux-vdso.so.1 (0x00007ffdd7f36000)
libssl.so.1.0.0 => /lib64/libssl.so.1.0.0 (0x00007f48031de000)
libcrypto.so.1.0.0 => /lib64/libcrypto.so.1.0.0 (0x00007f4802d76000)
libc.so.6 => /lib64/libc.so.6 (0x00007f48029d1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f48027cd000)
libz.so.1 => /lib64/libz.so.1 (0x00007f48025b6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f480344c000)

Version information:
/usr/bin/openssl:
libc.so.6 (GLIBC_2.15) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libssl.so.1.0.0 (OPENSSL_1.0.0) => /lib64/libssl.so.1.0.0
libcrypto.so.1.0.0 (OPENSSL_1.0.0) => /lib64/libcrypto.so.1.0.0
/lib64/libssl.so.1.0.0:
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libcrypto.so.1.0.0 (OPENSSL_1.0.0) => /lib64/libcrypto.so.1.0.0
/lib64/libcrypto.so.1.0.0:
libdl.so.2 (GLIBC_2.2.5) => /lib64/libdl.so.2
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
/lib64/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
/lib64/libdl.so.2:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/lib64/libz.so.1:
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6

server2:~ # ldd -v /usr/bin/openssl
/usr/bin/openssl: /opt/novell/lib64/libssl.so.1.0.0: no version information available (required by /usr/bin/openssl)
/usr/bin/openssl: /opt/novell/lib64/libcrypto.so.1.0.0: no version information available (required by /usr/bin/openssl)
linux-vdso.so.1 (0x00007ffd638df000)
libssl.so.1.0.0 => /opt/novell/lib64/libssl.so.1.0.0 (0x00007f6486dcb000)
libcrypto.so.1.0.0 => /opt/novell/lib64/libcrypto.so.1.0.0 (0x00007f64869bd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6486618000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6486414000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6486d2e000)

Version information:
/usr/bin/openssl:
libc.so.6 (GLIBC_2.15) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libssl.so.1.0.0 (OPENSSL_1.0.0) => not found
libcrypto.so.1.0.0 (OPENSSL_1.0.0) => not found
/opt/novell/lib64/libssl.so.1.0.0:
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/opt/novell/lib64/libcrypto.so.1.0.0:
libdl.so.2 (GLIBC_2.2.5) => /lib64/libdl.so.2
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/lib64/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
/lib64/libdl.so.2:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6

There are missing libraries, and for Server2, it seems to be getting some libraries from /opt/novell/lib64, and others from /lib64.

/lib64 has newer (by date/time stamp) files than /opt/novell/lib64

There were similar results using ldd on dig, host, and nslookup. It appears that something may have updated/patched Server2 via SUSE, and changed something, but what?

An echo $LD_LIBRARY_PATH shows nothing on both, and their $PATH statements are identical. Both servers have iManager 3.0.3 installed, but Server2 has started throwing this: Unable to create AdminNamespace. java.lang.NoClassDefFoundError: novell/jclient/JCException where it worked fine previously.

This is our testing environment, and we need this answered before we move Production to IDM 4.7.2. Prod runs on SLES 11 SP3, and eDir 8.8.8.8, so this is a multi-step upgrade. Not very easy to roll back.

Thoughts or ideas?
Labels (1)
0 Likes
2 Replies
Knowledge Partner
Knowledge Partner

Re: Cannot send email from template

Do you have proper CA/cert in your keystore? I believe that your issue related to certificate required by your mail server
0 Likes
Highlighted
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Cannot send email from template

On 05/24/2019 09:56 AM, tse7147 wrote:
>
> I am chasing down an issue with do-send-email-from-template. This is
> something that was working, post upgrade, but now has quit.


It's strange that it would break after it worked after the upgrade.
Usually this breaks with 4.7, or one if its SPs, because how the system
defaults to try using TLS whenever possible, and until now it never did
TLS at all, so certificates for the mail server are missing and things break.

> The error I get is this:
>
> DirXML Log Event -------------------
> Driver: \TREENAME\Services\DriverSetName\Company_AD_Driver_Stage
> Channel: Subscriber
> Object: \TREENAME\CompanyName\TestUser002
> Status: Error
> Message: Code(-9195) Error in
> vnd.nds.stream://TREENAME/Services/DriverSetName/Company_AD_Driver_Stage/itp-SendNewUserInfo#XmlData:336
> : Couldn't send email: javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find valid certification path to requested target
> [05/24/19 09:22:29.576]:MF_AD_STAGE ST: Query from policy
> [05/24/19 09:22:29.576]:MF_AD_STAGE ST:


This error means what you think it mans: the truststore does not have the
CA trusted. I wonder if it would show up if there were other certificate
validation issues, like if you pointed to mail.company.tld but the
certificate actually has relay.company.tld in its subject or Subject
Alternative Names (SAN) fields. It may be worth verifying that the e-mail
server's certificate is completely valid per all the possible checks a
client will do. openssl is pretty good for this, especially with the SMTP
starttls extension..

> I found a related (somewhat) TID (forgot the number) that said to add
> the smtp server's cert to the cacert keystore. Did that. The Cert
> signers were already in the keystore. No joy.


Maybe try adding the endpoint certificate itself. You should not need to
do that, of course, but it might be worth trying as a troubleshooting
step, or even a workaround.

> I started checking access using nc, dig, host, and nslookup, just to
> verify settings and found an interesting problem. It seems this server
> has a mismatch in where it is getting it's library files.
> This is SLES 12 SP4. eDir 9.1.3, IDM is 4.7.2, AD Driver is 4.1.1.0.


Look under /etc/ld.so.conf.d/ and see if you have an ntls.conf or
something else mentioning 'novell' or 'netiq' within. If so, delete it,
run '/sbin/ldconfig' as 'root', and bounce your system.

Also keep in mind that 'ldd' uses your current (user's) environment
variables to do its looking up magic, meaning if your current user has a
different LD_LIBRARY_PATH or LD_PRELOAD environment set (or maybe more
likely, unset) than the running eDirectory process, things will be
different. You can see a running process's environment by using something
like this (for the 'ndsd' process, assuming you only have one as is normal):


strings /proc/$(pgrep ndsd)/environ


> This environment has 3 virtually identical servers. I tried doing the
> send-email on another server, and it works fine. Digging deeper, I used
> the linux ldd command to trace some libraries, and found this.
>
> Server 1 is the working server, Server 2 is the one that has the email
> problem. Server 1 did NOT get the smtp server cert added to cacerts, yet
> works fine.
>
> Using ldd on openssl, here are the differences...
>
> server1:~ # ldd -v /usr/bin/openssl
> linux-vdso.so.1 (0x00007ffdd7f36000)
> libssl.so.1.0.0 => /lib64/libssl.so.1.0.0 (0x00007f48031de000)
> libcrypto.so.1.0.0 => /lib64/libcrypto.so.1.0.0
> (0x00007f4802d76000)
> libc.so.6 => /lib64/libc.so.6 (0x00007f48029d1000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00007f48027cd000)
> libz.so.1 => /lib64/libz.so.1 (0x00007f48025b6000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f480344c000)
>
> Version information:
> /usr/bin/openssl:
> libc.so.6 (GLIBC_2.15) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> libssl.so.1.0.0 (OPENSSL_1.0.0) =>
> /lib64/libssl.so.1.0.0
> libcrypto.so.1.0.0 (OPENSSL_1.0.0) =>
> /lib64/libcrypto.so.1.0.0
> /lib64/libssl.so.1.0.0:
> libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> libcrypto.so.1.0.0 (OPENSSL_1.0.0) =>
> /lib64/libcrypto.so.1.0.0
> /lib64/libcrypto.so.1.0.0:
> libdl.so.2 (GLIBC_2.2.5) => /lib64/libdl.so.2
> libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
> /lib64/libc.so.6:
> ld-linux-x86-64.so.2 (GLIBC_2.3) =>
> /lib64/ld-linux-x86-64.so.2
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) =>
> /lib64/ld-linux-x86-64.so.2
> /lib64/libdl.so.2:
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) =>
> /lib64/ld-linux-x86-64.so.2
> libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> /lib64/libz.so.1:
> libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6


Look for that conf file under /etc/ld.so.conf.d/ on this box as I believe
you may have upgraded and leaving that file behind, causing issues with
all kinds of things other than eDirectory (e.g. zpyper), is an issue I
reported against eDirectory 9.x.

> server2:~ # ldd -v /usr/bin/openssl
> /usr/bin/openssl: /opt/novell/lib64/libssl.so.1.0.0: no version
> information available (required by /usr/bin/openssl)
> /usr/bin/openssl: /opt/novell/lib64/libcrypto.so.1.0.0: no version
> information available (required by /usr/bin/openssl)
> linux-vdso.so.1 (0x00007ffd638df000)
> libssl.so.1.0.0 => /opt/novell/lib64/libssl.so.1.0.0
> (0x00007f6486dcb000)
> libcrypto.so.1.0.0 => /opt/novell/lib64/libcrypto.so.1.0.0
> (0x00007f64869bd000)
> libc.so.6 => /lib64/libc.so.6 (0x00007f6486618000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00007f6486414000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f6486d2e000)
>
> Version information:
> /usr/bin/openssl:
> libc.so.6 (GLIBC_2.15) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> libssl.so.1.0.0 (OPENSSL_1.0.0) => not found
> libcrypto.so.1.0.0 (OPENSSL_1.0.0) => not found
> /opt/novell/lib64/libssl.so.1.0.0:
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> /opt/novell/lib64/libcrypto.so.1.0.0:
> libdl.so.2 (GLIBC_2.2.5) => /lib64/libdl.so.2
> libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
> /lib64/libc.so.6:
> ld-linux-x86-64.so.2 (GLIBC_2.3) =>
> /lib64/ld-linux-x86-64.so.2
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) =>
> /lib64/ld-linux-x86-64.so.2
> /lib64/libdl.so.2:
> ld-linux-x86-64.so.2 (GLIBC_PRIVATE) =>
> /lib64/ld-linux-x86-64.so.2
> libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
> libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
>
> There are missing libraries, and for Server2, it seems to be getting
> some libraries from /opt/novell/lib64, and others from /lib64.
>
> /lib64 has newer (by date/time stamp) files than /opt/novell/lib64


Timestamps do not matter directly, of course, but using the wrong
libraries is perhaps the issue. See the ntls.conf comments from above.

> There were similar results using ldd on dig, host, and nslookup. It
> appears that something may have updated/patched Server2 via SUSE, and
> changed something, but what?


Yes, because of that include file in /etc/ld.so.conf.d/ I presume, as that
impacts the whole system. Be sure you run ldconfig, as root, anytime you
change anything in this directory, or /etc/ld.so.conf itself (which you
should likely never do).

> An echo $LD_LIBRARY_PATH shows nothing on both, and their $PATH
> statements are identical. Both servers have iManager 3.0.3 installed,
> but Server2 has started throwing this: Unable to create AdminNamespace.
> java.lang.NoClassDefFoundError: novell/jclient/JCException where it
> worked fine previously.


PATH should not matter as we are dealing with libraries, which is the
domain of LD_* variables, but there are automatic libraries set too, and
that's what /etc/ld.so.conf (and friends) control.

> This is our testing environment, and we need this answered before we
> move Production to IDM 4.7.2. Prod runs on SLES 11 SP3, and eDir
> 8.8.8.8, so this is a multi-step upgrade. Not very easy to roll back.


If you follow the documentation then yes, rolling back is a pain. If you
are ever interested, I've written a few times about another way of doing
your upgrade/migration by moving the NICI and eDirectory information from
one box to another. One side effect of doing that is that rolling back is
pretty easy, since you are moving from one (old) system to a new one, so
if new is broken, you can turn old back on.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.