bmc Frequent Contributor.
Frequent Contributor.
1744 views

Environment check issue post patch installation on destination Linux server

Jump to solution

Hello,

I have created 4 environments more than a month back to connect (SSH2 and Secure Copy 2) to Linux based systems. Environments were working fine untill the patch is installed on destination Linux application server.

My PPM application is hosted on windows server 2008. Before patch installation when i did telnet (telnet servername 22) from my PPM windows server to destination Linux box, it was showing "SSH-2.0-OpenSSH_6.6.1". After patch installation on linux box, SSH version changed to "SSH-2.0-OpenSSH_7.4".

Environment checks are failing post patch installation on Linux servers. Below is the environment check error message.

Any help on this will be much appreciated. Thanks.

**********************************************************************************************************************

An error occured while executing the Environment Check commands. This may be due to incorrect or unsupported Environment settings; please review the log and make necessary adjustments. (KNTA-11064) Execution Log

KSC Connect

Source Command:Env Check: Shell connection to Env server

Protocol: com.kintana.core.net.SSH2Client
Host: Linux_servername.com
Username: username
Stream Encoding: UTF-8
[2017/09/11 23:43:09 -0500] LOGON_ATTEMPT username@Linux_servername.com:22
SSH version received from remote host: SSH-2.0-OpenSSH_7.4
SSH version sent to remote host: SSH-2.0-1.0 Kintana SSH client
Initiating key exchange.
Succeeded in key exchange.
Authenticating user.
Succeeded in authenticating user.
Opening channel.
ERROR: Could not logon to [Linux_servername.com].
open failure

**********************************************************************************************************************

Please note, we have integration with Oracle app as well. Oracle app environment checks are still working fine.

Regards,

Manas

Tags (1)
0 Likes
1 Solution

Accepted Solutions
Outstanding Contributor.. Amishra Outstanding Contributor..
Outstanding Contributor..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Thanks Manas, 

Couple of thoughts and options.

1. Did you check with HPE if OpenSSH_7.4 is supported on PPM 9.2x (I couldnt find anything in the system compatibility matrix)?

2. I am still not convinced that you have to generate a private public key on PPM and distribute it to the destination server (given it was working on destination server prior to patching)

3. After you made changes to the sshd_config file did you restart the service? 

4. Did you try and update the following parameters in the sshd_config file? (suggested in my previous comment)

You should edit sshd_config file at your host like that
TCPKeepAlive yes
PermitOpen [host_ip]:[port]
AllowTcpForwarding yes
PermitTunnel yes

ref: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open

Please take a backup of the sshd_config file before you make any changes.

5. From PPM server do a manual ssh to the destination server ssh weblogic@DEST_LINUX_SERVER; see if you are able to connect?

6. From Desitnation server try to do a manual ssh i.e. ssh weblogic@DEST_LINUX_SERVER; see if that helps.

7. On the working server go to the home folder <Home_Directory>/.ssh ; find authorized_keys (if any)  see if there is any reference to PPM account.  You are highly unlikely to find it because you are using password based authentication. 

8. Finally last thing, have you thought of downgrading OpenSSH from SSH-2.0-OpenSSH_7.4 to SSH-2.0-OpenSSH_6.6.1 (working version)

Please note if you are making changes to sshd_config file, please bounce the services for the changes to take affect.

Good Luck!

Let me know how this goes. 

 

Regards,

Ajay

 

Regards,
Ajay Mishra
10 Replies
Outstanding Contributor.. Amishra Outstanding Contributor..
Outstanding Contributor..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Good Morning Manas, 

Looking at the error logs, seems the authentication has went through; error happens while opening channel.

Can you try to do a manual ssh on the destination linux server? e.g. ssh localhost; to see what happens. 

Also is it possible to revert to the previous ssh version on the linux server?

 

Regards,

Ajay

Regards,
Ajay Mishra
0 Likes
Established Member.. manaschandra
Established Member..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Thanks for your reply Ajay.

Here is the "ssh localhost" result from destination linux server. As i did "ssh localhost" for the first time, system seems to be key in the knwon_hosts file.

Environment check still fails with the same error message, even after key is added to known_hosts. Any suggestions will be appreciated.  

"**************************************************************************************************************************

[09:38:04] weblogic@Dest_Linux_Server:~>
cd .ssh/
[09:38:34] weblogic@Dest_Linux_Server:~/.ssh>
ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:3wYMD1lheGaxmv4TjgF7UGzvp8VP8GPGBoptoTdk8g.
ECDSA key fingerprint is MD5:50:a7:92:8c:63:c9:73:0c:cb:69:ad:2a:12:7b:d8:86.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.

#########################################################
# This computer system is private property. Use of this #
# system is restricted to authorized users only and #
# shall be limited to activities permitted under #
# applicable law. In addition, users must comply with #
# the owner's acceptable use and other applicable #
# policies. Unauthorized access, use, or modification #
# of this system is strictly prohibited and may result #
# in criminal prosecution, civil action, or employee #
# discipline. Users of this system should have no #
# expectation of privacy irrespective of any security #
# measures imposed by the owner of this system since #
# such measures are solely for the benefit of the #
# owner. Activities on this system may be monitored, #
# recorded, and subject to audit. Use of this system, #
# authorized or unauthorized, constitutes consent to #
# such monitoring and recording. #
#########################################################

weblogic@localhost's password:
Last login: Wed Sep 13 09:38:03 2017 from 10.132.10.66

-------------------------------------------------------------------
Defaulting environment to DEV RMS:/app01/wls1036forms
Use: ". setwlenv.sh" to change
-------------------------------------------------------------------

[09:39:34] weblogic@Dest_Linux_Server:~>

**************************************************************************************************************************

 

Regards,

Manas

0 Likes
Outstanding Contributor.. Amishra Outstanding Contributor..
Outstanding Contributor..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Hello Manas,

Looks like your ssh localhost connnectivity on the destination server was successful [after you added it to the known_hosts file]

Seems you had to enter password from keyboard for the authentication to go through when connecting via ssh localhost. 

Other thing to check would be: On the destination server check sshd.config file to see if PrivatePublicKeyAuthentication is enabled? If not  then turn it on and restart sshd service on the destination server.

After you have done this, try the environment checks again. 

 

Regards,

Ajay

 

 

Regards,
Ajay Mishra
0 Likes
bmc Frequent Contributor.
Frequent Contributor.

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Hi Ajay,

If we enable PrivatePublicKeyAuthentication on the destination server, i think we need to enable ssh on PPM server too. Right now, i do not see ssh enabled on PPM server.

As per the HP support document, if configured private Key Authentication with Secure Shell then there should be id_rsa.pub file under <PPM_Home> directory, which I could not see.

As mentioned in the issue, only few environments are not working which has openSSH version 7.4 on the destination server.

Other environments where we have OpenSSH 6.6.1 or lower are working fine.

We are using HP PPM 9.2x version.

Regards,

Manas 

0 Likes
Outstanding Contributor.. Amishra Outstanding Contributor..
Outstanding Contributor..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Hi Manas,

Yes, you are correct. Apologies. I did some further reading on internet it seems the connection from the destination server is being rejected. Could you do a comparision of the sshd_config files from the server where ssh is working (OpenSSH 6.6.1) and servers where ssh is not (openSSH version 7.4).

Try to find discrepancies between the two and update sshd_config files

***From the internet***

You should edit sshd_config file at your host like that
TCPKeepAlive yes
PermitOpen [host_ip]:[port]
AllowTcpForwarding yes
PermitTunnel yes

ref: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open

Please take a backup of the sshd_config file before you make any changes.

 

Regards,

Ajay 

Regards,
Ajay Mishra
0 Likes
bmc Frequent Contributor.
Frequent Contributor.

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Thanks for your reply Ajay.

I have requested my Linux administrator to compare the sshd_config files on both the servers. I will update you the status soon.

0 Likes
bmc Frequent Contributor.
Frequent Contributor.

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Hi Ajay,

We did compare the sshd_config files on both working and not working servers. We do find some descripancies which we have corrected but still environment check is not passing to the Linux server where ssh version is OpenSSH_7.4. Our Unix administrator changed server configurations to debug mode so that when trying to connect to the destination servers from PPM, we could see the logs. 

Below are the logs.

 

******************************Connection NOT working from PPM: SSH version is OpenSSH_7.4******************************
274_Server:root# /sbin/sshd -d
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:y+/HmbAojKQ3fHarJJ/B7kycan8iMhyYQMIBRJ+GTkw
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:3wYMD1lheGaWUmN4TjgsjuGzvp8VP8GPGBoptoTdk8g
debug1: private host key #2: ssh-ed25519 SHA256:K+zPTKHboqv+Z3NyuujshyVCF3BxTF1jqfzYkJludEM
debug1: rexec_argv[0]='/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from PPM_Server port 62140 on MOMTEST_Server port 22
debug1: Client protocol version 2.0; client software version 1.0 Kintana SSH client
debug1: no match: 1.0 Kintana SSH client
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug1: SELinux support disabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: diffie-hellman-group1-sha1 [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: 3des-cbc MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: 3des-cbc MAC: hmac-sha1 compression: none [preauth]
debug1: kex: diffie-hellman-group1-sha1 need=24 dh_need=20 [preauth]
debug1: kex: diffie-hellman-group1-sha1 need=24 dh_need=20 [preauth]
debug1: expecting SSH2_MSG_KEXDH_INIT [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user weblogic service ssh-connection method password [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "weblogic"
debug1: PAM: setting PAM_RHOST to "PPM_Server"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth_send_banner: sent [preauth]
debug1: PAM: password authentication accepted for weblogic
debug1: do_pam_account: called
Accepted password for weblogic from PPM_Server port 62140 ssh2
debug1: monitor_child_preauth: weblogic has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: temporarily_use_uid: 1001/1003 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
debug1: SELinux support disabled
debug1: PAM: establishing credentials
User child is on pid 5343
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1001/1003
debug1: rekey after 134217728 blocks
debug1: rekey after 134217728 blocks
debug1: ssh_packet_set_postauth: called
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch
debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
************************************************************END************************************************************

Above highlighted changes we found in the logs for the connections which is failing from PPM. If you observe and compare the bolow connection working log from PPM, in the non working environment "private host key" is generated. As mentioned before. PPM server is windows based and SSH is not installed / i do not see id_rsa.pub file in <PPM_Home> directory. I assume if generating "private host key" in destination enviroment, then we need to add the public / private key in the authorized_keys file in PPM. Let me know if i am wrong?

If my understanding is correct, then as there is no SSH enabled on PPM server, what steps i need to take to resolve the issue? Any clues will be much appreciated.

 

******************************Connection working from PPM: SSH version is OpenSSH_6.6.1******************************
270_Server:root# /sbin/sshd -d
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #1 type 3 ECDSA
debug1: private host key: #2 type 4 ED25519
debug1: rexec_argv[0]='/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from PPM_Server port 62232 on MOM_Server port 22
debug1: Client protocol version 2.0; client software version 1.0 Kintana SSH client
debug1: no match: 1.0 Kintana SSH client
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: SELinux support disabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server arcfour hmac-md5 none [preauth]
debug1: kex: server->client arcfour hmac-md5 none [preauth]
debug1: kex: diffie-hellman-group1-sha1 need=16 dh_need=16 [preauth]
debug1: kex: diffie-hellman-group1-sha1 need=16 dh_need=16 [preauth]
debug1: expecting SSH2_MSG_KEXDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user weblogic service ssh-connection method password [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "weblogic"
debug1: PAM: setting PAM_RHOST to "PPM_Server"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth_send_banner: sent [preauth]
debug1: PAM: password authentication accepted for weblogic
debug1: do_pam_account: called
Accepted password for weblogic from PPM_Server port 62232 ssh2
debug1: monitor_child_preauth: weblogic has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: temporarily_use_uid: 1001/1003 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
debug1: SELinux support disabled
debug1: PAM: establishing credentials
User child is on pid 29964
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1001/1003
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: SELinux support disabled
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on pts/1 for weblogic from PPM_Server port 62232
debug1: Setting controlling tty using TIOCSCTTY.
Connection closed by PPM_Server
debug1: channel 0: free: server-session, nchannels 1
debug1: session_close: session 0 pid 29965
debug1: do_cleanup
debug1: session_by_tty: session 0 tty /dev/pts/1
debug1: session_pty_cleanup: session 0 release /dev/pts/1
Transferred: sent 6856, received 5664 bytes
Closing connection to PPM_Server port 62232
************************************************************END************************************************************

Regards,

Manas

0 Likes
Outstanding Contributor.. Amishra Outstanding Contributor..
Outstanding Contributor..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Thanks Manas, 

Couple of thoughts and options.

1. Did you check with HPE if OpenSSH_7.4 is supported on PPM 9.2x (I couldnt find anything in the system compatibility matrix)?

2. I am still not convinced that you have to generate a private public key on PPM and distribute it to the destination server (given it was working on destination server prior to patching)

3. After you made changes to the sshd_config file did you restart the service? 

4. Did you try and update the following parameters in the sshd_config file? (suggested in my previous comment)

You should edit sshd_config file at your host like that
TCPKeepAlive yes
PermitOpen [host_ip]:[port]
AllowTcpForwarding yes
PermitTunnel yes

ref: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open

Please take a backup of the sshd_config file before you make any changes.

5. From PPM server do a manual ssh to the destination server ssh weblogic@DEST_LINUX_SERVER; see if you are able to connect?

6. From Desitnation server try to do a manual ssh i.e. ssh weblogic@DEST_LINUX_SERVER; see if that helps.

7. On the working server go to the home folder <Home_Directory>/.ssh ; find authorized_keys (if any)  see if there is any reference to PPM account.  You are highly unlikely to find it because you are using password based authentication. 

8. Finally last thing, have you thought of downgrading OpenSSH from SSH-2.0-OpenSSH_7.4 to SSH-2.0-OpenSSH_6.6.1 (working version)

Please note if you are making changes to sshd_config file, please bounce the services for the changes to take affect.

Good Luck!

Let me know how this goes. 

 

Regards,

Ajay

 

Regards,
Ajay Mishra
bmc Frequent Contributor.
Frequent Contributor.

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Thanks Ajay for your assistance.

Issue resolved by downgrading SSH version as SSH7.4 is not compatible with PPM9.22.

 

Regards,

Manas

0 Likes
Outstanding Contributor.. Amishra Outstanding Contributor..
Outstanding Contributor..

Re: Environment check issue post patch installation on destination Linux server

Jump to solution

Good to know Manas - Well done! 

Can you mark the solution as solved? 

  

 

Regards,
Ajay Mishra
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.