prasenjitmass Respected Contributor.
Respected Contributor.
902 views

PAM service stopped all of sudden

Hi ,
We are using PAM 3.2 (OS RedHat7.3). We have approx 350 agent servers including Windows, Linux and DB .RDP Relay, SSH Relay and DB monitoring Rule is configured.Yesterday We tried to access a MSSQL DB through PAM and it was not connecting. We tried to remove and add the DB connector for that DB server. It took little more time to save . During that period we noticed that npum service of PAM server got stopped ad tried to restart the service , message arrived " npum service not running" we tried for 2 more time to restart that. Then we restarted the PAM server. After restarting it was working well.

Why this sudden service failure occured did not found reason. searched the log. following is the details.Can anyone tell me what was the reason?

Thu Oct 25 12:26:44 2018, 490, 793008000, 19809, Error, Failed to open server sockets
Thu Oct 25 12:26:44 2018, 490, 793008000, 19809, Info, Server process exiting rc(98)
Thu Oct 25 12:26:44 2018, 490, 793008000, 19809, Error, main line: 669 rv=98:Address already in use

Thu Oct 25 12:27:25 2018, 844, 1652987776, 20526, Error, Failed to open server sockets
Thu Oct 25 12:27:25 2018, 844, 1652987776, 20526, Info, Server process exiting rc(98)
Thu Oct 25 12:27:25 2018, 844, 1652987776, 20526, Error, main line: 669 rv=98:Address already in use


Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Info, ========================================
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Info, Version 3.2.0-5 (Rev:27321,Bld:220) [linux-3.10-x86_64]
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Info, Database Version 3.7.15.2
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:29120: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:443: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3001: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3002: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3003: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3004: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3005: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3006: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3007: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3008: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3009: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3010: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3011: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3012: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3013: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3014: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3015: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3016: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3017: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3018: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3019: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3020: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3100: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3101: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3102: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3103: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:3104: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:8889: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:13389: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Unable to open connector 0.0.0.0:2222: Address already in use
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, Failed to open server sockets
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Info, Server process exiting rc(98)
Thu Oct 25 12:28:16 2018, 945, 358885248, 21381, Error, main line: 669 rv=98:Address already in use
Thu Oct 25 12:30:15 2018, 735, 1307924352, 3742, Info, ========================================
Thu Oct 25 12:30:15 2018, 736, 1307924352, 3742, Info, Version 3.2.0-5 (Rev:27321,Bld:220) [linux-3.10-x86_64]
Thu Oct 25 12:30:15 2018, 736, 1307924352, 3742, Info, Database Version 3.7.15.2
Thu Oct 25 12:30:15 2018, 773, 1307924352, 3903, Info, [admin 3.2.0-5 (Rev:27352,Bld:220) ] module loaded
Thu Oct 25 12:30:15 2018, 956, 1307924352, 3903, Info, [audit 3.2.0-4 (Rev:26918,Bld:199) ] module loaded
Thu Oct 25 12:30:16 2018, 36, 1307924352, 3903, Info, [auth 3.2.0 (Rev:26324,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 106, 1307924352, 3903, Info, [cmdctrl 3.2.0-4 (Rev:26930,Bld:200) ] module loaded
Thu Oct 25 12:30:16 2018, 129, 1307924352, 3903, Info, [dbaudit 3.2.0-2 (Rev:26570,Bld:180) ] module loaded
Thu Oct 25 12:30:16 2018, 161, 1307924352, 3903, Info, [distrib 3.2.0-4 (Rev:26864,Bld:192) ] module loaded
Thu Oct 25 12:30:16 2018, 180, 1307924352, 3903, Info, [ldapagnt 3.2.0 (Rev:24835,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 237, 1307924352, 3903, Info, [msgagnt 3.2.0 (Rev:24742,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 281, 1307924352, 3903, Info, [pkgman 3.2.0 (Rev:26252,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 314, 1307924352, 3903, Info, [prvcrdvlt 3.2.0-1 (Rev:26380,Bld:176) ] module loaded
Thu Oct 25 12:30:16 2018, 341, 1307924352, 3903, Info, [rdprelay 3.2.0 (Rev:26194,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 408, 1307924352, 3903, Info, [regclnt 3.2.0 (Rev:26139,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 489, 1307924352, 3903, Info, [registry 3.2.0-1 (Rev:26409,Bld:176) ] module loaded
Thu Oct 25 12:30:16 2018, 513, 1307924352, 3903, Info, [resreqagnt 3.2.0 (Rev:25675,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 565, 1307924352, 3903, Info, [rexec 3.2.0-5 (Rev:27372,Bld:220) ] module loaded
Thu Oct 25 12:30:16 2018, 577, 1307924352, 3903, Info, [secaudit 3.2.0 (Rev:26171,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 614, 1307924352, 3903, Info, [sshagnt 3.2.0-5 (Rev:27307,Bld:220) ] module loaded
Thu Oct 25 12:30:16 2018, 616, 1307924352, 3903, Info, [sshrelay 3.2.0-5 (Rev:27376,Bld:220) ] module loaded
Thu Oct 25 12:30:16 2018, 619, 1307924352, 3903, Info, [strfwd 3.2.0 (Rev:24407,Bld:158) ] module loaded
Thu Oct 25 12:30:16 2018, 620, 1307924352, 3903, Info, [syslogemit 3.2.0-4 (Rev:26918,Bld:199) ] module loaded
Thu Oct 25 12:30:16 2018, 666, 1307924352, 3903, Info, Created default audit video path = '/opt/netiq/npum/service/local/audit/video/capture'
Thu Oct 25 12:30:16 2018, 738, 1307924352, 3903, Info, COLUMN audit group already exits. No need to populate it as it would have been populated in previous run.
Thu Oct 25 12:30:16 2018, 738, 1307924352, 3903, Info, Attempt to create IDX_CMD_GRP INDEX
Thu Oct 25 12:30:16 2018, 738, 1307924352, 3903, Info, Create IDX_CMD_GRP INDEX operation completed.
Thu Oct 25 12:30:16 2018, 826, 1307924352, 3903, Info, Creating RDP 512 certificate
Thu Oct 25 12:30:16 2018, 836, 1307924352, 3903, Info, Creating RDP 2048 certificate
Thu Oct 25 12:30:16 2018, 987, 1307924352, 3903, Info, Process ID and listening port of XQD are 4050 and 34198.
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:29120
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:443
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3001
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3002
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3003
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3004
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3005
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3006
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3007
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3008
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3009
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3010
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3011
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3012
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3013
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3014
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3015
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3016
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3017
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3018
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3019
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3020
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3100
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3101
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3102
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3103
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:3104
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:8889
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:13389
Thu Oct 25 12:30:16 2018, 998, 1307924352, 3903, Info, Service listening on 0.0.0.0:2222
Thu Oct 25 12:30:16 2018, 999, 1278973696, 3903, Info, SQL Browser Service is started successfully
Thu Oct 25 12:30:17 2018, 0, 1275815680, 3903, Info, Checking service registration for prodidamapp04.finopaymentbank.in (Novell Privileged User Manager)
0 Likes
1 Reply
Micro Focus Expert
Micro Focus Expert

Re: PAM service stopped all of sudden

This leads me to believe that there must have been at least a couple of PAM services attempting to run at the same time.
This looks like these ports were already bound to by another running application (likely another PAM instance running).
A restart of the server should resolve.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.