Post Installation of OBA 3.04 Failing with PostgreSQL connection refused error

[Sanchari Gupta]

HI All,

Need your expertise to determine solution of below issue I am facing while running post installation of opsa analytics for OBA 3.04.

Issue: After installation step, I am running the post installation script  ./opsa-server-postinstall.sh as opsa user on the analytics machine. As soon as I provide inputs for the PostgreSQL configurations, it fails with the below error:

Also attaching relevant logs for your reference.

[opsa@oba-server bin]$ opsa-server-postinstall.sh
****************************************************************

OBA server post installation configuration

****************************************************************

PostgreSQL configuration database configuration details:

Enter PostgreSQL database hostname [localhost]: oba-server.demo.local
Enter PostgreSQL database port number [5434]: 5434
Enter new PostgreSQL database administrator (postgres) password:
Re-enter password:
Valid password entered
Enter new PostgreSQL OBA user password:
password must contain a least 13 characters
password must contain a digit
password must contain lower case character
password must contain upper case character
Enter new PostgreSQL OBA user password:
Re-enter password:
Valid password entered
Checking configuration database connection with the given credentials...
Jan 23, 2020 8:41:12 AM org.postgresql.core.v3.ConnectionFactoryImpl log
WARNING: ConnectException occurred while connecting to oba-server.demo.local:5434
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:69)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:158)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.getConnectionFromDM(PgPoolingDataSource.java:153)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.canConnect(PgPoolingDataSource.java:211)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.canConnect(PgPoolingDataSource.java:190)
at com.hp.opsa.install.postinstall.PostInstallConfigReader.readConfigParametersFromCLIInteractively(PostInstallConfigReader.java:148)
at com.hp.opsa.install.postinstall.PostInstallConfigReader.getConfigurationParameters(PostInstallConfigReader.java:57)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.getConfigurationParameters(PostInstallConfiguration.java:536)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.start(PostInstallConfiguration.java:508)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.main(PostInstallConfiguration.java:544)

Jan 23, 2020 8:41:13 AM org.postgresql.Driver connect
SEVERE: Connection error:
org.postgresql.util.PSQLException: Connection to oba-server.demo.local:5434 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:247)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.getConnectionFromDM(PgPoolingDataSource.java:153)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.canConnect(PgPoolingDataSource.java:211)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.canConnect(PgPoolingDataSource.java:190)
at com.hp.opsa.install.postinstall.PostInstallConfigReader.readConfigParametersFromCLIInteractively(PostInstallConfigReader.java:148)
at com.hp.opsa.install.postinstall.PostInstallConfigReader.getConfigurationParameters(PostInstallConfigReader.java:57)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.getConfigurationParameters(PostInstallConfiguration.java:536)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.start(PostInstallConfiguration.java:508)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.main(PostInstallConfiguration.java:544)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:69)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:158)
... 14 more

Error: Connection to PostgreSQL configuration database failed. Please input the details again

Few observations:

1.      The status of opsa-postgresql is “stopped”.

      [opsa@oba-server bin]$ opsa status
Operations Analytics - Process Manager is not running.
OpsA Server status:
opsa-postgresql is stopped
opsa-server is stopped
opsa-storm-nimbus is stopped
opsa-task-manager is stopped
opsa-zookeeper is stopped

2.      Port availability and firewall checks have already been done, seems to be no issue on that end

3.      Found similar issues recorded on SW Support but as they are marked internal, couldn’t check further details.

https://softwaresupport.softwaregrp.com/doc/SD02252309

4.      Do I need to add/modify anything on var/opt/OV/databases/opsaconf/pg_hba.conf? As I don’t see any mention of such on the official documentation.

Any inputs will be really helpful.

Thanks,

Sanchari Gupta

[Priyadarshini B]

Hi Sanchari,
Did you try to address as per the error line from the log:
Connection to oba-server.demo.local:5434 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Cheers,
Priya

[Sanchari Gupta]

Hi Priya,

Seeing the error initially, I thought it might be a port issue. So we got it checked by the VM team, seems everything okay from port and firewall side.

Then I got one error line in pgsql log which says below:

020-01-23 10:14:33.987 UTC @:[16661]: LOG: specifying both host name and CIDR mask is invalid: "/32"
2020-01-23 10:14:33.987 UTC @:[16661]: CONTEXT: line 91 of configuration file "/var/opt/OV/databases/opsaconf/pg_hba.conf"
2020-01-23 10:14:33.987 UTC @:[16661]: FATAL: could not load pg_hba.conf
2020-01-23 10:14:33.988 UTC @:[16661]: LOG: database system is shut down

I checked the pg_hba.conf file and found the line which says:

host all all /32 md5

I tried modifying the line by adding IP address of opsa analytics server:

host all all 172.16.239.***/32 md5

and tried to start the PostgreSQL service by command: systemctl start opsa-postgresql

Now atleast I can see the PostgreSQL service running on "opsa status":

[root@oba-server opsaconf]# opsa status
Operations Analytics - Process Manager is not running.
OpsA Server status:
opsa-postgresql is running
opsa-server is stopped
opsa-storm-nimbus is stopped
opsa-task-manager is stopped
opsa-zookeeper is stopped
OpsA Outpost status:
opsa-outpost is running

However, post that when I tried to run the postinstall steps again, I found a new error: I am not sure how to handle this error now. I can see the opsaconf directory under "/var/opt/OV/databases" path though.

Jan 23, 2020 11:10:12 AM org.postgresql.Driver connect
SEVERE: Connection error:
org.postgresql.util.PSQLException: FATAL: database "opsaconf" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2422)
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2555)
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:131)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:212)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.getConnectionFromDM(PgPoolingDataSource.java:153)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.canConnect(PgPoolingDataSource.java:211)
at com.hp.opsa.infra.postgresql.PgPoolingDataSource.canConnect(PgPoolingDataSource.java:190)
at com.hp.opsa.install.postinstall.PostInstallConfigReader.readConfigParametersFromCLIInteractively(PostInstallConfigReader.java:148)
at com.hp.opsa.install.postinstall.PostInstallConfigReader.getConfigurationParameters(PostInstallConfigReader.java:57)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.getConfigurationParameters(PostInstallConfiguration.java:536)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.start(PostInstallConfiguration.java:508)
at com.hp.opsa.install.postinstall.PostInstallConfiguration.main(PostInstallConfiguration.java:544)

[dougg]

It might be that a system that already has postgres installed (or - has an LDAP or /etc/passwd user named postgres) will cause OBA app server installation to fail in this manner. 

If there was no trace of postgres installation or postgres user on the system prior to installation, I would suggest taking this into a formal Support call. 

[Sanchari Gupta]

Hi Doug,

In such case, can I use the "support/postgres-setup-fix.sh" script to fix this as mentioned on below link?

https://softwaresupport.softwaregrp.com/doc/KM03305848

Regards,

Sanchari Gupta

[dougg]

I was personally unaware of that support script. If you do have a postgres ldap user, it seems like it might help.