Fresh ESM Installation stops at "Set up ArcSight Storage"


Has someone seen this happen before?

Currently installing this on a Centos 6.7, all prerequisites installed, installer works without any issues, but when the last part of the first time wizard starts "About to Configure ESM v6.9.1c" i get this:

About to Configure ESM v6.9.1c

You are about to configure the software with the settings specified in previous panels. Once you click Next, you will not have the option to go back to the previous panels.

The configuration will accomplish the following tasks:

Set up ArcSight Storage

Start ArcSight Storage

Initialize schema and content

Configure the ArcSight Manager

Start the ArcSight Manager

Continue [yes] [yes/no/back/cancel]?yes

Set up ArcSight Storage

>>> Executing Command: /opt/arcsight/logger/current/arcsight/install/logger_init.sh <<<

And there it stays forever.

Now i have looked through some log entries, and this is what i see:


Logger temp logfile:

I see here that it stays and just spams "[Sat Mar 25 09:55:02 UTC 2017] is_server_up.sh: arcsight_server_ready does not exist" forever

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh 1 IA /opt/arcsight/logger/current/arcsight/logger /opt/arcsight/logger/current /opt/arcsight/logger/data/logger is called

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: prepare_new_layout_links starting

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/logger/bin/scripts/create_g8_links.sh :  Creating symlinks (if not already created) for re-laying out some directories

[Sat Mar 25 09:45:03 UTC 2017] Moving /opt/arcsight/logger/current/arcsight/logger/logs to /opt/arcsight/logger/userdata/logger

[Sat Mar 25 09:45:03 UTC 2017] - Creating link from /opt/arcsight/logger/current/arcsight/logger/logs to /opt/arcsight/logger/userdata/logger/logs

[Sat Mar 25 09:45:03 UTC 2017] Moving /opt/arcsight/logger/current/arcsight/logger/user to /opt/arcsight/logger/userdata/logger

[Sat Mar 25 09:45:03 UTC 2017] Backing up logger.properties

mv: cannot stat `/opt/arcsight/logger/userdata/logger/user/logger/logger.properties': No such file or directory

[Sat Mar 25 09:45:03 UTC 2017] Restoring logger.properties

mv: cannot stat `/opt/arcsight/logger/userdata/logger/user/logger/logger.properties.bkup': No such file or directory

[Sat Mar 25 09:45:03 UTC 2017] - Creating link from /opt/arcsight/logger/current/arcsight/logger/user to /opt/arcsight/logger/userdata/logger/user

[Sat Mar 25 09:45:03 UTC 2017] - Editing stuff in /etc/my.cnf

[Sat Mar 25 09:45:03 UTC 2017] Moving /opt/arcsight/logger/current/arcsight/connector/current/logs to /opt/arcsight/logger/userdata/connector

/opt/arcsight/logger/current/arcsight/logger/bin/scripts/create_g8_links.sh: line 177: cd: /opt/arcsight/logger/current/arcsight/connector/current/: No such file or directory

tar: logs: Cannot stat: No such file or directory

tar: Exiting with failure status due to previous errors

[Sat Mar 25 09:45:03 UTC 2017] - Creating link from /opt/arcsight/logger/current/arcsight/connector/current/logs to /opt/arcsight/logger/userdata/connector/logs

ln: creating symbolic link `/opt/arcsight/logger/current/arcsight/connector/current/logs': No such file or directory

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/logger/bin/scripts/create_g8_links.sh :  FATAL: Cannot set up symbolic link from /opt/arcsight/logger/current/arcsight/connector/current//logs to /opt/arcsight/logger/userdata/logger/logs.

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: prepare_new_layout_links done

ls: cannot access /opt/arcsight/logger/current/arcsight/logger/user/logger/receiver_folder_follower*.xml: No such file or directory

sed: can't read /opt/arcsight/logger/current/current/local/monit/watchdog/monitrc: No such file or directory

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Logger Software

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: prepare_appliance

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Appliance prepared

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: prepare_data_dir

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Data directory prepared

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Fresh installation

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: init_postgres

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Initializing postgres ...

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

Importing logger dump...

[Sat Mar 25 09:45:04 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Postgres initialized

[Sat Mar 25 09:45:04 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: init_or_upgrade_mysql

[Sat Mar 25 09:45:04 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Initializing mysql for logger ...

[Sat Mar 25 09:45:04 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Copying logger my.cnf ...

ESM detected

[Sat Mar 25 09:45:04 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Massage sizing in /opt/arcsight/logger/data/logger/../mysql/my.cnf for Boxster

[Sat Mar 25 09:45:04 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Shutting down logger server ...

Waiting for com.arcsight.logger.distributed.Server to terminate...

[Sat Mar 25 09:45:05 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: server status is ArcSight Logger Servers was not running.

[Sat Mar 25 09:45:05 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Server shutdown complete, server status is [ArcSight Logger Servers was not running.]

[Sat Mar 25 09:45:05 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Logger server shut down ...

[Sat Mar 25 09:45:05 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_logger_install.sh: Starting logger server ...

[Sat Mar 25 09:45:11 UTC 2017] is_server_up.sh: arcsight_server_ready does not exist

[Sat Mar 25 09:45:16 UTC 2017] is_server_up.sh: arcsight_server_ready does not exist

[Sat Mar 25 09:45:21 UTC 2017] is_server_up.sh: arcsight_server_ready does not exist

[Sat Mar 25 09:45:26 UTC 2017] is_server_up.sh: arcsight_server_ready does not exist

Then when i look at the actual logs i see that:

logger_init.sh.log:

2017-03-25 09:44:21 0000 logger_init.sh XXXXX command line: /opt/arcsight/logger/current/arcsight/install/logger_init.sh

2017-03-25 09:44:21 0000 logger_init.sh XXXXX installation directory: /opt/arcsight

2017-03-25 09:44:21 0000 logger_init.sh XXXXX generatedPropDir: /opt/arcsight/manager/tmp

2017-03-25 09:44:21 0000 logger_init.sh XXXXX loggerSilentPropsFile: logger.silent.properties

2017-03-25 09:44:21 0000 logger_init.sh XXXXX logfile: /opt/arcsight/logger/current/arcsight/logger/logs/logger_init.sh.log

2017-03-25 09:44:21 0000 logger_init.sh calling initSetupLogger

2017-03-25 09:44:21 0000 logger_init.sh  setLoggerLicenseFileSymLink(): For Logger's License file, create a sym link that points to ESM's license file location.

2017-03-25 09:44:21 0000 logger_init.sh  setLoggerLicenseFileSymLink(): About to finish this subroutine... Here is 'ls -l' output for the sym link (note the link's target file may not yet exist):

2017-03-25 09:44:21 0000 logger_init.sh  setmaxvolsize(): MAX storage volume size has been set to: 1000

2017-03-25 09:44:21 0000 logger_init.sh  setmaxgrpsize(): MAX storage group size has been set to: 1000

2017-03-25 09:44:21 0000 logger_init.sh  initSetupLogger(): Run Logger Installer as 'arcsight' user

logger_init_driver.log:

Read SQL template from /opt/arcsight/logger/current/arcsight/install/postgresql.template.sql

Using database URL jdbc:postgresql://localhost:1976/rwdb

org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "::1", user "web", database "rwdb", SSL off

        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:276)

        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:95)

        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:124)

        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

        at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24)

        at org.postgresql.Driver.makeConnection(Driver.java:386)

        at org.postgresql.Driver.connect(Driver.java:260)

        at java.sql.DriverManager.getConnection(DriverManager.java:571)

        at java.sql.DriverManager.getConnection(DriverManager.java:215)

        at com.arcsight.product.platform.business.dbupgrade.Main.getConnection(Main.java:331)

        at com.arcsight.product.platform.business.dbupgrade.Main.main(Main.java:50)

Could not obtain database connection.

Starting APS...ok

Configuring the monitor process

DELETE 0

[Sat Mar 25 09:45:03 UTC 2017] /opt/arcsight/logger/current/arcsight/install/post_doc_install.sh 1 IA /opt/arcsight/logger/current/arcsight/logger /opt/arcsight/logger/current is called

[2017-03-25 09:45:03 0000] /opt/arcsight/logger/current/arcsight/install/post_doc_install.sh:  If it does not exist, create the help symlink.

/opt/arcsight/logger/current/local/tomcat/webapps/logger /opt/arcsight/logger

[2017-03-25 09:45:03 0000] /opt/arcsight/logger/current/arcsight/install/post_doc_install.sh:  Setting the ./help symlink.

`help' -> `help.logger'

/opt/arcsight/logger

Sat Mar 25 09:45:03 UTC 2017 /opt/arcsight/logger/current/arcsight/install/post_storageengine_install.sh 1 IA /opt/arcsight/logger/current/arcsight/logger /opt/arcsight/logger/current /opt/arcsight/logger/current/arcsight/service is called

[Sat Mar 25 09:45:03 UTC 2017] storageengine.spec: Previous logger version is [0]

[Sat Mar 25 09:45:03 UTC 2017] storageengine.spec: startmysqld [/opt/arcsight/logger/current/arcsight/service/mysql start]

[Sat Mar 25 09:45:03 UTC 2017] storageengine.spec: stopmysqld  [/opt/arcsight/logger/current/arcsight/service/mysql stop]

[Sat Mar 25 09:45:03 UTC 2017] storageengine.spec: Register Logger StorageEngine as a MySQL plugin

[Sat Mar 25 09:45:03 UTC 2017] storageengine.spec: Logger StorageEngine registration succeeded ...

initmysqluser.log:

Database and User creation ----->

done @ 2017-03-25 09:45:04

logger_server.out.log:

*************/opt/arcsight/logger/current/arcsight/logger**************

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

Assuming ARCSIGHT_HOME: /opt/arcsight/logger/current/arcsight/logger

Assuming CONAPP_HOME: /opt/arcsight/logger/current/arcsight/logger

Assuming TOMCAT_HOME: /opt/arcsight/logger/current/local/tomcat

Assuming JAVA_HOME: /opt/arcsight/logger/current/local/jre

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

ArcSight Logger Servers starting...

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

ARCSIGHT_HOME=/opt/arcsight/logger/current/arcsight/logger

AbandonedObjectPool is used (org.apache.commons.dbcp.AbandonedObjectPool@10d42d49)

   LogAbandoned: true

   RemoveAbandoned: true

   RemoveAbandonedTimeout: 300

org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host "::1", user "web", database "rwdb", SSL off)

        at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:855)

        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)

        at com.arcsight.logger.db.CommonDBConnectionProvider.getConnection(CommonDBConnectionProvider.java:144)

        at com.arcsight.logger.db.CommonDBConnectionProvider.pingDatabase(CommonDBConnectionProvider.java:116)

        at com.arcsight.logger.db.CommonDBConnectionProvider.<init>(CommonDBConnectionProvider.java:85)

        at com.arcsight.logger.db.DBConnectionProvider.<init>(DBConnectionProvider.java:51)

        at com.arcsight.logger.db.DBConnectionProvider.getInstance(DBConnectionProvider.java:106)

        at com.arcsight.logger.db.DBConnectionProvider.shutdownIfDatabaseUnavailable(DBConnectionProvider.java:118)

        at com.arcsight.logger.distributed.Server.main(Server.java:712)

Caused by: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "::1", user "web", database "rwdb", SSL off

        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:275)

        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:94)

        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:65)

        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:116)

        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

        at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24)

        at org.postgresql.Driver.makeConnection(Driver.java:369)

        at org.postgresql.Driver.connect(Driver.java:245)

        at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)

        at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)

        at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:877)

        at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:851)

        ... 8 more

= Initializing Database =============================

URL: jdbc:postgresql://localhost:1976/rwdb

User: web

Database ping: Cannot create PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host "::1", user "web", database "rwdb", SSL off)

=====================================================

com.arcsight.common.exception.InitializationException: Exception initializing 'com.arcsight.logger.db.DBConnectionProvider': Could not connect to database at jdbc:postgresql://localhost:1976/rwdb using user web!

        at com.arcsight.logger.db.CommonDBConnectionProvider.<init>(CommonDBConnectionProvider.java:97)

        at com.arcsight.logger.db.DBConnectionProvider.<init>(DBConnectionProvider.java:51)

        at com.arcsight.logger.db.DBConnectionProvider.getInstance(DBConnectionProvider.java:106)

        at com.arcsight.logger.db.DBConnectionProvider.shutdownIfDatabaseUnavailable(DBConnectionProvider.java:118)

        at com.arcsight.logger.distributed.Server.main(Server.java:712)ls

arcsight_logger.log:

Sat Mar 25 09:45:04 UTC 2017 [15465] /opt/arcsight/logger/current/arcsight/service/arcsight_logger stop servers

No need to update permissions/ownership

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: ARCSIGHT_HOME is /opt/arcsight/logger/current/arcsight/service/../logger

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: ARCSIGHT_LOGGER_HOME is /opt/arcsight/logger/current/arcsight/logger

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: SERVER_READY_FILE: /opt/arcsight/logger/current/arcsight/service/../logger/tmp/.arcsight_server_ready

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: Waiting for server using /opt/arcsight/logger/current/arcsight/bin/query_platform need_process servers

cd /opt/arcsight/logger/current/arcsight/service/../logger/bin ; ./arcsight servers 2>&1 | /opt/arcsight/logger/current/arcsight/bin/safeAppend /opt/arcsight/logger/current/arcsight/service/../logger/logs/logger_server.out.log 2>&1 &

Sat Mar 25 09:45:04 UTC 2017 [ 15465] /opt/arcsight/logger/current/arcsight/service/arcsight_logger: Stopping ArcSight Logger Servers...

Sat Mar 25 09:45:04 UTC 2017 [15465] /opt/arcsight/logger/current/arcsight/service/arcsight_logger: ArcSight Logger Servers was not running.

Sat Mar 25 09:45:05 UTC 2017 [15542] /opt/arcsight/logger/current/arcsight/service/arcsight_logger status servers

No need to update permissions/ownership

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: ARCSIGHT_HOME is /opt/arcsight/logger/current/arcsight/service/../logger

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: ARCSIGHT_LOGGER_HOME is /opt/arcsight/logger/current/arcsight/logger

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: SERVER_READY_FILE: /opt/arcsight/logger/current/arcsight/service/../logger/tmp/.arcsight_server_ready

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: Waiting for server using /opt/arcsight/logger/current/arcsight/bin/query_platform need_process servers

cd /opt/arcsight/logger/current/arcsight/service/../logger/bin ; ./arcsight servers 2>&1 | /opt/arcsight/logger/current/arcsight/bin/safeAppend /opt/arcsight/logger/current/arcsight/service/../logger/logs/logger_server.out.log 2>&1 &

ArcSight Logger Servers was not running.

Sat Mar 25 09:45:05 UTC 2017 [15614] /opt/arcsight/logger/current/arcsight/service/arcsight_logger start servers

No need to update permissions/ownership

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: ARCSIGHT_HOME is /opt/arcsight/logger/current/arcsight/service/../logger

ARCSIGHT_LOGGER_HOME=/opt/arcsight/logger/current/arcsight/logger

ARCSIGHT_LOGGER_DATA=/opt/arcsight/logger/data

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: ARCSIGHT_LOGGER_HOME is /opt/arcsight/logger/current/arcsight/logger

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: SERVER_READY_FILE: /opt/arcsight/logger/current/arcsight/service/../logger/tmp/.arcsight_server_ready

Output will be logged to /opt/arcsight/logger/current/arcsight/service/../logger/logs/*.log

/opt/arcsight/logger/current/arcsight/service/arcsight_logger: Waiting for server using /opt/arcsight/logger/current/arcsight/bin/query_platform need_process servers

cd /opt/arcsight/logger/current/arcsight/service/../logger/bin ; ./arcsight servers 2>&1 | /opt/arcsight/logger/current/arcsight/bin/safeAppend /opt/arcsight/logger/current/arcsight/service/../logger/logs/logger_server.out.log 2>&1 &

Sat Mar 25 09:45:05 UTC 2017 [15614] /opt/arcsight/logger/current/arcsight/service/arcsight_logger: Starting ArcSight Logger Servers...

Sat Mar 25 09:45:11 UTC 2017 [15614] /opt/arcsight/logger/current/arcsight/service/arcsight_logger: Failed to start ArcSight Logger Servers

UPDATE:

After seeing the PSQL errors pointing towards a ::1 address (ipv6), i disabled IPV6 on the machine and tried again, now it is not connecting properly to the localhost psql.

Anyone got any ideas? ​ maybe?