ArcSight manager service not starting and web service is in mixed_statuses

[sagard1]

ArcSight manager service not starting and web service is in mixed_statuses.

The Manager Service is "Unavailable" and sometimes "initializing" and the Web Service is in "mixed_statuses".

We have tried restarting the problematic services. But since the services are in "mixed_statuses" and "initializing" so the services are not getting started neither getting getting stopped.

All the services were running properly previously hence there is no issue with port.

6046 [ERROR] /opt/arcsight/logger/current/local/mysql/libexec/mysqld: Incorrect key file for table './mysql/db'; try to repair it

134 [ERROR] Fatal error: Can't open and lock privilege tables: Incorrect key file for table './mysql/db'; try to repair it

1 0x7f71be098020-1]flag=18, query: SELECT event_id, external_id, name, message, event_type, start_time, end_time, manager_receipt_time, app_protocol,

trans_protocol, vulnerability, raw_event, bytes_in, bytes_out, session_id, generator, crypto_signature, concentrator_agents, concentrator_devices,

It seems table "DB" in MySQL is corrupted.

while repairing we are getting the error :  Can't connect to local MySQL server through socket '/opt/arcsight/logger/current/runtime/mysql.sock' (2).

Kindly help.

[michael.lumsden]

Hello ​,

Problems like this, especially involving system downtime/outages, are best addressed by contacting HPE ArcSight Support and working through a support ticket.  At first glance, this does appear to be some sort of database corruption, which results in a domino effect on the services - mysqld down means manager will be initializing/unavailable, which in turn means web will always be in "mixed_status" because it depends on manager.

Thanks and Regards,

Michael Lumsden

ESM Product Manager

[sagard1]

Thanks Michael,

We have lodge a call to support team and its resolved at the moment.

[mr.akashbhardwa]

Hi Sagar, how did this issue got resolved. I meant what changes the support team did to make it works again.

[sagard1]

Our database was corrected which couldn't be recovered. database was restored from backup.

The main issue was stopping the services. They had to forcefully close all the services including monit service so that services doesn't start automatically.

[sagard1]

Our database was corrupted which couldn't be recovered. database was restored from backup.

The main issue was stopping the services. They had to forcefully close all the services including monit service so that services doesn't start automatically.

[achmad.abrar1]

Hi Sagar,

I had the same problem, could you explain step by step to solve this?

I really apreciate your help!

Regrads,

Achmad Abrar

[sagard1]

Hi Achmad,

Find the moint service.

ps -ef |grep monit

Kill the monit services forcefully and monitor if the service is getting up once again.

watch -d ps -ef | grep monit

kill -9 <Process ID>

OR

You can also do

Kill /bin/bash /etc/init.d/arcsight_services start monit

OR

Run /etc/init.d/arcsight_service stop monit

We did tried below two commands to stop the monit services but it didn't help.

After monit service is off stop services one by one and start one by one.

Sequence of starting the service :

/sbin/service arcsight_services start mysqld

/sbin/service arcsight_services start logger

/sbin/service arcsight_services start manager

/sbin/service arcsight_services start arcsight_web

[achmad.abrar1]

Hi Sagar,

Thankyou!

Regards,

Achmad Abrar

[sagard1]

You are most welcome