Logger - EPS out not working

Dear Community,

Our Arcsight logger rebooted and after that, we had no more EPS Out. (EPS in are OK)

Logger version : 7.2.2

ESM Version : 7.5

Our logger si sending the logs to our ESM. The Logger have the log, but they never leave the logger and never reach the ESM.

We :

-Checked the flow

-Recreated the forwarder ( we dont have querie )

-Recreated the ESM destination

-Restarted the logger

-Restarted the conector (in systemadmin tab and in SSH )

-Desactive SELinux

-Restored the VM as a day before the reboot

-Re-imported the certificate

-Check the java memory

-Re-installed the connector.

We also raised a ticket to Micro Focus, bu unfortunatly they have no idea of what's going on yet.

As this as a real impact on our production, we need as many help as we can.

We have those kind of error message in our agent.log and agent.out.wrapper.log :

INFO   | jvm 1    | 2023/06/28 16:39:25 | [Wed Jun 28 16:39:25 CEST 2023] [ERROR] com.arcsight.agent.transport.o: com.arcsight.common.an.e: Failed to communicate with 'https://internal.esm:8443/arcsight/agent/event'. HTTP response: 401
ERROR  | wrapper  | 2023/06/28 16:39:26 | JVM exited unexpectedly.

-----------------------------

[2023-06-29 10:55:51,465][ERROR][com.arcsight.agent.configtool.e] [adjustConfigBasedOnManagerVersion]Could not detect manager version for destination [3EnddBokBABC5w8MptCKgmQ==]
[2023-06-29 10:58:45,123][ERROR][com.arcsight.agent.d8.e] [runConsumer]The thread stopped while waiting for commands.
[2023-06-29 10:58:59,144][ERROR][com.arcsight.agent.util.a.c] [run]java.lang.InterruptedException

----------------------------

If you have any idea of possible action we can take or test we can do please let me know !

Regards,

Alexandra

  • Tough to say what might be happening here, checking the username/password and certificates to ESM might be a good idea if you haven't already.  But I would also suggest to check ESM if you have a connector created from Logger that's previously there.  If you hadn't deleted it previously, you'll want to delete it and either try to reconnect or recreate the ESM destination from Logger. 

  • Dear Vtham1, 

    We already checked the credentials and 2 other admin account and we have the same results.

    We also reimported the certificate and restart the logger, no results..

    We also deleted the connector on the ESM before recreating the ESM destination in the logger, but nothing changed.

  • Hi Alexandra... this is hard to diagnose from apart, things i would check:

    - is the communication working telnet to ESM port 8443 (from the logger)

    - is the certificate of the ESM is not expired

    - check if the certificate shows the hostname you configured the logger to talk to

    - check DNS/IP address resolution

    - it might be, that the logger is unable to negotiate the right tls version, pls remove.

    • server.properties :

    ssl.protocols.nonfips=TLSv1,TLSv1.1,TLSv1.2

    - check for cacerts file in  "current/arcsight/connector/current/jre/lib/security", and if it has >0 bytes.

  • or check which ciphers the ESM uses, and change the connectore somehow to align (in agent.properties)

    ssl.cipher.suites=TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

    follow link to find the chipers.

    superuser.com/.../how-do-i-list-the-ssl-tls-cipher-suites-a-particular-website-offers

  • Dear vitz1,

    - is the communication working telnet to ESM port 8443 (from the logger) => Yes

    - is the certificate of the ESM is not expired => Yes (expiration 2026)

    - check if the certificate shows the hostname you configured the logger to talk to => Certificate is OK

    - check DNS/IP address resolution => Everything is OK on this side

    - it might be, that the logger is unable to negotiate the right tls version, pls remove. ==> I didnt find the server.properties in my logger.

    I checked the ciphers in the ESM and he uses ECDHE_RSA_WITH_AES_128_GCM_SHA256.

    I tried to check my agent.properties but i don't see where I need to make the change to 'align' my connector.

    Thanks !

    Alexandra

  • > I tried to check my agent.properties but i don't see where I need to make the change to 'align' my connector.
    current/arcsight/connector/current/ ... i assume user agent.

    > it might be, that the logger is unable to negotiate the right tls version, pls remove. ==> I didnt find the server.properties in my logger.

    the server.prop lives on ESM

  • Other things i would take into consideration:  

    - is it possible to register a syslog smart connector to the esm, and send some dummy events?

    - are you using the design "lots of connectors send to logger" and "logger sends to ESM"?

    -- if so, this design is not recommended any more, also because the forwarind connector mosty is not able to handle >1000 EPS.

    - what was the cause of the reboot?

    -- Disk full?

    - are you using a logger appliance or SW?

  • do you send audit(health) logs along that connector as well?

  • Dear Vitz,

    Thanks for all the recommendation !

    - About the TLS version => Removed it didnt change anything


    - is it possible to register a syslog smart connector to the esm, and send some dummy events => No i cannot test that. ( We already receive the internal event in the ESM but nothing else)


    - I'm not sure to understand your question but , we have this desgin :
    1 client = 1 logger
    Every logger forward the log to the ESM
    We have 20 logger who forward their log to the ESM
    The 20 logger always worked well until now
    But, now your saying we added a new logger last week, can it be related ? If so can i override the 1000eps ? Or where can i see if this limit is reached ?

    - We had a SPM on the VM and this caused the reboot. We already restored the VM but the problem is still here even if the state was restored 1 day before the reboot.

    - The disk is half empty.

    - We are using a SW logger.

    - Yes we have the audit logs along that connector.

  • ... wait ... i read:

    - the connector from this one logger is up and running (also showing as up in the ESM?)

    - you are getting the audit logs from that particular logger via the "forwarding" connector of that logger

    - you are not getting other events via that connector

    things i would check:

    - the config to forward events is right

    - there is no filter out on the connector for the ESM-Destination /check on ESM itself

    - there is no filter out in the connector itself

    - can you send Dummy events via this connector?  "echo abc > /dev/[tcp/udp]localhost/[port]

    - is ther a filter out for unparsed events? (check ESM config for the connector)

    - is the logger "sending" events to the connector?/ tcpdump?