SmartConnector installation fails since version 6.0.1.6574

[jpvasgard]

Hi all,

I am having serious issues installing SmartConnectors since version 6.0.1. I have tried all available 6.0.x versions, 6.0.1, 6.0.2 and 6.0.3, all give me the same result.

Vanilla RedHat 6.2 setup with all recommended libraries starting installation in console mode (SSHing into system, no X11 forwarding supported). Running the .bin file copies in "root" context, all the required Java-bruhaha in the specified path and asks to manually run "runagentsetup.sh". Running the script, again in "root" context, I get the following error:

[root@host]# ./runagentsetup.sh 

Assuming ARCSIGHT_HOME: /opt/arcsight/sconnectors/syslog_udp/current
Assuming JAVA_HOME: /opt/arcsight/sconnectors/syslog_udp/current/jre

ArcSight Agent Setup starting...

Connector Setup Wizard starting in mode [CONSOLE]
[Wed Jun 19 14:05:07 UTC 2013] [INFO ] Checking for a running instance of connector...
[Wed Jun 19 14:05:07 UTC 2013] [INFO ] Starting up connector...
FATAL EXCEPTION:
Could not launch an instance of Connector
FATAL EXCEPTION:
No connector found at the specified port [10001]... exiting
[Wed Jun 19 14:07:08 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.
[Wed Jun 19 14:07:08 UTC 2013] [ERROR] You may need to shut it down manually.
[Wed Jun 19 14:07:08 UTC 2013] [INFO ] Shutting Down Agent Framework Version [6.0.3.6664.0]

In the agentinstall.log I can find loads of error messages, about "connection refused". Logfile is attached, if you want to inspect it for further information.

This is repeated until a timeout is reached. Every once in a while (about 1 out of 5 attempts), I do get a connection after a looong wait on "Starting up connector ...". However, I don't trust the resulting installation.

Is anybody else experiencing issues with smart connector installation timing out? What could be causing this long timeout issue? I tried small letter hostname, different domain, I double checked installed libraries and I have a support ticket open at hp. So far no results.

Cheers

JP

[jpvasgard]

And even more info. Somone suggested to check selinux settings. Forgot to mention, I did on another machine, but not on my third fresh trial installation. So, yep, blame myself

[root@smalltest ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   permissive
Mode from config file:          permissive
Policy version:                 24
Policy from config file:        targeted

I also made sure no firewall rules are possibly interfering:

[root@smalltest ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         


Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         


Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination  

It now seems to work reliably.

Let me sum this up for future reference or others having issues:

1) Make sure all libraries are installed, both 64 and 32 bit versions on 64 bit machines (both .i686 and .x86_64)

glibc
libX11
libXi
libXext
libXtst
libXau
libxcb
binutils
compat-libstdc++-33
libaio
sysstat

I am not sure, why the X libs are required for pure console mode installation, but everybody always tells you to install them, including ArcSight support, so I did.

2) Make sure, SELinux is set to "permissive" or "disabled" mode in /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 

3) Make sure, no local firewall rules are set up to interfere with connections by issuing an "iptables -F" or carefully examining the ruleset.

[root@smalltest ~]# iptables -L
Chain INPUT (policy ACCEPT)
target      prot     opt    source     destination

Chain FORWARD (policy ACCEPT)
target     prot     opt     source     destination

Chain OUTPUT (policy ACCEPT)
target     prot     opt     source     destination          

I also attached a complete list of installed packages to this post, in case someone still has difficulties and wants to compare. If all of this does not work, then all I can suggest is go out and find yourself a shaman to conjure the machine into compliance.

Cheers

JP

[jring1]

Thanks for the wrap-up for posterity... so it probably was SELinux in enforcing mode? Seems our systems guys did not enable that one...

Joachim

[jpvasgard]

Hi Joachim,

it looks indeed like SELinux was a bit restrictive. I am still uncomfortable with this, because I was not able to precisely point at the cause of the error. There seem to be differences in behaviour between GUI and console mode and I had trouble defining a clear baseline for the experiments. When I have some more time at hand I will try this again and post any new findings I might come accross.

Meanwhile I hope I remain to be the only person experiencing the problem. If not, this post should give enough hints where to start looking

Cheers

JP

[jpvasgard]

Hi all,

Summary: Depleted entropy pool causes installation to hang during key generation phase.

Contrary to my initial assumption, the problems persisted even after changing settings to SELinux and iptables. After some more research and debugging, we finally got to the core of the problem:

During the installation of a SmartConnector, one of the earlier steps during the installation is the generation of a cryptographic key pair for remote management. You can check in your agent.log file for the following line:

Creating Remote Management Keypair [<ArcsightHome>/current/jre/bin/keytool -genkey -alias tomcat -keyalg RSA -storepass ***** -storetype pkcs12 -keypass ***** -keysize 2048 -validity 3650 -keystore <ArcSightHome>/current/user/agent/remote_management.p12 -dname CN=,OU=7jKXOy4BABCPAflKtoW9cQ,O=ArcSight,L=NA,ST=NA,C=US ]

In order to generate those keys, keytool requires suficcient data available in the entropy pool. On a Linux VM under VMWare this is not the case, if you log in via SSH. Network traffic is not accepted as entropy source. Due to a depleted entropy pool, the key generation process hangs and after two minutes the installer pulls the plug. Look for the line killing the process in agent.log after the default timeout, usually one or two lines down from the call of the command:

Killed process: java.lang.UNIXProcess@10382a9 (<ArcSightHome>/current/jre/bin/keytool -genkey -alias tomcat -keyalg RSA -storepass changeit -storetype pkcs12 -keypass changeit -keysize 2048 -validity 3650 -keystore <ArcSightHome>/current/user/agent/remote_management.p12 -dname CN=,OU=7jKXOy4BABCAAflKtoW9cQ,O=ArcSight,L=NA,ST=NA,C=US  -- null) after 120002 (timeout was: 120000)

However, as a leftover, you will have a file under <ArcSightHome>/current/user/agent/ named remote_management.p12 with a size of 0 Bytes. In subsequent calls, the installer just checks for the existence of this file and skips the key generation phase. This is the reason, we only observed the issue during the initial installation phase.

You can check how much entropy data is available on your system by using

cat /proc/sys/kernel/random/entropy_avail

which returns the number of bits still available. But careful: the "cat" command itself reduces the amount of available entropy. On a system with sufficient entropy sources, this should be 4096, the default poolsize in a modern Linux kernel

cat /proc/sys/kernel/random/poolsize

There are several ways to work around this issue:

1. If you have direct console level access to the VM (TTY), you will most likely not see the issue, because pressing keys is used a valid source for entropy. If you get stuck, you can press keys like "shift", "ctlr" or "alt" to help generate some more entropy. You can also generate loads of disk access to achieve the same. Try dd if=/dev/sda of=/dev/null for example. Please set the input "if=" to some large source drive you have configured and access to on your machine.
2. If you have a slow but steady flow of entropy, you can extend the timeout in <ArcSightHome>/current/user/agent/agent.properties by manipulating the parameter agent.defaults.properties.remote.management.ssl.keypair.creation.timeout=<SECONDS>. Default is 120. We tested running the keytool command directly on the console and on our machine it took between 5 and 6 minutes to generate the keypair. So a value of 360 seconds was sufficient. Your mileage my vary, as this depends on your individual setup.
3. You can add sources for entropy using third party tools such as entropy broker (URL: Entropy Broker ), rng-tools in combination with a hardware entropy generator (URL: rng-tools) or HAVEGE to use other sources as well (URL: haveged - a simple entropy daemon)
4. You can off-load key generation to a secure machine with sufficient entropy available, using the keytool. You have to use the string found under <ArcSightHome>/current/user/agent/agent.properties remote.manaegment-ssl.organizational.unit=<STRING>. and import the result manuall.

Either way should give you the desired result. I hope this will help other people running into the same issue.

Cheers

JP

[jring1]

Thanks for the update... this sounds like hell to debug - especially since it only happens on a fresh vm...

I mean it's a good idea to check for sufficient entropy before key generation since otherwise one gets weak keys like it often happens on soho routers and the like...

Joachim

[Dean Farrington]

Try this with the new 6.0.4 build. There was a bug we encountered with installing on RHEL that was supposed to be fixed in the 6.0.4 build that was recently released so this may be resolved in the latest connector build.

[jpvasgard]

Hi Farridem,

I cannot test this right now but will give feedback once I gave it a try.

However, the issue we encountered was not due to a bug in the ArcSight SmartConnector software, but rather a general issue with entropy in Linux VM Environments and a feature of the Java "keytool" to use strong cryptographic methods rather than relying on a weaker source of entropy.

This is nothing that can be patched away by ArcSight without compromising security in some way. It is more VMWares job to come up with a way to feed entropy to guest systems without opening up security breaches such as side channel attacks due to predictable entropy data.

Cheers

JP

[yix@fusionskye.]

thanks for sharing the solution ， i have the same issue，but i solve it just by removing the agent.lock file

[slacker]

Hello, I had the same exact problem and errors, but was able to get past the error and configure the agent.  Here is how I did it, but I am unsure about the root cause:

Agent Version: 6.0.4.6719

OS Version: RedHat 64-bit, 6.2, No X-11 / GUI (running on a virtual machine / Citrix)

Error: "Caused by: java.rmi.RemoteExecption: Unable to retrieve entities from [http://localhost:10001/cwsapi/services/v1]

Steps to resolve:

1.) rm {Install Dir}/current/run/agent.lock -f

2.) Temporarily stop iptables during setup Only.  service iptables stop

3.) ./runagentsetup.sh (from connector bin directory)

[tejeu_tejeu1]

Hi,

I am facing same kind of issue on windows platform(Windows 2008 R2).

  Currently I am installing Arcsight smart connector for CISCO IRON Port email mail gateway. After starting the services automatically agent.lock and agent2.lock files are getting created on the current\run path. I tried to delete and restart the services but again I am getting the same.

I am using ArcSight-6.0.7.6901.0-Connector version smart connector file

Regards,

Tejesh

[hpgbrice1]

Did anyone ever get this issue resolved?

I am running into the same problem on my Linux machine.

Please help!

[jpvasgard]

Hi,

can you give us the exact version and flavor of connector you are trying to install? I recently installed from 7.1.3.7445.0 Linux64 Folder Follower Flexconnector without issues.

Have you checked for the entropy problems mentioned above in this thread?

Cheers

JP