SmartConnector installation fails since version 6.0.1.6574

Hi all,

I am having serious issues installing SmartConnectors since version 6.0.1. I have tried all available 6.0.x versions, 6.0.1, 6.0.2 and 6.0.3, all give me the same result.

Vanilla RedHat 6.2 setup with all recommended libraries starting installation in console mode (SSHing into system, no X11 forwarding supported). Running the .bin file copies in "root" context, all the required Java-bruhaha in the specified path and asks to manually run "runagentsetup.sh". Running the script, again in "root" context, I get the following error:

[root@host]# ./runagentsetup.sh

Assuming ARCSIGHT_HOME: /opt/arcsight/sconnectors/syslog_udp/current

Assuming JAVA_HOME: /opt/arcsight/sconnectors/syslog_udp/current/jre

ArcSight Agent Setup starting...

Connector Setup Wizard starting in mode [CONSOLE]

[Wed Jun 19 14:05:07 UTC 2013] [INFO ] Checking for a running instance of connector...

[Wed Jun 19 14:05:07 UTC 2013] [INFO ] Starting up connector...

FATAL EXCEPTION:

Could not launch an instance of Connector

FATAL EXCEPTION:

No connector found at the specified port [10001]... exiting

[Wed Jun 19 14:07:08 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.

[Wed Jun 19 14:07:08 UTC 2013] [ERROR] You may need to shut it down manually.

[Wed Jun 19 14:07:08 UTC 2013] [INFO ] Shutting Down Agent Framework Version [6.0.3.6664.0]

In the agentinstall.log I can find loads of error messages, about "connection refused". Logfile is attached, if you want to inspect it for further information.

This is repeated until a timeout is reached. Every once in a while (about 1 out of 5 attempts), I do get a connection after a looong wait on "Starting up connector ...". However, I don't trust the resulting installation.

Is anybody else experiencing issues with smart connector installation timing out? What could be causing this long timeout issue? I tried small letter hostname, different domain, I double checked installed libraries and I have a support ticket open at hp. So far no results.

Cheers

JP

  • Hi,

    I have not yet installed a regular SmartConnector on 6.2 yet (only forwarding connectors up to now) but I always use a non-privileged account (usually called arcsight) to run the installer file and later the connector service under.

    I don't think that an installation under root is a good idea because you'll either have to correct a lot of permission problems to run the service under a non-privileged account later or you run the service under root, which is not advisable for security reasons...

    Also I never had to install java myself for any arcsight product - they come with an intrgrated JRE and they'll set the environment up in their scripts...

    PS: Just tried a test install of Smart Connector 6.0.3 on RHEL 6.x as user arcsight and had no problems...

    Joachim

    Edited for test install

  • try running setup this way:

    "./arcsight agentsetup -w -i console"

    if that doesn't work try this

    "arcsight agentsetup -i console"

    Hope that helps

  • If you had no problems under user "arcsight", how do you handle privileged ports like 514/udp for a syslog connector? You have to have "root" permissions to bind to low ports. Otherwise, you are right. Always use least privileges.

    Oh and I meant the Java-Stuff that gets copied by the initial ArcSight installer, not a custom Java environment. Indeed, since Java is so wonderfully portable, every developer ships their own JRE including all libraries etc. to make sure, the software actually runs in the target environment

    Cheers

    JP

  • Hi,

    Thank you for your time and help. I tried it all. Not that it matters much, since console mode is chosen automatically in case the X11 does not work (as shown in my original post)


    Connector Setup Wizard starting in mode [CONSOLE] <-- hint hint

    It still gives the same errors:

    [jens@smalltest bin]$ ./runagentsetup.sh -i console

    Assuming ARCSIGHT_HOME: /opt/arcsight/syslog/current

    Assuming JAVA_HOME: /opt/arcsight/syslog/current/jre

    ArcSight Agent Setup starting...

    Connector Setup Wizard starting in mode [console]

    [Wed Jun 19 18:09:37 UTC 2013] [INFO ] Checking for a running instance of connector...

    [Wed Jun 19 18:09:37 UTC 2013] [INFO ] Starting up connector...

    FATAL EXCEPTION:

    Could not launch an instance of Connector

    FATAL EXCEPTION:

    No connector found at the specified port [10001]... exiting

    [Wed Jun 19 18:11:37 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.

    [Wed Jun 19 18:11:37 UTC 2013] [ERROR] You may need to shut it down manually.

    [Wed Jun 19 18:11:37 UTC 2013] [INFO ] Shutting Down Agent Framework Version [6.0.3.6664.0]

    Interesting enough, I do spot a difference. The "Connector Setup Wizard starting in mode [console]" now has console in lower case. It is all upper case, if I leave out the -i console option. Hmm.

    Can someone that has no problems attach his/her output of "yum list installed"? I would like to compare it to my installation.

    I also found some sort of "race condition". If I start another installation right after the first one aborts, it runs like a charm. I checked and sure enough, there is an instance of an arcsight connector running in the process list

    [jens@smalltest bin]$ ps aux | grep "arcsight"

    jens      6857  0.0  0.0 106188  1560 pts/0    S    18:09   0:00 /bin/sh /opt/arcsight/syslog/current/bin/scripts/execjava.sh -noEvntProc

    jens      6860  0.9  4.8 438848 191356 pts/0   Sl   18:09   0:06 /opt/arcsight/syslog/current/jre/bin/java -server -verbose:gc -Djava.security.policy=/opt/arcsight...

    and "netstat -lnt" shows me some java process listening on port 10001:

    [jens@smalltest bin]$ netstat -lntp

    Aktive Internetverbindungen (Nur Server)

    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name

    [...]

    tcp        0           0           ::ffff:127.0.0.1:10001      :::*                                       LISTEN      6860/java

    Why is it not detecting it running earlier? I am puzzled.

    Cheers

    JP

  • Jens Prüfer wrote:

    If you had no problems under user "arcsight", how do you handle privileged ports like 514/udp for a syslog connector? You have to have "root" permissions to bind to low ports. Otherwise, you are right. Always use least privileges.

    Well, we usually use a piece of software that knows how to drop privileges after opening the socket - like it's been standard on *nix since 20 years or so - take the syslogd of your choice and use a syslogfilereader to parse the logs.

    Other methods might include:

    - setcap 'cap_net_bind_service= ep' /path/to/arcsight/java - not nice due to giving the capability to the jvm and might not even work due to a lot of ifs...

    See

    Is there a way for non-root processes to bind to &amp;quot;privileged&amp;quot; ports (&amp;lt;1024) on Linux? - Stack O…

    - iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-ports 5514  - and have the conector running under arcsight listen to 5514. I would use this if I absolutely had to use the syslog port connector for some reason.

    PS: If you absolutely want to run under root - try installing under arcsight and see if this works. You can always give root as the user to run the service under in the runasroot script. With root having all rights regardless of DAC bits it should run fine, or you could chown -R root $AGENT_HOME if you want to be extra tidy...

    Joachim

    Added postscriptum

  • Hi Joachim,

    thank you for the "iptables" hint. Nice one that seems to be feasible in this case. Ok. Will use arcsight user for priv. ports from now on

    However, this does not solve the original issue of the connector not starting up properly for initial installation (or the connection problems and race condition described above).

    Any more hints on that one? I wish someone could at least reproduce it ...

    Cheers

    JP

  • Hi Jens,

    What about to try the installation of the last version 5.x.x ?

    Just wonder if the same error occurs.

    Pavel

  • Seems like there's sth wrong with your box... I tried install and config as root to test and it worked.

    Btw. if you're running RHEL6 64bit you might need some 32 bit compat packages... we have the following installed in the standard build...

    compat-libstdc -296.i686    2.96-144.el6       @clone-prod-rhel-x86_64-server-6

    glibc.i686                   2.12-1.80.el6_3.4  @clone-prod-rhel-x86_64-server-6

    libgcc.i686                  4.4.6-4.el6        @clone-prod-rhel-x86_64-server-6

    nss-softokn-freebl.i686      3.12.9-11.el6      @clone-prod-rhel-x86_64-server-6

    Joachim

  • Hello Joachim,

    These libraries are installed:

    compat-libstdc -33-3.2.3-69.el6.i686

    compat-libstdc -33-3.2.3-69.el6.x86_64

    glibc-2.12-1.107.el6.i686

    glibc-2.12-1.107.el6.x86_64

    libgcc-4.4.7-3.el6.x86_64

    libgcc-4.4.7-3.el6.i686

    nss-softokn-freebl-3.12.9-11.el6.i686

    nss-softokn-freebl-3.12.9-11.el6.x86_64

    Is it ok?

    Thanks

    Pavel

  • Hi all,

    I tested with version 5.2.6.6424.0 and have the same issue:

    Connector Setup Wizard starting in mode [console]

    [Thu Jun 20 02:22:22 UTC 2013] [INFO ] Checking for a running instance of connector...

    [Thu Jun 20 02:22:23 UTC 2013] [INFO ] Starting up connector...

    FATAL EXCEPTION:

    Could not launch an instance of Connector

    FATAL EXCEPTION:

    No connector found at the specified port [10001]... exiting

    [Thu Jun 20 02:24:24 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.

    [Thu Jun 20 02:24:24 UTC 2013] [ERROR] You may need to shut it down manually.

    [Thu Jun 20 02:24:24 UTC 2013] [INFO ] Shutting Down Agent Framework Version [5.2.6.6434.0]

    Also, directly after this "shutting down" message, I can see the connector running and port 10001/tcp listening for connections.

    Can someone with no such issues please provide me a copy of "yum list installed" so I can compare packages?

    Cheers

    JP