Re: SmartConnector installation fails since version 6.0.1.6574 - Micro Focus Community

jpvasgard · ‎2013-06-19

Hi all,

I am having serious issues installing SmartConnectors since version 6.0.1. I have tried all available 6.0.x versions, 6.0.1, 6.0.2 and 6.0.3, all give me the same result.

Vanilla RedHat 6.2 setup with all recommended libraries starting installation in console mode (SSHing into system, no X11 forwarding supported). Running the .bin file copies in "root" context, all the required Java-bruhaha in the specified path and asks to manually run "runagentsetup.sh". Running the script, again in "root" context, I get the following error:

[root@host]# ./runagentsetup.sh 

Assuming ARCSIGHT_HOME: /opt/arcsight/sconnectors/syslog_udp/current
Assuming JAVA_HOME: /opt/arcsight/sconnectors/syslog_udp/current/jre

ArcSight Agent Setup starting...

Connector Setup Wizard starting in mode [CONSOLE]
[Wed Jun 19 14:05:07 UTC 2013] [INFO ] Checking for a running instance of connector...
[Wed Jun 19 14:05:07 UTC 2013] [INFO ] Starting up connector...
FATAL EXCEPTION:
Could not launch an instance of Connector
FATAL EXCEPTION:
No connector found at the specified port [10001]... exiting
[Wed Jun 19 14:07:08 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.
[Wed Jun 19 14:07:08 UTC 2013] [ERROR] You may need to shut it down manually.
[Wed Jun 19 14:07:08 UTC 2013] [INFO ] Shutting Down Agent Framework Version [6.0.3.6664.0]

In the agentinstall.log I can find loads of error messages, about "connection refused". Logfile is attached, if you want to inspect it for further information.

This is repeated until a timeout is reached. Every once in a while (about 1 out of 5 attempts), I do get a connection after a looong wait on "Starting up connector ...". However, I don't trust the resulting installation.

Is anybody else experiencing issues with smart connector installation timing out? What could be causing this long timeout issue? I tried small letter hostname, different domain, I double checked installed libraries and I have a support ticket open at hp. So far no results.

Cheers

JP

jring1 · ‎2013-06-19

Hi,

I have not yet installed a regular SmartConnector on 6.2 yet (only forwarding connectors up to now) but I always use a non-privileged account (usually called arcsight) to run the installer file and later the connector service under.

I don't think that an installation under root is a good idea because you'll either have to correct a lot of permission problems to run the service under a non-privileged account later or you run the service under root, which is not advisable for security reasons...

Also I never had to install java myself for any arcsight product - they come with an intrgrated JRE and they'll set the environment up in their scripts...

PS: Just tried a test install of Smart Connector 6.0.3 on RHEL 6.x as user arcsight and had no problems...

Joachim

Edited for test install

brian.freedman@1 · ‎2013-06-19

try running setup this way:

"./arcsight agentsetup -w -i console"

if that doesn't work try this

"arcsight agentsetup -i console"

Hope that helps

jpvasgard · ‎2013-06-19

If you had no problems under user "arcsight", how do you handle privileged ports like 514/udp for a syslog connector? You have to have "root" permissions to bind to low ports. Otherwise, you are right. Always use least privileges.

Oh and I meant the Java-Stuff that gets copied by the initial ArcSight installer, not a custom Java environment. Indeed, since Java is so wonderfully portable, every developer ships their own JRE including all libraries etc. to make sure, the software actually runs in the target environment

Cheers

JP

jpvasgard · ‎2013-06-19

Hi,

Thank you for your time and help. I tried it all. Not that it matters much, since console mode is chosen automatically in case the X11 does not work (as shown in my original post)

Connector Setup Wizard starting in mode [CONSOLE] <-- hint hint

It still gives the same errors:

[jens@smalltest bin]$ ./runagentsetup.sh -i console

Assuming ARCSIGHT_HOME: /opt/arcsight/syslog/current
Assuming JAVA_HOME: /opt/arcsight/syslog/current/jre

ArcSight Agent Setup starting...

Connector Setup Wizard starting in mode [console]
[Wed Jun 19 18:09:37 UTC 2013] [INFO ] Checking for a running instance of connector...
[Wed Jun 19 18:09:37 UTC 2013] [INFO ] Starting up connector...
FATAL EXCEPTION:
Could not launch an instance of Connector
FATAL EXCEPTION:
No connector found at the specified port [10001]... exiting
[Wed Jun 19 18:11:37 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.
[Wed Jun 19 18:11:37 UTC 2013] [ERROR] You may need to shut it down manually.
[Wed Jun 19 18:11:37 UTC 2013] [INFO ] Shutting Down Agent Framework Version [6.0.3.6664.0]

Interesting enough, I do spot a difference. The "Connector Setup Wizard starting in mode [console]" now has console in lower case. It is all upper case, if I leave out the -i console option. Hmm.

Can someone that has no problems attach his/her output of "yum list installed"? I would like to compare it to my installation.

I also found some sort of "race condition". If I start another installation right after the first one aborts, it runs like a charm. I checked and sure enough, there is an instance of an arcsight connector running in the process list

[jens@smalltest bin]$ ps aux | grep "arcsight"

jens 6857 0.0 0.0 106188 1560 pts/0 S 18:09 0:00 /bin/sh /opt/arcsight/syslog/current/bin/scripts/execjava.sh -noEvntProc

jens 6860 0.9 4.8 438848 191356 pts/0 Sl 18:09 0:06 /opt/arcsight/syslog/current/jre/bin/java -server -verbose:gc -Djava.security.policy=/opt/arcsight...

and "netstat -lnt" shows me some java process listening on port 10001:

[jens@smalltest bin]$ netstat -lntp

Aktive Internetverbindungen (Nur Server)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

[...]

tcp 0 0 ::ffff:127.0.0.1:10001 :::* LISTEN 6860/java

Why is it not detecting it running earlier? I am puzzled.

Cheers

JP

jring1 · ‎2013-06-20

Jens Prüfer wrote:

If you had no problems under user "arcsight", how do you handle privileged ports like 514/udp for a syslog connector? You have to have "root" permissions to bind to low ports. Otherwise, you are right. Always use least privileges.

Well, we usually use a piece of software that knows how to drop privileges after opening the socket - like it's been standard on *nix since 20 years or so - take the syslogd of your choice and use a syslogfilereader to parse the logs.

Other methods might include:

- setcap 'cap_net_bind_service=+ep' /path/to/arcsight/java - not nice due to giving the capability to the jvm and might not even work due to a lot of ifs...

See

Is there a way for non-root processes to bind to &quot;privileged&quot; ports (&lt;1024) on Linux? - Stack O…

- iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-ports 5514 - and have the conector running under arcsight listen to 5514. I would use this if I absolutely had to use the syslog port connector for some reason.

PS: If you absolutely want to run under root - try installing under arcsight and see if this works. You can always give root as the user to run the service under in the runasroot script. With root having all rights regardless of DAC bits it should run fine, or you could chown -R root $AGENT_HOME if you want to be extra tidy...

Joachim

Added postscriptum

jpvasgard · ‎2013-06-20

Hi Joachim,

thank you for the "iptables" hint. Nice one that seems to be feasible in this case. Ok. Will use arcsight user for priv. ports from now on

However, this does not solve the original issue of the connector not starting up properly for initial installation (or the connection problems and race condition described above).

Any more hints on that one? I wish someone could at least reproduce it ...

Cheers

JP

pavel.novotny · ‎2013-06-20

Hi Jens,

What about to try the installation of the last version 5.x.x ?

Just wonder if the same error occurs.

Pavel

jring1 · ‎2013-06-20

Seems like there's sth wrong with your box... I tried install and config as root to test and it worked.

Btw. if you're running RHEL6 64bit you might need some 32 bit compat packages... we have the following installed in the standard build...

compat-libstdc++-296.i686 2.96-144.el6 @clone-prod-rhel-x86_64-server-6

glibc.i686 2.12-1.80.el6_3.4 @clone-prod-rhel-x86_64-server-6

libgcc.i686 4.4.6-4.el6 @clone-prod-rhel-x86_64-server-6

nss-softokn-freebl.i686 3.12.9-11.el6 @clone-prod-rhel-x86_64-server-6

Joachim

pavel.novotny · ‎2013-06-20

Hello Joachim,

These libraries are installed:

compat-libstdc++-33-3.2.3-69.el6.i686

compat-libstdc++-33-3.2.3-69.el6.x86_64

glibc-2.12-1.107.el6.i686

glibc-2.12-1.107.el6.x86_64

libgcc-4.4.7-3.el6.x86_64

libgcc-4.4.7-3.el6.i686

nss-softokn-freebl-3.12.9-11.el6.i686

nss-softokn-freebl-3.12.9-11.el6.x86_64

Is it ok?

Thanks

Pavel

jpvasgard · ‎2013-06-20

Hi all,

I tested with version 5.2.6.6424.0 and have the same issue:

Connector Setup Wizard starting in mode [console]
[Thu Jun 20 02:22:22 UTC 2013] [INFO ] Checking for a running instance of connector...
[Thu Jun 20 02:22:23 UTC 2013] [INFO ] Starting up connector...
FATAL EXCEPTION:
Could not launch an instance of Connector
FATAL EXCEPTION:
No connector found at the specified port [10001]... exiting
[Thu Jun 20 02:24:24 UTC 2013] [ERROR] An instance of connector was launched, but communication was lost with it.
[Thu Jun 20 02:24:24 UTC 2013] [ERROR] You may need to shut it down manually.
[Thu Jun 20 02:24:24 UTC 2013] [INFO ] Shutting Down Agent Framework Version [5.2.6.6434.0]

Also, directly after this "shutting down" message, I can see the connector running and port 10001/tcp listening for connections.

Can someone with no such issues please provide me a copy of "yum list installed" so I can compare packages?

Cheers

JP

jpvasgard · ‎2013-06-21

ok, some more info. I tried installing on a 32 bit RedHat 6 with all bells and whistles installed (X11, Gnome Desktop, etc.) and I still got stuck using console mode.

If you want to reproduce my issue, do a minimal RH 6 installation, add the required libraries (don't forget the 32 bit versions in case you install a 64 bit RH) and try to install a SmartConnector 6.0.3 in console mode without having done any successful installation already running on the system.

And can someone that does not see the issue please send me or post the output of "yum list installed", so I can compare with my configuration?

Cheers

JP