• State Not Answered
  • Date
  • Date 11 May 2016 2:11
  • Replies 13 replies
  • Subscribers 53 subscribers
  • Views 3782 views

Osagent process quited in AIX

Hi All,

I have started the osagent process in AIX and then my Corba App Server process.  Initially, all server services can be registered and client application can connect via osagent. And the osagent logs records all the remote call service between Client App, Osagent, Server service.

However, recently the osagent process has problems and started to quit frequently. And the osagent logs show the "Process Quiting" message at the end of the log file. I am wondering what makes this happened. Thanks for any advise in advance.

Logs showing this.

==>> Wed May 11 14:36:54 2016, dsaclnt.C, 0, Inf
getProvider() Received <GetProvider> from client at
Host : 10.100.129.182
User : java
PID : -1635202208
CAddr : 10.100.129.182
VPort : 28001
CPort : 28001
Requesting for the following service:
IDL:com.ex..........er.mw.IGeneralHelperCorbaPoaSS:1.0_SET_GENERAL_HELPER1

==>> Wed May 11 14:36:54 2016, dsaclnt.C, 0, Inf
getProvider() Replying that service is located at,
Address : 10.100.61.24
Port : 8010

==>> Wed May 11 14:36:57 2016, dsaclnt.C, 0, Inf
getProvider() Received <GetProvider> from client at
Host : 10.100.129.182
User : java
PID : -1635202208
CAddr : 10.100.129.182
VPort : 28001
CPort : 28001
Requesting for the following service:
IDL:com.........IOperUnitHelperCorbaPoaSS:1.0_SET_OPER_UNIT_HELPER1

==>> Wed May 11 14:36:57 2016, dsaclnt.C, 0, Inf
getProvider() Replying that service is located at,
Address : 10.100.61.24
Port : 8010

==>> Wed May 11 14:37:05 2016, dsa.C, 0, Inf
Process Quiting.

Regards,

Albert

  • When OS Agent exits these logs are not reporting any errors.

    For the crash/exit we can see the agent finished processing a call for getProvider(),

    Requesting for the following service: IDL:com.........IOperUnitHelperCorbaPoaSS:1.0_SET_OPER_UNIT_HELPER1.

    From the log shown requests had been successfully processed previously for this provider. And this last request appears to have been replied to successfully as well.

    When the OS agent crashed at 14:37:05, it was immediately after replying to "getProvider" to

    the agent at "10.100.61.24:8010".

    The logs do not show any error message, but suggest a possible issue with the agent at IP address "10.100.61.24:8010". Please provide the logs for this agent from the time of the crashes?

    Can you also please provide details for the version of  VisiBroker and the platform you are using for this application?

    Are you potentially running anything such as a network security or vulnerability scanner at the time when the crash/exit occurred?

    Cheers,

    -Scott

  • In reply to scott.kay:

    Hi Scott,

    Thank very very much.

    There are 2 env in which osagent are running 61.25 (Prod Supp) and 61.24 (UAT).  In the last log of 61.25, the osagent was quited. However, the osagent 61.24 also seems in problem. I think the 2 osagent talked to each other, and causing this problem.

    The 2 osagent run on the same port 15149, but on different machine (server time of machine is different). The 2 env runs the same server Application and Client, which should depends only on its osagent.

    How can we separate the 2 osagent and block them from communicating.

    They are using the localaddr file as below.

    UAT :

    10.100.61.24  255.255.255.0 10.100.61.255

    Prod Supp

    10.100.61.25 255.255.255.0 10.10.61.255

    UAT osagent log is as below......

    ==>> Tue Sep 27 10:31:51 2016, dsaclnt.C, 0, Inf

    areYouAlive() Received <AreYouAlive> from client at

    Host  : 10.100.61.24

    User  : java

    PID   : 1762978979

    CAddr : 10.100.61.24

    VPort : 8112

    CPort : 8112

    ==>> Tue Sep 27 10:33:01 2016, dsaclnt.C, 0, Inf

    areYouAlive() Received <AreYouAlive> from client at

    Host  : 10.100.61.24

    User  : java

    PID   : 1762929821

    CAddr : 10.100.61.24

    VPort : 8109

    CPort : 8109

    ==>> Tue Sep 27 10:33:06 2016, dsaclnt.C, 0, Inf

    areYouAlive() Received <AreYouAlive> from client at

    Host  : 10.100.61.24

    User  : java

    PID   : 1762933792

    CAddr : 10.100.61.24

    VPort : 8111

    CPort : 8111

    ==>> Tue Sep 27 10:33:08 2016, dsa.C, 0, Inf

    Process Quiting.

    Regards,

    Albert

  • In reply to AlbertTsun:

    Can you also please provide details for the version of  VisiBroker and the platform you are using for this application?

    Are you potentially running anything such as a network security or vulnerability scanner at the time when the crash/exit occurred?

    Have you read the KB on the best practices for running osagent?

    community.microfocus.com/.../13865.guidelines-for-use-of-the-osagent.aspx

    Osagent processes communicate to provide directory and location services, why do you want to stop them from communicating?

    Cheers,

    -Scott

  • In reply to scott.kay:

    Hi Scott,

    Actually, as I said before, there are 2 env (61.25 (Prod Supp) and 61.24 (UAT)), each running the same set of Server application (Java Server process and OSAGENT with different OSAGENT PORT 15159 (UAT) and 15149 (Prod Supp)). However,  when PS osagent has a location request from PS Client UI application, the Prod Supp Osagent may find a service (same name) in UAT Java Server process, or vice versa. This behavior is not desired as application user wants the 2 env working independently.

    Also this may cause either one of 2 OSAGENT quitting suddenly.  As a results, both env cannot bring up the application at the same time. Although each can work properly when only either PS or UAT is startup.

    That's why I am asking if the 2 OSAGENT is not talking to each other, can both env be working properly?

    The Visibroker version is 8.5 running on AIX, the server application is pure Java.

    Thank you very much.

    Regards,

    Albert

  • In reply to AlbertTsun:

    Dear Scott,

    After I checked the UAT and PS osagent log, it was found that the UAT oagent log has a Quitting message, whereas PS osagent and App is still working. Attached are the UAT and PS log for your reference. Thanks.2350.startosa-20161125085605_uat.zip

    UAT logs

    ==>> Fri Nov 25 09:41:16 2016, dsaclnt.C, 0, Inf
    areYouAlive() Received <AreYouAlive> from client at
    Host : 10.100.61.24
    User : java
    PID : -1728309527
    CAddr : 10.100.61.24
    VPort : 8110
    CPort : 8110

    ==>> Fri Nov 25 09:41:29 2016, dsa.C, 0, Inf
    Process Quiting.

  • In reply to AlbertTsun:

    Dear Scott,

    Here is the Prod Supp log.6470.startosa-20160520102202_prodsupp.zip

  • In reply to AlbertTsun:

    Dear All,

    Any system properties flag can be added when startup the osagent, so that more debug lines can be shown when the osagent log show "Process Quiting".

    I want to know what exactly causes the OSAGENT to quit.  It seems that the OSAGENT was not killed by any system scanning services.

    Thank you very much in advance.

    Regards,

    Albert

  • In reply to AlbertTsun:

    osagent debug logs can be enabled by using the options below:

    osagent +l oa -d <log_directory>

    When a client application needs to talk to a particular osagent, then it has to set "vbroker.agent.port" property with the respective osagent port.

    For example, PS Client UI should set "vbroker.agent.port" property to 15149. This ensures that PS Client UI will always talks to PS Osagent. The same property with the same port should be configured in the PS Server Application as well so that all three of them forms a domain.

    The other option is to set OSAGENT_PORT environment variable with the respective osagent port in the console window where the Server and Client applications are started. But, in your case, your client seems to be a GUI application so you have to use the "vbroker.agent.port" property and set it to 15149 for PS. For UAT environment, the UAT Server and Client UI applications should set "vbroker.agent.port" property to 15159.

    Cheers,

    Karthi.

  • In reply to Karthikeyan:

    Dear Karthi,

    Thanks. In my UAT env, I use "-DORBagentPort=15159" as  a parameter to launch my java Server and Client apps. Is it OK?

    Cheers,

    Albert

  • In reply to AlbertTsun:

    Hi Albert,

    The ORBagentPort is the older way of configuring the osagent port. I recommend to use "-Dvbroker.agent.port=15159" instead in your UAT Server and Client UI applications. Use the same property in PS as well.

    Cheers,

    Karthi.

  • In reply to Karthikeyan:

    Hi Karthi,

    My osagent has quited again. I have added more debug flag as you told. Please help to check any hints why osagent quited. Thank you very much.

    Cheers, Albertstartosa-20170131085023.zip

  • In reply to AlbertTsun:

    From the log file provided you can see the following:

    ==>> Tue Jan 31 13:50:31 2017, dsachdlr.C, 0, Dbg

    timerExpired() Prepare for next firing at 120 seconds interval.

    ==>> Tue Jan 31 13:50:56 2017, dsa.C, 0, Dbg

    signalHandler() Received signal <1>.

    ==>> Tue Jan 31 13:50:56 2017, dsaclnt.C, 0, Dbg

    close() Sending <Close> request to client <10.100.61.24, 8109>

    ==>> Tue Jan 31 13:50:56 2017, dsaclnt.C, 0, Dbg

    close() Sending <Close> request to client <10.100.61.24, 8111>

    ==>> Tue Jan 31 13:50:56 2017, dsaclnt.C, 0, Dbg

    close() Sending <Close> request to client <10.100.61.24, 8110>

    ==>> Tue Jan 31 13:50:56 2017, dsaclnt.C, 0, Dbg

    close() Sending <Close> request to client <10.100.129.182, 28001>

    ==>> Tue Jan 31 13:50:56 2017, dsaclnt.C, 0, Dbg

    close() Sending <Close> request to client <10.100.61.24, 8112>

    ==>> Tue Jan 31 13:50:56 2017, dsahdlr.C, 0, Dbg

    close() Sending <OrderlyRelease>.

    ==>> Tue Jan 31 13:50:56 2017, dsahdlr.C, 0, Dbg

    _sendToAllAddr() Broadcast is now 1. Sending message to the following agent addresses:

    <10.100.61.24, 15159>

    ==>> Tue Jan 31 13:50:56 2017, ncudpstr.C, 0, Dbg

    _broadcast() Broadcast on <10.100.61.255, 15159>.

    ==>> Tue Jan 31 13:50:56 2017, dsa.C, 0, Inf

    Process Quiting.

    So at 

    ==>> Tue Jan 31 13:50:56 2017, dsa.C, 0, Dbg

    osagent received:

    signalHandler() Received signal <1>.

    Which is a SIGHUP

    Signal NameNumberDescription
    SIGHUP 1 Hangup (POSIX)

    Or a hangup signal from another process. 

    Daemon programs sometimes use SIGHUP as a signal to restart themselves, the most common reason for this being to re-read a configuration file that has been changed. 

    You can also run the osagent using nohup in order to ignore the hangup signal. Read the following for more background:

    http://unix.stackexchange.com/questions/137759/why-use-nohup-rather-than-exec

    Cheers and best regards,

    -Scott Kay

  • In reply to scott.kay:

    Hi Scott Kay,

    The startup shell scrip already use nohup. Why does it still receive signal <1> ? Thanks.

    nohup /opt/MicroFocus/VisiBroker/bin/osagent -v -p $OSAPORT +loa -d > $SET_LOG_PATH/$OSAGENT_LOG 2>&1 &

    Cheers,

    Albert