SCSMTP stopping - not restarting - advice?

[Beth Tuttle]

Put in a case with HP today...

Attached you will find todays sm.log, scsmtp.log, scsmtp.ini, sm.ini, and sm.cfg.

 

At 10:00:12 yesterday, scsmtp.exe got an error and was unable to recover.  The log shows:

09/21/2010 10:00:00   pid (5864)(2068) Sent email <evsysseq=0000000000305784> to <Tim.Beatty@nscorp.com>

09/21/2010 10:00:12   pid (5864)(2068) Error connecting to SMTP server <mailhub.nscorp.com> on port <25>

09/21/2010 10:00:12   pid (5864)(2068) Error: 102 - Hostname lookup failed

09/21/2010 10:00:12   pid (5864)(2068) Network error while sending email <evsysseq=0000000000305785> to <Kimmy.Solomon@nscorp.com>

09/21/2010 10:00:12   pid (5864)(2068) Terminating....

09/21/2010 10:00:12   pid (5864)(2068) Terminated.

 

We have set the scsmtp.ini as sc_reconnect:0 since the comments in scsmtp.ini state this value should be 0 if starting scsmtp.exe from sm.cfg.

 

"#sc_reconnect

1 = Enable reconnect to SC if the connection is lost

0 = Disabled (default) Enabling this option is NOT recommended if SCSMTP is started from the SC.CFG file because SCSMTP will go into a reconnect loop when ServiceCenter is shutdown and the connection is lost. When ServiceCenter is restarted, you would have multiple SCSMTP processes running."

 

Our options at this point are few. 

a. We can cycle the SM service.  This requires an outage for all SM users.

b. We can cycle the SM server.   This requires an outage for all SM users.

c. We can log into the server and from the command line start scsmtp.exe.  However, we cannot log out.

 

Problems/issues:

1. I need a way to start scsmtp without cycling anything. 

2.Is there a better way to start scsmtp than from sm.cfg?  Can it be started like other internal tasks to sm (like inactivity timer)?

3. Request an enhancement for scsmtp to wait/retry after an error before terminating.

 Anyone have experience with this issue?

[Jacob Heubner]

Well, if you have the "us.launch.external.NEW" RAD application, you can get a little tricky and start up scsmtp from inside Service Manager.

I found this works well on any node set to debug, because then you know what server you're connecting to and you can set up the other parts you need to make this work.

So, what I did is create a batch file on the C: drive of the primary app server for Service Manager.  I called the file scsmtp.bat, and it contained the following:

----

e:

cd e:\Program Files\HP\Service Manager 7.11\Server\scsmtp

scsmtp.exe

----

Then, using the us.launch.external.NEW RAD application, you can create a wizard/script that calls the RAD application with the following:

$name="C:\\scsmtp.bat"

$boolean1=val("false", 4)

RAD Application: us.launch.external.NEW

Parameter Names | Parameter Values

name | $name

boolean1 | $boolean1

Then, whenever your SCEMAIL user is not connected, run the script/wizard to start it back up.

If you want to get _really_ fancy, you could build something that periodically checks to see if SCEMAIL is connected, and, if it isn't, trigger the script automatically, but I don't know if you want to go that far.

EDIT:  had a typo in the bat file... changed scsmpt.exe to scsmtp.exe

[Beth Tuttle]

Run that way, do you see it in Windows Services or in HPSM status?

[Jacob Heubner]

Both.

[Jacob Heubner]

So here's what I did for fun... I wrote a javascript which will check the running Processes to see if SCEmail is logged in, and, if it isn't will run that batch job to start the process.  I haven't tested this in Production or a horizontally scaled environment, but it works in my Dev environment.

Add the following to your ScriptLibrary

ScriptName: Processes

Script:

function CheckOn(name){
 var Processes = system.functions.processes("user")
 for (i=0;i<system.functions.lng(Processes);i++){
  var A = system.functions.str(Processes[i])
  var A = system.functions.strrep(A, "{", "")
  var A = system.functions.strrep(A, "}", "")
  var A = system.functions.strrep(A, "[", "")
  var A = system.functions.strrep(A, "]", "")
  var B = new Array
  B = A.split(",")
  var C=B[3].replace(/\"/g,"")
  if (C==" "+name){
   return true
  }
 }
 return false
}

function SCEmail(){
 var IsOn = system.library.Processes.CheckOn("SCEMAIL")
 if (IsOn != true){
  var RetCode
  var File = "C:\\scsmtp.bat"
  system.functions.rtecall("callrad", RetCode, "us.launch.external.NEW", ["name", "boolean1"], [File, false], false)
 }
 return
}

Then, create a schedule record of class "problem" like the following:

Name: Restart SCSMTP

Class: problem

Description-

Runs a Javascript to check on the SCEMAIL process and restarts it if stopped.

Repeat Interval -

Other: 00:05:00

Javascript -

system.library.Processes.SCEmail()

Every 5 minutes, the job will run, check the proceses, and run the batch file to start the SCEmail connection.

[Beth Tuttle]

I am currently 7.10app/7.11.228rte and do not have us.launch.external.NEW.

Do I have to ask support for us.launch.external.NEW? Or does it come with 7.11app or 9.20app?

[Jacob Heubner]

Don't know if it comes with a particular version - I had to ask for it. 

[Beth Tuttle]

Requested enhancement:  http://support.openview.hp.com/selfsolve/document/KM1030338?searchIdentifier=42167dad%3a12e2de6e8bc%3a20bc&resultType=document&documentURL=KM1030338&resultsURL=%2fselfsolve%2fdocuments&allowReturn=true

[-m-]

Consider changing the way scsmtp is started in order to take advantage of the sc_reconnect parameter.

Running transient commands (e.g. processes() function or sm -reportlocks) on a regular basis is not recommended in a horizontally-scaled environment. These commands create a significant load on the Jgroups communication (information passed between servlets). This can lead to performance issues, hanging problems, and might expose software defects depending on the version of Service Manager.

KB Article with download links for us.launch.external.NEW:

http://support.openview.hp.com/selfsolve/document/KM549139

[Beth Tuttle]

Is us.launch.external.NEW supported in 9.20/9.21?  I'd hate to get dependant on it just to have it yanked out.

[Kristen Branson]

I am working on adding this to our Service Manager implemenation and am wondering...

What are the reasons behing pointing the Javascript to the bat file to start scsmtp.exe instead of diirectly to the scsmtp.exe in the RUN directory? 

[Kristen Branson]

I've got this working so a schedule runs every 30 minutes to check for SCEMAIL in System Status.  If it's not found, it uses the us.launch.external.NEW to restart scsmtp.exe.  It works great.

Now I have the need to stop/start SCEMAIL during time change.  I can't figure out how to terminate the SCEMAIL process from System Status if it is running.  Any suggestions?