Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Honored Contributor.. Alpha_1_1 Honored Contributor..
Honored Contributor..
5439 views

Sessions hanging or failing to complete in DP 9.01

Jump to solution

Hi,

 

I've recently upgraded my environment to 9.01 on a W2K8 R2 cluster, and I've begun to notice some unusual behaviour.

 

We have an issue where a job seems to hang, yet the session drops out of the monitor context as if it has completed and is thus impossible to abort.

 

The sessions list in the Internal Database tab shows the job as running, however there are no corresponding BSM or CSM processes to be found on the CM's Task Manager.

 

The session logs of the affected jobs show no errors, they simply tail off, and the job itself doesn't time out after the usual interval.

 

The command omnistat only shows those sessions that are visible in Monitor.

 

Debug.log shows repeat entries of this message:

 

02/01/2015 13:29:34  RMA-NET.9060.4836 ["/ma/xma/rmanet.c $Rev: 41801 $ $Date:: 2014-03-26 16:29:59":161] A.09.00 b87
DbgStack: DbgFcnOut() and level already 0!
    ExitFcn="MaProtocolCheck"

 

If anyone has seen this behaviour and has managed to get past it, I'd appreciate any advice they might have.

 

Thanks,

 

Bob

 

DP 9.01 on W2K8 Enterprise (with DPWIN_00711,712)

CM has 32GB/4*Quad-core CPU/10Gb Ethernet/4Gb SAN connectivity (3PAR storage)

GTS I&O - "When the job's too big for S.H.I.E.L.D...."
0 Likes
1 Solution

Accepted Solutions
Highlighted
Bob_Clark Absent Member.
Absent Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

If your environment is mostly Windows, you are missing 2 lines in your 'omnirc' file

 

OB2IPCKEEPALIVETIME=900

OB2IPCKEEPALIVEINTERVAL=60

 

These used to have to be set in the Registry, but, a few versions ago, they were able to be added to the 'omnirc' file

 

If you do add these lines, don't forget to stop and restart the DP INET service on the machine where you edited the file

16 Replies
Bob_Clark Absent Member.
Absent Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Can you be more specific on the 'Session seems to hang' behavior?  Do you see session statistics at the end of the backup?  Are there any hanging agent processes (Disk agent, media agent, OB2BAR) processes still running on the client machines?

 

What you are describing sound to me like you need KeepALive implemented on the Dsk Agent and Media Agent machines, just based on the symptoms.  If you do a search on the Forum for KeepAlive, you will find 3 articles I wrote (HPUX< Linux, Windows) on how to implement this, or I can send you copies as forum attachments. I would need to know the OS-types of the servers involved.

 

 This is probably the least invasive possible solution to this problem

0 Likes
Honored Contributor.. Alpha_1_1 Honored Contributor..
Honored Contributor..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Hi Bob,

 

Thanks for responding.

 

In answer to your question, it appears as though sessions simply tail off - no statistical summary or normal end-of-job detail, although now I think about it, tape devices appear to unload without difficulty since we've not seen any 'device in use' errors on subsequent jobs.

 

No outstanding processes of any kind on CM or clients, and no error message patterns to suggest a root cause. We do have an isolated case involving a single client whose job would hang on encountering an I/O error, but I've 'quarantined' that client so that it doesn't appear in any scripts other than its own.

 

I have the following parameters set in omnirc (since March 2009):

 

OB2SHMIPC=0

OB2IPCKEEPALIVE=1

OB2INETTIMEOUT=150

 

I'm not seeing any errors that would suggest a loss of connectivity or timeout, and the effect on backups is to show them as still running, while in the case of copy operations, virtual media are not recycled when a job fails to complete.

 

It's more of an annoyance than anything else - it's not happening every night, nor in the same job each time, but it's adding extra housekeeping I could do without :(

 

I'll have a look for your posts on Keepalive - perhaps there's something I've misssed...

 

Cheers,

 

Bob

 

My environment consists mostly of W2K8 systems, with small numbers of W2K3, W2012 and a couple of Linux and OpenVMS clients.

 

 

GTS I&O - "When the job's too big for S.H.I.E.L.D...."
0 Likes
Highlighted
Bob_Clark Absent Member.
Absent Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

If your environment is mostly Windows, you are missing 2 lines in your 'omnirc' file

 

OB2IPCKEEPALIVETIME=900

OB2IPCKEEPALIVEINTERVAL=60

 

These used to have to be set in the Registry, but, a few versions ago, they were able to be added to the 'omnirc' file

 

If you do add these lines, don't forget to stop and restart the DP INET service on the machine where you edited the file

Honored Contributor.. Alpha_1_1 Honored Contributor..
Honored Contributor..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Thanks Bob - I'll add those elements and deploy a new omnirc to my clients.

 

It'll take about a week to get it out to all 250+ systems, but I'll keep tabs on performance as I go...

GTS I&O - "When the job's too big for S.H.I.E.L.D...."
0 Likes
Absent Member.. svollrat Absent Member..
Absent Member..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Suggest to use "util_cmd -setomnirc <hostname> <name> <value>" to set the omnirc-Values, much quicker than logging onto every server and pretty simple to script.

Bob_Clark Absent Member.
Absent Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

No kidding... this is the first I have heard of this.  It is not documented in the CLI guide for DP 7 or 8, so, it must be new for DP 9

 

Have you used it to see if it really works? I am trying to find a DP 9 machine to try it in myself.  If it is as advertised, this is going to be of great help in large installations

0 Likes
Absent Member.. svollrat Absent Member..
Absent Member..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

It's new with DP9 and descirbed in CLI Guide

Tried it on around a dozend Win and Linux Systems, no issues so far.

0 Likes
Bob_Clark Absent Member.
Absent Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

On Windows systems, when editing the 'omnirc' file, you usually have to stop/start the DP INET service to make the change effective.  Do you still have to do that with the new functionality?

0 Likes
Absent Member.. svollrat Absent Member..
Absent Member..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Most times I don't restart the Inet on Windows and the omirc-Values are recognized anyway. Only do that when setting the IPC-Timeouts that affect the already running Service, not for processes that are dynamically loaded like vbda or sql_bar.

Haven't tried setting those via the command so far, or not checked if they are enforced. Sorry :-/

0 Likes
Honored Contributor.. Alpha_1_1 Honored Contributor..
Honored Contributor..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

That appears to have done the trick, Bob - I've had no further instances since applying the changes.

 

Thanks for the advice and assistance - much obliged!

 

 

GTS I&O - "When the job's too big for S.H.I.E.L.D...."
0 Likes
Honored Contributor.. Alpha_1_1 Honored Contributor..
Honored Contributor..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Hi Svollrat,

 

Thanks - I tried your CLI command from my CM and successfully changed a setting in omnirc on one of my MAs.

 

That'll be very useful in the future - cheers!

 

Bob

GTS I&O - "When the job's too big for S.H.I.E.L.D...."
0 Likes
TurboT Absent Member.
Absent Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Hi Bob.

 

i have looked at your atricale and was hoping that this would be the same for me however i already have the keep alive allready confugured.   i have a backup job that hangs to.   everything just tails off and i have to abort it.  there are processes still running and i have to taskkill them.  its really annoying at this only happens for one backup job,  it started intermittantly but seems to be almost every day now and is becomming a payne.

 

there are no issues with tapes or connections.  windows reports no error only that cell manager has hung if you could shine some light on this it would be great.

 

many thanks in advnace 

T

0 Likes
Honored Contributor.. Alpha_1_1 Honored Contributor..
Honored Contributor..

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

Hi T,

Since this happened over two years ago I've applied numerous updates to my environment (I'm at 9.08 now) so I'm not certain I can offer much in the way of advice; however if you have a support contract then I'd suggest opening a case with Data Protector Support - they're great guys and they'll help you diagnose what's happening pretty quickly.

Best of luck,

Bob

 

GTS I&O - "When the job's too big for S.H.I.E.L.D...."
0 Likes
DAMU
New Member.

Re: Sessions hanging or failing to complete in DP 9.01

Jump to solution

I have issues in DP 9.08 BACKUP, HANGING INSTANTLY WHILE RUNNING VM BACKUPS....

 

Any commands to fix the issue

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.