Welcome Serena Central users! CLICK HERE
The migration of the Serena Central community is currently underway. Be sure to read THIS MESSAGE to get your new login set up to access your account.
Jenni Wolgast Absent Member.
Absent Member.
202 views

DP 5.5 "stalls" during backup

I am backing up 2 HP-UX 11.11 servers from a Windows XP cell manager. I run full backups on Saturday nights at 5:30 pm on both servers... The development server has no issues, there are consistant backup speeds throughout the whole session and the total elapsed time is fairly consistant as well. My production server on the other hand seems to start out just fine then take a nap for an hour or 4 hours then kicm back in and resume normal speed until it completes...

I cannot find anything going on on either the Cell Manager or the HP-UX during the times of really, really slow backup speeds... I checked the system performance stats on the HP-UX and it doesn't seem to be getting the same volume of read requests throughout the duration of the backup... It seems to get a bunch for like the first 10 minutes, then they drop off for a while, then they pick back up again until the back finishes.

The length of the stalls is not consistant either, this weekend it was about 4 hours, last weekend it was about an hour and a half, the weekend before was about 5 hours... This started over a month ago and some weekends are better than others. We did not make any changes to any of the servers before this started, it just started happening all of a sudden out of nowhere...

Any ideas?? Let me know if you need more info...
0 Likes
7 Replies
Paul McCleary Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

Hi Jenni,

Are these in separate backup policies or is the CM backing up both servers in the same policy to 1 or more tape drives?

I am guessing that the HP-UX systems backup to the CM over the network, which is directly connected to your drives - correct?

Have you tested the performance to all tape drives from the CM?

Have you tried any tests to backup a subset of the objects from the Prod system to see what speed you get?

Have you tried re-scheduling to a different time?

Have there been any significant changes to the data on the Prod system? Maybe a large increase in number of small files.
0 Likes
Tomislav P. Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

well,... first check debug.log ,...


maybe try with:

OB2SHMIPC=0
OB2IPCKEEPALIVE=1

http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1235557
0 Likes
Jenni Wolgast Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

There are no errors in the debug.log files on either system.

Each HP-UX box has it's own direct connected tape library and I back them up in separate sessions. We actually used to do full backups nightly on both systems. A couple months ago I used to backup this server up in less than 4 hours and select mount points on the dev system in 3 hours so I was kicking off the prodution full at 7:30 pm and the dev full at 8:30 pm. When the fulls on this system started taking more like 5-6 hours I had to switch to incrementals during the week. So basically I was having the same trouble when this system was the only one being backed up for the first hour and having the dev system backup running at the same time does not seem to have any impact on the run time of the prod backup...
0 Likes
Jenni Wolgast Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

Also, the number of files and total amount of data being backed up has not changed significantly... In fact I have been comparing numbers and two weeks ago in a backup that took 7 hours there were fewer files and a smaller total of Gb stored than a backup in April that took under 4 hours..
0 Likes
Paul McCleary Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

Hi Jenni,

Do you have more than 1 drive in this library that is directly connected to your system?
- I'm wondering if you can completely rule out media/drive/cabling issues.

What about using a non-DP method (such as tar) to test the speed you can read from disk and write to tape. There are also some HP tools you can test this with:

http://h18006.www1.hp.com/products/storageworks/ltt/index.html

(Software links at bottom of page)
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50460#hptapeperf

Do you do any monitoring of what is happening on the system during the period of the backup? If so is there any contention for resources?

I would run the session with debug on and see if you can spot anything interesting.
0 Likes
Jenni Wolgast Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

We have ruled out drive/cable/configuration/media issues by trying the backups to a single tape drive that is connected to the same system and we also restore files backed up on production server to the development server every night to do a "data refresh" and the restore times have not changed...

I have a case open with HP support and they had me try running a backup with debug on. I sent them the files but they are claiming this is a performance issue rather than a break/fix issue so they are not responding in a timely manner at all... Is there anything I can look for in the debug files? Also I have Measureware running on the server and it show little to no server activity during these "stalls" in the backup... It seems like Data Protector is not even attempting to read any data because the read requests are very very low and my array even gets quite a bit of optimizing done because it is receiving so few requests...
0 Likes
Paul McCleary Absent Member.
Absent Member.

Re: DP 5.5 "stalls" during backup

I assume you're saying that the backup to the single drive on this system also exhibited the same stall behaviour?

If you're looking at Glance or Measureware during these stalls do you see what the DP or other processes are doing? Are they waiting on anything?

Don't know if you have the disk space and time, but do you know if the restore performance to Prod is as expected?

I would be testing the performance from the disk array to the box and also to the tape drives with the tools I listed and trying non-DP alternatives (ftio, fbackup, tar etc.) to try and isolate where the problem lies.

Maybe your best bet to interpret the debugs yourself would be to collect them from DEV as well so you can compare a good backup against the debugs from the Prod troublesome backup.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.