combellack Absent Member.
Absent Member.
4289 views

Slow backups with BackupExec 2010R3

Using Backup Exec 2010 R3 on a Windows 2008 R2. Target server is Netware 6.5 Sp8 virtual machine running on a VMWare Esxi infrastructure. The server has 4GB RAM.
We are having a problem where our backups have started to slow down to the extent that we cannot backup our 3TB of data (on a single volume) which is on a san storage, over a weekend. Although I do appreciate that as our volume of data to be backed increases, so will the time that it takes to back it up, but recently there has been a noticeable slowing down of the back up rate. As an example, speeds that were once over 1000MB/min are now lower that 700MB/min. This makes a significant difference to the overall backup time.
We also back up this server to an off site TSM library. The owners of which have recently contacted us complaining that the backups have started to hog their system as well!

I've done some checking on the Symantec website and the main thing that I could find was a suggestion to 'SET TCP DELAYED ACKNOWLEDGEMENT=OFF' and 'SET TCP NAGLE ALGORITHM=OFF'. Unfortunately these were already set.

I don't know if it's related but I've had to cancel a Diff which had overrun (although this is becoming the norm now!). It was hogging the processor and slowing down user access. The guilty process was a 'VxNDMP..' file. I cancelled the backup and things got back to normal.

Are there any other standard Netware configuration changes or checks (memory management issues, more RAM, etc) that have been found which could make a difference in my situation? Have we reached a volume size that is to big for NW 6.5 to process at normal speed?
NB Our other main NW 6.5 server, which has 1.25TB of data, is able to backup at a rate of over 900MB/Min.

As ever any help is very much appreciated.
Labels (2)
0 Likes
7 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

Hello,

please simulate a backup with TSATEST.NLM on the netware server. This
should give an impression whether the local disk where the volume
resides is the culprit.
Has the data grown on this volume, are there many more small files now?
If you have another netware server please issue a TESTEST also from
there to see how the backup behaves over the network.
But i am neither used to vmware nor backup exec.

Regards

Burkhard Wiegand
Novell Admin
Debeka-Versicherungen
D-56068 Koblenz
0 Likes
combellack Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

Burkhard

Thanks for getting back to me.

>please simulate a backup with TSATEST.NLM on the netware server. This
>should give an impression whether the local disk where the volume
>resides is the culprit.

I shall do a few tests and let you know the results.


>Has the data grown on this volume, are there many more small files now?

Consistantly over time. It has gone from 1 TB to 3TB over the last couple of years. The file sizes are various, although I guess that most are on the smaller side, only a few hundred KB's to a few MB. Although there is the odd smattering of files up to a few GB's!


>If you have another netware server please issue a TESTEST also from
>there to see how the backup behaves over the network.

I've got a few lined up that I will try.

>But i am neither used to vmware nor backup exec.

Any help is very appreciated.

Regards
Ian.
0 Likes
combellack Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

Burkhard

>please simulate a backup with TSATEST.NLM on the netware server. This
>should give an impression whether the local disk where the volume
>resides is the culprit.

I've done some tsatest's and the results are below.

Backup Exec on the affected server (a virtual machine) reports a backup rate of 627MB/Min. I used a small subset of this job on tsatest and got a rate of 1423MB/Min!

Backing Up :DATA/PROJECTS/
Read Count: 4246006 Min. Read Time: 000us
Last Read Size: 14721 Last Read Time: 000us
Total Bytes Read: 232563055463 x. Read Time: 1810000us
Raw Data MB/min: 1551.23 Avg. Read Time: 2000us
Backup Sets: 663542 Avg. Scan Time: 1100us
Avg. Open Time: 000us Total Read Time: 8649s
Avg. Close Time: 000us Elapsed Time: 9433s
Effective MB/min: 1423.96 Total TSA Time: 9422s
Max. Error: 6.21%
Avg. File Read Time:13000us


LOG:
Open Data Set failed: FFFDFFF5
Open Data Set failed: FFFDFFF5
Open Data Set failed: FFFDFFF5
Open Data Set failed: FFFDFFF5
Open Data Set failed: FFFDFFF5
Open Data Set failed: FFFDFFF5
Open Data Set failed: FFFDFFF5
Disconnected from TSA


<Press any key to close screen>

This test was carried out on a folder that is 230GB in size, it's parent job is 1.3TB in size. I guess that for the sake of completeness I should backup the same folder using Backup Exec so that results can be compared like for like. I'll try and do this afternoon and will post the results afterwards.



I then did a tsatest of a physical server that is connected to the same switch as the backup server. The results were:


The physical server reports that it's data volume (which has many small files) is backed up at a rate of 443MB/Min. Tsatest reports that it backs up at a rate of 854MB/Min.

SMS - TSA Statistical Analyzer
Backing Up :DATA:
Read Count: 972631 Min. Read Time: 000us
Last Read Size: 17772 Last Read Time: 1800us
Total Bytes Read: 48350405194 ax. Read Time: 589100us
Raw Data MB/min: 886.86 Avg. Read Time: 3200us
Backup Sets: 268987 Avg. Scan Time: 300us
Avg. Open Time: 000us Total Read Time: 3145s
Avg. Close Time: 000us Elapsed Time: 3281s
Effective MB/min: 854.38 Total TSA Time: 3264s
Max. Error: 5.17%
Avg. File Read Time:11700us


LOG:
LOGS:
Volume statistics: Total blocks: 0x1ce0a9 Block size : 0x1000
0 - DOS 4 - LONG 1 - MACINTOSH 2 - NFS
Backing up: DATA1:
Data Set Begun
Disconnected from TSA

Volume statistics: Total blocks: 0xe76e46 Block size : 0x1000
0 - DOS 4 - LONG 1 - MACINTOSH 2 - NFS
<Press any key to close screen>

There almost seems to be a fairly obvious conclusion coming here, but I'm not an expert at reading tsatest results, so I'll leave that to better men than I!

Many thanks for the help.
Regards
Ian.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

Hello,

the backup speed of 1423MB/Min for the 230 GB folder would be quite ok.
Meaning that the harddisk subsystem of this server (HD and nss
filesystem) could not go beyond this value with current settings.

> This test was carried out on a folder that is 230GB in size, it's
> parent job is 1.3TB in size. I guess that for the sake of completeness I
> should backup the same folder using Backup Exec so that results can be
> compared like for like. I'll try and do this afternoon and will post the
> results afterwards.


Yes! There may be regions in the volume with many small files where the
opening time of many small files in the sum sow down the backup.

Please try also a tsatest from the physical netware server to that
virtual netware server (see TID 10092890) in order to test the backup
over network (LAN).
Because network may be a reason for speed or not speed; may be settings
on the LAN card of the back-up server or the server who is being backed up.
Example: In our old phsysical netware server [HP Proliant DL 380G4 with
Broadcom-NICs onboard, Q57.LAN, with Arcserve 11.1, speed not beyond 900
MB/min] we replaced them with Intel-NICs, N1000.LAN, and got up to
speeds of 2000 MB/min.

Are any other LAN cards recommended than VMXNet for a virtual netware
server (see vmware docs)?
Are the vmware tools installed on the virtual netware server?
Have you tried TID 10089921 about setting tcp minshall algorithm to on?

Regards

Burkhard


0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

Hello,

TID 3341011 "Slow backup restore performance tuning parameters for NSS
and TSAFS.NLM" gives also some ideas where to look for parameters.

Regards

Burkhard
0 Likes
combellack Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

>Yes! There may be regions in the volume with many small files where the
>opening time of many small files in the sum sow down the backup.

When I ran this as a Backup Exec job on the affected server the job rate was 520MB/Min (tsatest was over 1400MB/Min). The job was cancelled after 2Hrs 30mins (approx the same time as the tsatest ran) and only 80GB of data was backed up (tsatest reported that it read 232GB). This job was run at the same time as the tsatest, just 1 day later. So, it does look like the sms setup on the server is not the fault!

>Please try also a tsatest from the physical netware server to that
>virtual netware server (see TID 10092890) in order to test the backup
>over network (LAN).

I'll try this and see what happens. I'll report the results when I've completed the tests.


I'll check out your other suggestions and let you know what I find.
The server is being rebooted tonight. I'm not sure how much difference this will make. I guess that I'm hoping that it will sort out a memory issue!

Many thanks
Ian.
0 Likes
Lmosla Absent Member.
Absent Member.

Re: Slow backups with BackupExec 2010R3

Hi, Are you updated to the most recent live updates for Backup Exec? It is interesting that you have another server that is able to backup at normal speeds. You also might want to try breaking your job into smaller chunks.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.