Helpdesk_User Honored Contributor.
Honored Contributor.
2361 views

Slow performance on file system backup even to disk


I have a few jobs which can take days to backup a 100gb drive. Agreed it has alot of files on it but it still doesn't move very quick. I've checked with the network team and they can't see the firewalls being maxed out or anything. The system doesn't seem to be under load (cpu / memory) so I can't see why sometimes this backup can take over 24 hours to complete. Any advice greatly appreciated.

I didn't know if i had sent the file libraries correctly. I have several 2tb drives provisioned from an eva to a server, and i have them running as a file library and not a virtual tape library and its set to use distrubuted file media format. Don't know if this is good practice.

i've copied this from one of the jobs
Run Time ........... 36:44:22
Backup Speed ....... 0.57 MB/s

Directories ........ 16
Regular files ...... 477967
------------------------------
Objects Total ...... 477983
Total Size ......... 73.20 GB

0 Likes
8 Replies
Highlighted
Bob_Clark Absent Member.
Absent Member.

Re: Slow performance on file system backup even to disk

I am attaching a document that discusses various performance problems, and how to deal with them

 

Unfortunately, there is no simple answer for this problem.  I agree, the performance you are seeing is incredibly bad.  Ifyou are running file system backups, you can test by running Disk Agent (VBDA) in stand-alone mode to see if it is a problem with th emedia agent,  then try backing up the same sample to a local 'nul' device to see if it is a network issue, then doing a nomral backup using the same sample.  You need to be able to make an Apples-to-Apples conparison

 

You can try increasing your block size on your devie configuration.  Make sure that, in the Destination tab of your backup specification, concurrency is set to '1'

 

Distributed File Media Format (DFMF) is only used in a very limited setting, and it usually causes more trouble that it si worth (for exampe, you can't delete media), and I recommend against using this option

André Beck
Visitor.

Re: Slow performance on file system backup even to disk

Hi,

 

The system doesn't seem to be under load (cpu / memory) so I can't see why sometimes this backup can take over 24 hours to complete.

 

Well, neither RAM nor CPU utilization are in any way indicative of I/O saturation. You are missing on the most crucial performance metric for the case at hand, the disk duty cycle of the source drive. This metric has been ignored for a long time by most OSes and their monitoring toolkits and only recently more admins became aware of it (I see that repeatedly when being asked for advice on slowness, people look for the bottleneck everywhere except for the source, which, when you think about it, is perplexing - yet I know I was ignorant of that myself just a decade ago). You don't specify the OS the source disk is attached to, giving you hints on measuring disk saturation would require that. The most typical tools I use are atop on Linux (has a great per-blockdevice load display that goes colored when you reach certain thresholds, often that is an admin eye-opener), perfmon on Windows up to 2003 (Time% for the physical drives is even one of the three default graphs) or Ressource Monitor on newer Windows versions (the blue bar on the storage tab).

 

All that being said, your 0,5 million files in 73GB lead to an average file size of 153KB, which isn't exactly the worst case lots-of-small-files situation. It's still bad, and still causes lots of random I/O (how bad exactly it is depends on OS and file system block/cluster size in use), but IMO it's not bad enough to justify 570KB/s average throughput (or maybe it does, but only if the source is a single spindle of some slowly rotating nearline SATA disk that saturates at 90 IOPS like a WD Red - they are great PVR disks, but clearly not made for this use case). So there may be something else going wrong here. First of all, you should measure the throughput the source can actually achieve when reading all the files in depth-first traversal. Bob already mentioned the backup to a local nul device as an option to test that, Daniel describes an even quicker method by just calling the VBDA directly. There are also others. When you know the source performance, you know what the backup could achieve at all - it will only get slower due to other factors, not faster then the source. If you find out your source is blazingly faster reading this heap of files than you actually see in a backup, then things get really interesting.

 

HTH,

Andre.

0 Likes
Helpdesk_User Honored Contributor.
Honored Contributor.

Re: Slow performance on file system backup even to disk

Bob,

 

As always you're the man with a plan, so few questions with what you said. I've added my answer to the end of your point.

 

1.  You said about running Disk Agent (VBDA) in stand-alone mode - how do i do this?

 

2.  then try backing up the same sample to a local 'nul' device to see if it is a network issue - How do i do this?

 

3. Make sure that, in the Destination tab of your backup specification, concurrency is set to '1' - i have each writer set to a concurrency f 1, which is what i think you mean - i do have the load balancing set to min 1 max 4. as we have 8 writers available

 

4. Distributed File Media Format (DFMF) is only used in a very limited setting, and it usually causes more trouble that it si worth (for exampe, you can't delete media), and I recommend against using this option - So if you don't use DFMF what mode do you suggest? do you mean we should be using a VTL?

0 Likes
Helpdesk_User Honored Contributor.
Honored Contributor.

Re: Slow performance on file system backup even to disk

Also i currently have 7 jobs which are using all 8 drive writers. These drives are 2tb luns presented from an EVA to server 1. All of these backups are currently all saving to disks attached to server 1. I have found that programs seem a little buggy on server 1, but the CPU and memory look like there not doing much. also the NIC is barely being used, as its a teamed CNA card, so it should have 20gb available, but its not using anything close to 1gb.

 

Would 8 writers cause the server to go slow? Is there a recomendation abouut how many writers you can have?

0 Likes
Bob_Clark Absent Member.
Absent Member.

Re: Slow performance on file system backup even to disk

I didn't want to go into too much detail unless this was something you wanted to pursue outside of the normal case creation process

 

1. You said about running Disk Agent (VBDA) in stand-alone mode - how do i do this?

I would probably need to see a session report of a slow backup (hopefully with the option 'Display Statistical Info' enabled) and the OS-type and version of what you are backing up.  From this, I would have you run, from the command line of the affected erver:

 

   vbda -vol [volume] -trees [what is being backed up] --out [nul output] -profile

 

So, for example, to check the backup of the \Users folder on the 😧 partition  on a Windows system

 

  vbda -vol /D -trees /Users -out nul -profile

 

This would have the effect of not accessing the IDB or any device

 

2. then try backing up the same sample to a local 'nul' device to see if it is a network issue - How do i do this?

First, this has to be an apples-to-apples comparison, which means you are going to test against the same set of files used in test #1.  In your 'global' file, find this line

 

# FileMediumCapacity=MaxSizeInMBytes

 

Make a copy of thsi line on one of the lines, uncomment it and remove teh 'space that follows, and set the value somewhat over the amount you are testing with.  So, for example, if you are testing with 750MB

 

FileMediumCapacity=800

# FileMediumCapacity=MaxSizeInMBytes

 

Save the file, stop and restart DP

 

ON the Disk Agent you are testing with, create a 'file-type standalone' device in the Drive tab use 'nul' (Windows) or /dev/null (UNIX) as the destination file

 

Create a backup specification, with the option 'Display Statistical Info' enabled, and using the 'nul' device as a destination.  Run your backup, in the session report, compare the Statistics against what you got from backing up to VBDA.  This will write to the IDB, but will not go across the network, and will not use a Backup Device

 

Change the test backup specification to use a Normal writer, run the backup, and, again, display statistical Info, and compare it to the first 2 tests

 

4. Distributed File Media Format (DFMF) is only used in a very limited setting, and it usually causes more trouble that it si worth (for exampe, you can't delete media), and I recommend against using this option - So if you don't use DFMF what mode do you suggest? do you mean we should be using a VTL?

No, DFMF is a checkbox.  YOu either select it to use DFMF, or don't select it, so as not to use DFMF

DFMF is required when doing Advanced Incremental Backups with the intention on converting them to Full Backups. Very few people use this, those who do preferring to create Synthetic Full Backps method

 

If you want to send me a session report, but not expose it to the community, you can send it to me directly at

 

    r.clark@hp.com

0 Likes
Helpdesk_User Honored Contributor.
Honored Contributor.

Re: Slow performance on file system backup even to disk

Bob,

 

Because I already have media in DFMF format, can i just untick and it will eventually recycle the old media and then i wont be using it? or do you have to do something more elaborate to get rid of it.

0 Likes
Bob_Clark Absent Member.
Absent Member.

Re: Slow performance on file system backup even to disk

My feeling is that you would be better off creating a new file library without DFMF enabled, use that for your backups, and let the media in your original library expire.  When it is all gone, the DFMF type file library can be deleted

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Slow performance on file system backup even to disk

Hi.

   In addition to Bob and Andre's troubleshooting steps, you can also consider splitting your backup into multiple objects that group 2-3 directories together. If your directories are large enough, then even 1 directory per object will be viable. This will allow more disk agents to process data in parallel and should help speed up your backups. For more information about the process, please search DP's online help for this topic - 'Backing Up Using Multiple Disk Agents'.

Regards,

Shishir

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.