Exchange 2013 Backup slow....

Hi,

 

I am running DP 8.13, and last week on Thursday installed the "latest" patches (released in Dec 2014). Since then I am experiencing very slow performance in my exchange 2013 backups. Earlier the backup used to take 2-3 hours depending on number of logs (backup size is about 600 GB). But since Thursday it now takes more than 5 hours. Has anyone else experienced this?

 

CM is Windows 2008 R2

Exchange 2013 runs on Windows 2012 with latest CU,

backup is done over SAN,

nothing has changed in config...only change is the new patches....

 

here is a short snapshot of omnirpt -list_sessions...some columns omitted for better visibility....

 

Specification         Duration  GB Written # Files Success
________________________________________________
E2010 Exchange_2013       2:06      583,73    5659    100%
E2010 Exchange_2013       3:27      596,51   18352    100%
E2010 Exchange_2013       2:49      598,53   19399    100%
E2010 Exchange_2013       3:25      598,28   18755    100%

         ----------> Patch Installed <----------
E2010 Exchange_2013       5:15      598,71   18556    100%
E2010 Exchange_2013       5:48      598,19   17513    100%
E2010 Exchange_2013       4:48      586,03    5067    100%

 

 

I will open a case with HP, but wanted to check if anyone else also saw similar issues and how he/she resolved it.

 

Thanks and Regards,

Nikhil

Parents
  • I think they modified the way the DAG is queried and how the Logs are checked.

    Suggest to check the Time from DAG-Analysis till Backup starts and the Time the Log-Checks take.

    Als try if disabling the Log-Check helps speeding things up. Had to do that on E2010 to get any Backup done within Timeout periods.

  • Thanks for the reply. But I don't think it is related to log checking, as in our environment, where there are just 8 DBs and corresponding log files, the checks do not take more than a minutes to complete

     

    omnidb -sess 2015/01/28-1128  -report | grep -i seconds
        E00     Duration: 36,11 seconds
        E01     Duration: 45,75 seconds
        E03     Duration: 16,47 seconds
        E06     Duration: 16,78 seconds
        E04     Duration: 17,72 seconds
        E07     Duration: 15,39 seconds
        E05     Duration: 14,00 seconds
        E08     Duration: 16,39 seconds        

    omnidb -sess 2015/01/30-1211  -report | grep -i seconds
        E00     Duration: 31,34 seconds
        E01     Duration: 34,72 seconds
        E03     Duration: 14,80 seconds
        E06     Duration: 23,09 seconds
        E04     Duration: 18,05 seconds
        E05     Duration: 15,41 seconds
        E07     Duration: 26,94 seconds
        E08     Duration: 17,28 seconds

     

    However if I really see the backup speed (I've "display statistical info" enabled in the options), I see that speed has dropped to nearly half.

     omnidb -sess 2015/01/28-1128  -report | grep -i speed | grep -v 0.00
         Backup Speed ....... 9.45 MB/s
         Backup Speed ....... 8.75 MB/s
         Backup Speed ....... 9.34 MB/s
         Backup Speed ....... 13.47 MB/s
         Backup Speed ....... 0.83 MB/s
         Backup Speed ....... 1.67 MB/s
         Backup Speed ....... 1.08 MB/s
         Backup Speed ....... 1.32 MB/s
         Backup Speed ....... 8.58 MB/s
         Backup Speed ....... 8.88 MB/s
         Backup Speed ....... 8.61 MB/s
         Backup Speed ....... 8.60 MB/s
         Backup Speed ....... 5.42 MB/s
         Backup Speed ....... 6.33 MB/s
         Backup Speed ....... 6.56 MB/s
         Backup Speed ....... 9.12 MB/s
        
     omnidb -sess 2015/01/30-1211  -report | grep -i speed | grep -v 0.00
         Backup Speed ....... 4.05 MB/s
         Backup Speed ....... 3.74 MB/s
         Backup Speed ....... 3.89 MB/s
         Backup Speed ....... 3.24 MB/s
         Backup Speed ....... 4.07 MB/s
         Backup Speed ....... 1.83 MB/s
         Backup Speed ....... 3.16 MB/s
         Backup Speed ....... 2.04 MB/s
         Backup Speed ....... 3.85 MB/s
         Backup Speed ....... 3.91 MB/s
         Backup Speed ....... 3.96 MB/s
         Backup Speed ....... 3.97 MB/s
         Backup Speed ....... 3.00 MB/s
         Backup Speed ....... 4.18 MB/s
         Backup Speed ....... 3.64 MB/s
         Backup Speed ....... 3.78 MB/s

    The patches contained the MA and Core. Both are installed on the exchange nodes. Strangely the FS backups of C drive are not affected, it's just the Exchange integration which is suffering from performance.

  • Digging further on before/after patch speeds...the consistency check takes almost same, main difference is in the MA....actual writing on data on tape:

     


     

     

    Before Patch:

    Start of backup -     
    "MS Exchange 2010 Server"  Time: 28.01.2015 23:15:16
        Analyzing MS Exchange Server 2010 environment.

    Start of MA -
    Time: 28.01.2015 23:17:56
        STARTING Media Agent "XXX"


    Completed MA -
    Time: 29.01.2015 02:40:25
        COMPLETED Media Agent "XXX"

     

    Mbytes Total .................  613381 MB       

     


        
    After Patch:

    Start of Backup -
    "MS Exchange 2010 Server"  Time: 30.01.2015 23:15:19
        Analyzing MS Exchange Server 2010 environment.

    Start of MA-
    Time: 30.01.2015 23:17:55
        STARTING Media Agent "XXX"

    Completed MA -
    Time: 31.01.2015 05:03:37
        COMPLETED Media Agent "XXX"

     

    Mbytes Total .................  613289 MB       

  • You would be the first to say Backup of SystemDrive on w2k8 onwards is ok, compared to w2k3 it got painfully slow :-(

    Have you tried a Backup to Null-Device to check if your Disks or the VSS-Snapshot are the limitation?

    My currently inBuilt Prod Environment runs with 10-25MB/s per DB. But only have around 35GB in 20DBs so far.

Reply Children
  • well, system backup being ok was relative....when compared to Exchange 2013 it was fast....:)

     

    So, since then I've reinstalled the agent, exported/imported the clients and re-configured the FC paths of the drives....and viola....the backup ran fine last night !! 600 GB in 2 hrs including full consistency checks...good enough for me.

     

    One thing I noted was the drive addresses were changed from scsix:x:x:x to TapeX:X:X:X

     

    I am not sure if this was due to some OS patch, or the DP MA patch, but I am happy now !!

     

    Thanks and Regards,

  • with scsi-Addresses DP talks directly to the Hardware, with the Tape-Address it goes through the Windows-Driver.

    Probably somebody installed new Drivers in the meantime.

  • Update after long time....with a good news :)

    On both the exchange servers I reinstalled the DP patches, and the duration is back to normal again.

    I also re-scanned all the paths of the devices and found that all the drives are using windows drivers now.

     

    But all is ok now, so not digging further...

     

    thanks and regards,