Issues recovering Data Protector 9 cell manager internal database during disaster recovery rehearsal

We have recently upgraded from Data Protector 7 to 9.

Last week we performed our first disaster recovery rehearsal since upgrading and found the recovery of the DP Internal Database quite a bit different to what we were used to.

Previously you just restored the internal database to a temporary location, stop DP, copied the whole temporary directory contents back to the root directory and restarted DP, job done.

Now you have Restore internal database (with option to "Use the restored database as a new Internal Database"), Restore catalog binary files, restore configuration files.

Following a restore of all of these various parts you end up with a temporary directory(s) with various directories of restored data.

    DataProtector
    idb
    jce
    pg
    var

I could not find a easy to follow instruction of how to quickly restore the last good backup of DP9 to a temporary location and copy the various files back to their appropriate locations so that DP9 is running and configured exactly as it was when it was backed up, which is all we need to be able to do at a disaster recovery.

I experimented with ticking the "Use the restored database as a new Internal Database" option with bad results.  In our environment the whole of DP9 is stored in its own volume group.  We then have symbolic links from /opt/omni to /DataProtector/opt/omni, /var/opt/omni to /DataProtector/var/opt/omni and /etc/opt/omni to /DataProtector/etc/opt/omni.  After recovering with the "Use the restored database as a new Internal Database" option ticked the DP system hung.  On investigation the symbolic link /var/opt/omni had been changed and now pointed to /var/opt/omni - itself!

I did eventually get DP9 restored and working by looking at what had been restored, comparing it to the working system and then copying all the restored files to the appropriate locations.

Why has the recovery of Data Protector internal database been made so unnecessarily complicated?

Is there a set of easily followed instructions for doing a quick recovery of DP9 Internal DB from tape to get the system restored quickly and easily which is all that we need when rehearsing our DR plan.

While typing this I thought I would quickly restore my live DP Internal database to a temporary directory to see if I see the various strange messages I had when doing it a DR.  I get a number of major and minor errors which doesn't fill me with confidence in version 9.  I have attached the sessions messages below.

[Normal] From: RSM@brglb7 ""  Time: 17/11/2016 13:55:28
 Restore session 2016/11/17-74 started.

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 11/17/16 13:55:28
 Starting restore until latest possible time

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:55:29
 Restoring session ID : 2016/11/17 0057

[Normal] From: RMA@brglb7 "HP:C5683A_brglb7"  Time: 17/11/2016 13:56:01
 STARTING Media Agent "HP:C5683A_brglb7"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:56:48
 Starting OB2BAR Restore: brglb7:DatabaseLogs:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:56:54
 Completed OB2BAR Restore: brglb7:DatabaseLogs:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:57:45
 Starting OB2BAR Restore: brglb7:DatabaseFiles:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:57:59
 Completed OB2BAR Restore: brglb7:DatabaseFiles:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:58:01
 Starting recovery of the database at location : /reorg/DP9/pg

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:58:34
 Recovery of the database finished successfully

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:58:34
 Restoring session ID : 2016/11/17 0057

[Normal] From: RMA@brglb7 "HP:C5683A_brglb7"  Time: 17/11/2016 13:58:55
 COMPLETED Media Agent "HP:C5683A_brglb7"

[Normal] From: RMA@brglb7 "HP:C5683A_brglb7"  Time: 17/11/2016 13:59:07
 STARTING Media Agent "HP:C5683A_brglb7"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:59:59
 Starting OB2BAR Restore: brglb7:DPSPECs:0 "IDB"

[Major] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:59:59
 The OS reported error while accessing /reorg/DP9/var/opt/omni/server/db80/meta:
 [249] Too many levels of symbolic links

[Minor] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:59:59
 A non-essential file cannot be restored, restore continues

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 13:59:59
 Completed OB2BAR Restore: brglb7:DPSPECs:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:00:42
 Starting OB2BAR Restore: brglb7:SMBFs:0 "IDB"

[Major] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:00:42
 The OS reported error while accessing /reorg/DP9/var/opt/omni/server/db80/msg:
 [249] Too many levels of symbolic links

[Minor] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:00:42
 A non-essential file cannot be restored, restore continues

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:00:42
 Completed OB2BAR Restore: brglb7:SMBFs:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:01:41
 Starting OB2BAR Restore: brglb7:DCBFs:0 "IDB"

[Major] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:01:41
 The OS reported error while accessing /reorg/DP9/var/opt/omni/server/db40/dcbf:
 [249] Too many levels of symbolic links

[Minor] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:01:41
 A non-essential file cannot be restored, restore continues

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:01:41
 Completed OB2BAR Restore: brglb7:DCBFs:0 "IDB"

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:02:22
 Starting OB2BAR Restore: brglb7:ConfigurationFiles "IDB"

[Major] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:02:22
 The OS reported error while accessing /reorg/DP9/opt/omni:
 [2] No such file or directory

[Normal] From: OB2BAR_POSTGRES_BAR@brglb7 "DPIDB"  Time: 17/11/2016 14:02:22
 Completed OB2BAR Restore: brglb7:ConfigurationFiles "IDB"

[Normal] From: RSM@brglb7 ""  Time: 17/11/2016 14:02:23
 OB2BAR application on "brglb7" disconnected.

[Normal] From: RMA@brglb7 "HP:C5683A_brglb7"  Time: 17/11/2016 14:03:15
 COMPLETED Media Agent "HP:C5683A_brglb7"

[Normal] From: RSM@brglb7 ""  Time: 17/11/2016 14:03:49

 Restore Statistics:
         
  Session Queuing Time (hours)         0.00       
  -------------------------------------------     
  Completed Disk Agents ........          6         
  Failed Disk Agents ...........          0         
  Aborted Disk Agents ..........          0         
  -------------------------------------------     
  Disk Agents Total  ...........          6         
  ===========================================     
  Completed Media Agents .......          2         
  Failed Media Agents ..........          0         
  Aborted Media Agents .........          0         
  -------------------------------------------     
  Media Agents Total  ..........          2         
  ===========================================     
  Mbytes Total .................     214 MB       
  Used Media Total .............          1         
  Disk Agent Errors Total ......          0   


============================================================================
                      Session completed with errors!
============================================================================

Hopefully someone can shed some light on some of this for me.

Thanks
Fergus

Parents Reply Children
No Data