DP 9.06 Object Copy Failure from Tape to Catalyst Store
Got a doozie of a problem, hoping someone out there can help, please bear with me, it's not a short query!
So our process of backing up catalyst data is:
- We use DP Cell 1 in Site 1 to run Backups to D2D Catalyst Stores (SQL, Oracle etc..)
- We use DP Cell 1 Object Copy to Foreign Cell 2 to replicate data to D2D in Site 2, and import session in to Foreign Cell 2
- We then run manual object copy jobs of the replication sessions in DP Cell 2 to copy the data to tape
So all that works, no problems, everyone is happy........except......we have just attempted to copy data back from tape to D2D in Site 2 for recovery, and this does not work!!!
To get it back from tape we are using the Interactive Object Copy option again, selecting the original Object Copy session as input (point 3. above), and trying to write it to another D2D Catalyst Store we specifically have for restores, on the same D2D which held the original backups. It is this recovery process which is failing.
We are getting "Session Aborted" errors every time, "Bad Table Descriptor" in the pop up when in monitor of the job. The copy starts, loads the tape ok, attempts the restores, data counters go up, but within a min or two it fails with the above messages. We've attempted this many times, against many tapes, for many different orginal Catalyst Items, across multiple Stores, all exhibit the same error. When we look at the failing Object Copy recovery jobs in the Internal Database it does not even show any error messages at all, it just ends with a "CSM@host" message showing the Target-side Deduplication Statistics for one of the files we are restoring and it just stops! It's as though it's a panic of the job, it can't write out messages, so just stops half way through.
We have run it in debug mode, which we cannot upload (sorry!), but do see in the BMA debug log 52 occurances of "Bad file descriptor", but we can't work out what the real error is. Any ideas of what to search through in the debug logs?
So the Object Copy to tape from Catalyst Store works a treat, but Object Copy from tape to Catalyst Store does not. We cannot get our data back! Any help or advice gratefully received.
Apologies again for not being able to load up whole debug logs, customer doesn't allow it, our hands are tied. But if you can give me a word or phrase to look out for I can retype it up here.. We will raise a call with HP as well, but thought we would post here to see if anyone has any ideas.
when I tested T2D copies from physical tape to StoreOnce (Software and Catalyst), I ran into a set of problems, however they resulted in different messages than yours. They could all be tracked to SO B2D gateway devices not liking to be fed with multiplexed data streams, something that would happen when copying multiplexed tape sessions back to B2D gateways with insufficient allowed max number of devices in the destination tab. Resulting errors are related to the JSONizer, not bad file or table descriptors, though. I was able to avoid them for T2D of file backups by supplying the correct "Max devices", but failed to do the same for BAR backup copies (MSSQL in particular). That all was on 9.04, I gave up on T2D as part of my regular backup flow soon after so no idea if that is still a problem today.
Many thanks fpr your response. You've pretty much explained what we have now found out ourselves after a few days of testing!
The only way we could restore data from tape back to catalyst store was to give it more destination gateways. We proved this by initially just sending the copy to a few different catalyst stores, which thus used more gateways, and hey presto, backups restored.
So in our testing, Exchange backups reported the "Bad table descriptor 12:1004" error, and Oracle/SQL/Filesystems which were reporting JSONizer issues, with "Not enough devices to copy dependant streams" errors after, were all fixed by simply giving them more destination gateways. We've managed to get all data types back now without errors from tape.
so to us this is a logive issue of how we are using DP, and how DP works, i.e. illogical!
So now we are playing with setting up additional "logical" gateways on a single physical host to hopefully give us enough gateways to get the data off tape without adding extra hosts, allowing us to write to one catalyst store.
I have a question regarding this. If i have a database (sap for oracle) that was backed up using 5 parallel streams on store once 1 in site A, then data was replicated to store once in site B, then from site B data was copied to tape (on only one media). If i want to restore this db from tape do i need first to create an object copy to store once (considering data is multiplexed on tape and most likely i will need all 5 streams in the same time when executing the restore)?
If so, can you advise how to create more "logical" gateways on a single physical host?