UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21. Read more.
UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21.Read more.
Absent Member.
Absent Member.
882 views

GW12 SP2 PO crash cleared all user mailboxes, can't rebuild

Having installed and supported Groupwise for a long time -- my first Netware install was 2.15 -- this is an embarrassing problem to admit, but I've put myself squarely into Murphy's cross-hairs, and am hoping someone else can help me recover from my extensive temporal stupidity (a/k/a "I'll fix that next week"). Here's my problem.

I have a GW2012 post office that has experienced PO corruption when the POA was shut down gracefully on three prior occasions, producing an "8101 Memory function failure" or "8101 Error reading time zone record" message, crashing the POA at startup, and requiring a ConsoleOne PO "Rebuild database" to correct. The last time this happened, though, all of the user mailboxes were cleared: no contact, email, calendar, etc., although the message store appears to still have all of the message data intact. I've tried to rebuild the user databases, but have gotten only a few messages and calendar items, all from the last 30 days, a very small portion of the message store: dozens recovered, out of tens of thousands total. Checking the /ofuser folder, all of the USERnnn.db files appear to have been wiped clean for all mailboxes, but the /ofmsg and /offiles contents appear normal size.

The server is GW12.0.2, running on SLES11 SP2 and OES11 SP1. Physical hardware is a six core CPU, 16GB of RAM, and 300GB+ free on the raid'ed NSS volume that holds the ~90GB Groupwise store. Workstations are Win 7 Pro 64-bit, Win 8 Pro, or Win XP, and we see the same info via WebAccess as the direct clients, so this is not a client issue. The PO issues did not exist until a few weeks ago, when we finally got around to retiring our last Netware server and upgrading this box to OES11 SP1, which also required the upgrade to GW 12.0.2 from 12.0.0, and we consolidated two post offices at the same time. Just after the upgrade, the PO crashes started on the server, which behaved itself for a few weeks... until this last crash, which has been the worst of the bunch.

Normally, this would be bad, but that's what backups are for, right? Well... we screwed that up, as this past week we reformatted and reinstalled the Netware box as a new Linux server... the same Netware box that was the target of the Groupwise backups, so backups aren't available. Since we used to backup GW using dbcopy and rsync to the Netware server, we didn't include it in our normal file backups... yet... another fatal flaw in the plan. While the NSS volume on the new server has Salvage enabled, there are no salvageable copies of the USERnnn.db files. As part of the upgrade, we transported some of the affected users from a different PO in the same domain, but have nothing left there that would appear to be useful for recovery.

All I have left is thousands of messages in 90GB of message store which I can't seem to bring back, trying both "Recreate user database" and "Structural rebuild" on the mailboxes from both the C1 and standalone 12.0.2 GWCheck. The GWCheck log message counts and file sizes give every impression that the messages are still present, but I'm failing to recreate the user databases. I recognize that some elements (i.e. rules, contacts?) are probably toast, but being able to recover even the calendars and email messages would be a dramatic improvement.

Am I missing an option here that might bring this data back?

Michael Zore
Sterling Technologies Group, Ltd.
Labels (2)
0 Likes
1 Reply
Cadet 1st Class Cadet 1st Class
Cadet 1st Class

Hi,

It sounds a lot like you are asking if we know how to build a time machine. Or a whiskey still.

Regrettably the practices outlines above have led to a horrific data loss. Without the user databases, its pretty much 90GB of useless files. items smaller than 2KB in size are stored in the user databases, so many of the item types you are looking for would be lost. Some of the larger messages and messages to multiple recipients may still be viable.

I'm really trying to be as respectful and sympathetic as I possibly can be, given the description of your situation. And perhaps you have not had primary responsibility for these systems and have been inducted into the cleanup crew. But this sort of lack of stewardship has been cause for termination. I would pull out the credit card, call up Novell Technical Support and see if you can buy a miracle.

-- Bob
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.