Support Tip: Upgrade to 2021.05 causing huge Oracle UNDO TABLESPACE increase

0 Likes

Environment

UCMDB 2021.05 (upgrade from 2018.08) , Oracle database

Situation

A customer upgraded their UCMDB production and pre-production environments from UCMDB Version 2018.08 to version 2021.05 and this led to a rapid and progressive increase in the UNDO TABLESPACE  

This led to a potentially critical situation as the database table kept on growing threatening database failure, running out of disk space and production outage.

The upgrade itself was seemingly successful and UCMDB was able to start with no login issues

Cause


This was not a product issue but a database issue resulting from a product enhancement present in the upgrade.

Just before the upgrade, the UNDO TABLESPACE contained already more than 725 million records.

The upgraded UCMDB contained an enhancement to clean up the records in the database using the query:

delete from ddm_discovery_touch_results_queue where jobid is NULL 

In a clean database only containing a few (hundred) thousand records this would not have caused an issue

However, in their situation the table already contained > 725 Million Records causing the undo operations resulting from these DELETE actions not to managed properly as a result of a database issue on the customer side. 

This was causing the UNDO TABLESPACE progressive increase. Also the fact that this particular customer was using Integration Services rather than probes in one of their environments was aggravating the issue as this service writes to the ddm_discovery_touch_results_queue
 

Resolution

The resolution was on the customer DBA side to resolve the management of the UNDO TABLESPACE for a large number of write operations

Workaround 

The following steps were provided by Micro Focus to work around the issue to prevent a potentially disastrous size increase :

1) Backup the database with a tested rollback plan
2) Stop the UCMDB server
3) Stop all probes and Integration Services (if used) 
4) Clear the UNDO TABLESPACE
5) Truncate table - ddm_discovery_touch_results_queue
6) Restart all probes keeping Integration Services stopped during the DBA investigation into the actual cause
7) Restart the UCMDB server

You can also read the full support tip here.

Tags:

Labels:

Support Tip
Comment List
Anonymous
Related Discussions
Recommended