UCMDB 2021.05 (upgrade from 2018.08) , Oracle database
A customer upgraded their UCMDB production and pre-production environments from UCMDB Version 2018.08 to version 2021.05 and this led to a rapid and progressive increase in the UNDO TABLESPACE
This led to a potentially critical situation as the database table kept on growing threatening database failure, running out of disk space and production outage.
The upgrade itself was seemingly successful and UCMDB was able to start with no login issues
This was not a product issue but a database issue resulting from a product enhancement present in the upgrade.
Just before the upgrade, the UNDO TABLESPACE contained already more than 725 million records.
The upgraded UCMDB contained an enhancement to clean up the records in the database using the query:
delete from ddm_discovery_touch_results_queue where jobid is NULL
In a clean database only containing a few (hundred) thousand records this would not have caused an issue
However, in their situation the table already contained > 725 Million Records causing the undo operations resulting from these DELETE actions not to managed properly as a result of a database issue on the customer side.
This was causing the UNDO TABLESPACE progressive increase. Also the fact that this particular customer was using Integration Services rather than probes in one of their environments was aggravating the issue as this service writes to the ddm_discovery_touch_results_queue
The resolution was on the customer DBA side to resolve the management of the UNDO TABLESPACE for a large number of write operations
The following steps were provided by Micro Focus to work around the issue to prevent a potentially disastrous size increase :
1) Backup the database with a tested rollback plan
2) Stop the UCMDB server
3) Stop all probes and Integration Services (if used)
4) Clear the UNDO TABLESPACE
5) Truncate table - ddm_discovery_touch_results_queue
6) Restart all probes keeping Integration Services stopped during the DBA investigation into the actual cause
7) Restart the UCMDB server
You can also read the full support tip here.