OBM Upgrade to 2020.05 Issues - Postgres Database Empty

[David Gerrish]

I have been trying to upgrade from 2019.05 IP1 to 2020.05 on linux and have had a ton of issues.

The documentation appears to be massively lacking.  Posting some findings here to help others in future.

Current state = upgraded OBM but the database appears "empty" - ie no historical alerts, connected servers has been wiped, no monitored servers or anything...

The upgrade seems to do a backup but then doesn't restore using that backup - is there a vital step I have missed?

Previous errors we had during upgrade:

11/23/20 10:40:55 ERROR: (./upgrade.sh) Backup failed. Please delete all files in /var/opt/OV/backup/ and re run this tool.

MF support said change the postgresql.conf setting to 6144 so trying that – seems to me the upgrade was changing the file, so be careful if you comment out the original entry as the script seemed to be making dynamic changes to the file (poorly I might add).

Also MF support said to change pg_hba.conf (but why?! Undocumented!)

Added:

local all postgres trust

Stop OBM / stop postgres / startup again…

Upgrade command again… then it worked!  But why is this not documented or made clear?

But then OBM would not start...

Then it wasn't clear if we should run the silent xml install or not (this was a test server so no need to worry about reset to Small deployment).

Tried it but kept getting errors in the logs:

Message    : ERROR: permission denied to create extension "uuid-ossp"

MF suggested setting (but again why?!):

GRANT ALL ON DATABASE mgmt TO hpbsm;

GRANT ALL ON DATABASE rtsm TO hpbsm;

GRANT ALL ON DATABASE event TO hpbsm;

ALTER USER hpbsm WITH SUPERUSER;

Then if we then use our original silent config xml we can get OBM up and running, but the DB is empty...

So we have progress, but not the pain free upgrade I was hoping for.

MF support engaged now and I am waiting for advice on the database being empty...

So frustrating...

[CheeseTriangle]

Hello, I'm really sorry to hear it's not been easy.  If the DB is empty, then you might need to get your backups and restore the database.

upgrade.sh does this:

606 log "INFO: ($0) Backing up config data and DB to /var/opt/OV/backup before upgrade. Please be patient..."
607 log "INFO: Backup logs present in /var/opt/OV/log/opr-upgradeBackup.log"

starts this java to backup the database because it's embedded:

608 $SUDO $JAVA $JVM_FIPS_OPTS -client -cp $CLASSPATH $LOGOPTIONS -Djava.library.path=${TOPAZ_HOME}/bin -Dtopaz.home=${TOPAZ_HOME} com.hp.opr.premigration.PreUpdateTool -b -f /var/opt/OV/bac kup/OMi${SOURCEVERSION}cfg-backup.zip
609 if [ $? -ne 0 ]; then

So the above java has failed to run correctly, the exit code is not equal to zero then the error message you've seen is then shown.

610 log "ERROR: ($0) Backup failed. Please delete all files in /var/opt/OV/backup/ and re run this tool."
611 exit 1

So it's this com.hp.opr.premigration.PreUpdateTool class I would be interested in seeing how it worked - it's logged /var/opt/OV/log/opr-upgradeBackup.log and this is where I'd look next...

CT.

[David Gerrish]

Thanks for your reply - much appreciated.

I will check the logs but the upgrade did seem to complete and I can login and the version shows as 2020.05

But what can I do now - I can't run the upgrade again can I as it would back an empty DB?

I am wondering if therefore the restoreOMi10cfg.sh script could be re-run manually maybe?

That is what I am hoping MF support will suggest.

At the moment I seem to have an upgraded 2020.05 system with no nodes, no alerts, no LDAP config, no connected servers...

[CheeseTriangle]

Hello, sorry to hear you are not making progress with Micro Focus.  I would recommend escalation the call if you are not getting the answers you need if you are doing an in-place migration.  If you are using a side-by-side migration then I would start again.  Everything is stored in the database, so the database needs to be present in one form or another so if the DB is empty, yes, I can understand you don’t see anything...  CT.

[David Gerrish]

Thanks.  I am doing in-place migration in pre-prod as a test before I do this on production, so my confidence is extremely low at the moment.

Just wondering if I could run the script I see in the MF scripts manually - restoreOMi10cfg.sh

or if I should just run the upgrade again, but that will just backup a blank database, so I don't want that to be restored of course... so waiting on MF support to advise next steps.

A painful process, the upgrade just does not work as documented, poor documentation and poor scripting.

[CheeseTriangle]

The restoreOMi10cfg script recovers a DB dump from /opt/HP/BSM/db/pgdata/omiupgradedb.dump and some configuration files so this is used.  Given your circumstances, I would recover from backup because you'll need confidence when you start with your production setup.  But, if you don't want to recover from backup, that's your choice.  You can take a copy of your current production DB (see the DRP process) if that helps.

Next, I think the documentation for 2020.05 is ok - it's loads better than earlier versions.  I've made the mistake before on not selecting my options correctly before selection "view" which was a self induced problem from which I've learnt.  Now I print off the output and read it a few times.  I've also started selecting some of the other options and reading options there too because I think this helps.  I don't think the scripts or the java is that bad, but I can understand how you feel.  I don't have a half upgraded system to worry about right now.  I have been in this situation before and I know how it feels waiting for that phone call and the webex to start.  That's why I think you should escalate.

It's hard to know what to do next.  If OBM is running then something is in the database - unless you ran /opt/HP/BSM/bin/config-server-wizard.sh which will reload default configuration.  

[David Gerrish]

But how do I restore the backup - I can't use the production data, I want the preprod database back.

The upgrade created a backup in /var/opt/OV/backup (zip file) so I am guessing I just need to restore that somehow.  That is what I think I need confirmation of, can I restore, or do I need to re-run the upgrade (taking care what it will restore, as I don't want it restoring a blank DB).

How do I escalate?  Just ask them to escalate it?  I am getting 1 email per day at the moment at best, so it's frustratingly slow.

As for the script, if it worked I'd be happy, but to error and then have MF tell me to manually change a lot of things, when this was a fresh 2019 install, then patched to IP1, then upgraded, that gives me zero confidence in the doc or scripts.

[David Gerrish]

I think I can take thee manual postgres backup I did before the upgrade and use the OBM restore script:

opr-pgctl.sh -restore <file>

I guess I have nothing to lose trying this - I would just like confirmation from MF support first ideally...

[andreask]

Install OBM 2020.05 , start the configuration wizard, choose upgrade and give the filename and postgres Password.

then the the database should be restored.

[David Gerrish]

Thanks - are you saying re-run the upgrade?

And are you saying it will prompt for the dB to restore from?  It didn't do that when I did the initial install though.

Thanks,

Dave

[andreask]

The Upgrade works like this:

backup of DB

uninstall old OBM

install OBM

restore DB

configure OBM

as you have the backup file you just have to install the binaries like on a brand new server. then run configuration wizard and select upgrade.  I did this always with the config-server-wizard.sh but with oBM 2020.05 may be /opt/HP/BSM/bin/config-server-wizard.sh mstut use.

just try.

as long as you have a copy of backup.zip file you can start over again.

[root@omi-lin1 ~]# find /opt/HP/BSM/ -name *wizard*.sh
/opt/HP/BSM/bin/config-server-wizard.sh
/opt/HP/BSM/installation/bin/product_upgrade_wizard.sh

I used this method to do an upgrade of the OS to newer version. after uninstall of OBM i stopped.  Then fresh install of  the os. reinstalled OBM and started the config wizard to use the backup.zip file

[David Gerrish]

Thanks again - which one of those would I use though?

I was going to just run the upgrade.sh script again, but then I see these 2 scripts!

Thanks,

Dave