Trying to upgrade CDF version to 2018.11 But upgrade script (-l) fails on the worker node.
We are planning to upgrade our smax 2018.08 environment to 2019.05. For testing the upgrade steps we have started upgrading the dev environment first. Now we are trying the CDF version to 2018.11. There is 1 master node and 1 worker node in this environment. DB is external. Worker node is NFS server . (I had defined the NFS server as worker after some time from the installation) As you can know, upgrade steps are below:
- Step 1 (run command upgrade-g) takes about 1 minute.
- Step 2 (run command upgrade-l) takes about 18 minutes.
- Step 3 (run command upgrade-u) takes about 18 minutes.
We are on Step 2 . It (./upgrade.sh -l) worked in master. But in worker node, it fails. ( it is because of NFS. The Script can not find the NFS server. it can't define the NFS, because NFS is on the same server) After running it, it gives the error below.
What can we do to run this script? Any idea please?
Thanks in Advance.
from the error message, it seems can't get the NFS information. 2 checkpoints:
1. run the command: kubectl get pv itom-vol -o json, and check spec.nfs.server part.
2. check upgrade_temp/CDF_upgrade_parameters.txt, when you run upgrade.sh -g, the CDF parameters should be stored in this file, check whether the parameter: NFS_SERVER and it's value.
I got it run (kubectl get pv itom-vol -o json) NFS server ip in that json seems right.
I checked "upgrade_temp" folder but there is no file such CDF_upgrade_parameters.txt. There is only one file named UPGRADE_START_TIME there.
To continue to upgrade I gave up from worker.I removed the worker role from nfs server and then I started the upgrade procedure. Now, I have only one master server. All pods are on the master. Memory usage was high. I set suite runlevel to DOWN. I applied the upgrade processes when suite runlevel was down. But I did not touch to core pods. They were running. When checking kube-status in bin folder, everything was running.
i think the parameter file should be located in the master node, here is what's document described:\
Run the following commands on one of the master nodes to collect parameters used in CDF and save it into the configmap file.
On the terminal, you will see the following messages:
./upgrade.sh -g ** Generating CDF parameter file ... ** Checking IDM Server ... The IDM Server is running. ** Checking CDF Api Server ... The CDF Api Server is running. File CDF_upgrade_parameters.txt generated under /opt/kubernetes/upgrade_temp Save CDF_upgrade_parameters.txt into configmap successfully. ** [NOTE] Please run command '/root/ITOM_Suite_Foundation_Upgrade_2018.11.0059/upgrade.sh -l' on this node to load CDF Docker images.
According to CDF upgrade instructions (upgrade to 2018.11) described in the 2019.02 document I saw the same issue in the worker node when upgrading other environment. There is a file named upgrade_parameters.txt in the directory /opt/kubernetes/upgrade_temp in both master and worker node. In the master this file is full. But in the worker it has no content. What I did to solve the issue; I copied this file to the worker. So I could run the script.
Any idea about this issue?
Thanks in Advance..