Highlighted
Super Contributor.
Super Contributor.
300 views

SMAX 2020.02 Install Script for Azure AKS: PVC failed in pending state

Jump to solution

When running the SMAX 2020.02 install script for Azure AKS and using Azure Files Premium for storage, I encounter error:

FATAL Check PVC timeout. PVC cannot be bound to PV normally (STATUS: Pending).
Refer to /tmp/install.20200309175848.log file for detail information. If need, please contact system administrator or Micro Focus support.

I've attached my log.

I googled about PV = pending and found information about possibly an RBAC issue with persistent-volume-binder service account, but I do not see that service account being created here. 

I wondered if anyone has encountered this issue.

 

1 Solution

Accepted Solutions
Highlighted
Super Contributor.
Super Contributor.

I wanted to post the solution I figured out for anyone else who runs into the "PVC failed in pending state" issue, as far as it applied to our environment which uses Azure Files for storage.

  1. The root cause is that the install script, if it encounters an error (in our case the PVC pending), it deletes the entire core namespace via kubectl before exiting
  2. This doesn’t take into account the earlier manual steps in the guide that when using Azure Files, you need to create the core namespace, generate the azure-secret for the namespace and generate the persistent volumes (PV) for the namespace.
  3. If the namespace is deleted, the persistent volume claim done by the install script will remain pending and the script won’t continue, even though the script recreates the namespace (it appears the PVs are affected if the namespace is deleted, so you can't just delete the namespace and recreate it and have the previously created PVs still work in this situation).
  4. I had to figure out that this is what the install script was doing, go back and delete the PVs, recreate the namespace, regenerate the azure secret and recreate the PVs, then restart the install script
  5. On an aside, the latest “createPV.sh” script on the MF web site (wiki) to copy reported a syntax error in our RedHat Linux server, so I had to figure out how to fix it after running it through https://shellcheck.net to determine what the problems were with the pasted code. I’ve attached my fixed version here.

This means, to me, that if the core namespace is deleted, anything previously created in that namespace is going to have problems (in this case the PVs and azure-secret), so it seems you cannot just create the namespace again (as the install script does) and hope to successfully do a PVC. You have to create the namespace, but then go back and redo the earlier steps before the install script that relate to the core namespace: delete the PVs, recreate the core namespace, regenerate the azure secret, and recreate the PVs, then you can restart the install script.

It may be this is specific to Azure, but I suspect the install script shouldn't be deleting the core namespace when some previous steps are already creating elements in that namespace.

 

View solution in original post

4 Replies
Highlighted
Super Contributor.
Super Contributor.

I wanted to post the solution I figured out for anyone else who runs into the "PVC failed in pending state" issue, as far as it applied to our environment which uses Azure Files for storage.

  1. The root cause is that the install script, if it encounters an error (in our case the PVC pending), it deletes the entire core namespace via kubectl before exiting
  2. This doesn’t take into account the earlier manual steps in the guide that when using Azure Files, you need to create the core namespace, generate the azure-secret for the namespace and generate the persistent volumes (PV) for the namespace.
  3. If the namespace is deleted, the persistent volume claim done by the install script will remain pending and the script won’t continue, even though the script recreates the namespace (it appears the PVs are affected if the namespace is deleted, so you can't just delete the namespace and recreate it and have the previously created PVs still work in this situation).
  4. I had to figure out that this is what the install script was doing, go back and delete the PVs, recreate the namespace, regenerate the azure secret and recreate the PVs, then restart the install script
  5. On an aside, the latest “createPV.sh” script on the MF web site (wiki) to copy reported a syntax error in our RedHat Linux server, so I had to figure out how to fix it after running it through https://shellcheck.net to determine what the problems were with the pasted code. I’ve attached my fixed version here.

This means, to me, that if the core namespace is deleted, anything previously created in that namespace is going to have problems (in this case the PVs and azure-secret), so it seems you cannot just create the namespace again (as the install script does) and hope to successfully do a PVC. You have to create the namespace, but then go back and redo the earlier steps before the install script that relate to the core namespace: delete the PVs, recreate the core namespace, regenerate the azure secret, and recreate the PVs, then you can restart the install script.

It may be this is specific to Azure, but I suspect the install script shouldn't be deleting the core namespace when some previous steps are already creating elements in that namespace.

 

View solution in original post

Highlighted
Super Contributor.
Super Contributor.

I should add that this may not be a common occurrence, it is only if the script fails. I ran into a permissions issue and the script failed.

Once the script had failed though, and deleted the core namespace, no amount of re-running the install script (even though it starts by creating the core namespace again), would work; the PVC would always be pending. I believe it is because once you have deleted that namespace, anything created with that namespace, like the earlier creation of PVs and azure-secret, can't be used again just by recreating the core namespace. You have to go back and delete the PVs, redo them in the core namespace, regenerate the azure-secret, before you run the install script again.

So, probably the install script should not delete the core namespace, or else give a message that it is deleting the core namespace when it fails, and any PVs, azure-secret previously created will have to be redone before trying to run the install again.

 

0 Likes
Highlighted
Micro Focus Expert
Micro Focus Expert

I have similar issues on GCP and SMAX 2020.08.

0 Likes
Highlighted
Regular Contributor.. Regular Contributor..
Regular Contributor..

And the same issue on AWS with 2020.08?

Support ticket raised - will wait to hear and update.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.