Ensuring Proper Scheduling of Identity Provider and Access Gateway Pods when upgrade is performed for Access Manager docker images deployed on Kubernetes cluster

1 Likes

Description

During upgrade, when number of Kubernetes worker nodes is more than the number of Identity Provider and Access Gateway replicas, then there can be cases where Kubernetes scheduler may assign pods to a different node upon upgrade.

This article describes the steps to be followed to ensure that Identity Provider and Access Gateway pods are scheduled to same Kubernetes nodes as that of before, once a successful Access Manager upgrade is performed.

Steps to follow while upgrading Access Manager Deployment

  1. Before upgrade, assign node labels for all the nodes where Identity Provider and Access Gateway are deployed. 
    To do this, first identify the nodes where pods are scheduled by running the given command on master node -
    kubectl get pods -o wide -n <namespace>
    From the result of above command, tag the nodes with appropriate labels based on Identity provider or Access Gateway pods scheduled on them.
    a. If Identity Provider is scheduled on a node, assign node label using command -
    kubectl label node <k8_node_name> nam.app/idp=true
    b. If Access Gateway is scheduled on a node, assign node label using command -
    kubectl label node <k8_node_name> nam.app/ag=true
  2. Add nodeAffinity for Identity Provider and Access Gateway in access-manager-1.x.x/values.yml* file which will be used in upgrade

    a. Under affinity of “am-idp” section in values.yml, add below -
    affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nam.app/idp
                operator: In
                values:
                - "true"

   b. Under affinity of “am-ag” section values.yml, add below -

   affinity:
      nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: nam.app/ag
               operator: In
               values:
               - "true"
* - (1.x.x – version of new chart that is pulled)
3. Perform Access Manager upgrade to 1.x.x using helm upgrade command -
    helm upgrade --namespace <name-of-the-namespace> <release-name> access-manager-1.x.x --reuse-values

Example Scenario

Below is an example scenario where fresh installation of Access Manager 5.0 is upgraded to 5.0 Service Pack 1.

Deployment Specification of Access Manager 5.0

  • Kubernetes cluster has 2 master nodes and 4 worker nodes and all six nodes are schedulable.
    sudo kubectl get nodes
  • Access Manager 5.0 is deployed with one Primary and one Secondary Administration Console, three Identity Providers and three Access Gateways. Deployment is done using 'development' namespace. By default, Kubernetes nodes will not have any Access Manager related node labels.
    sudo kubectl get pods -n development -o wide
  • Cluster configuration for Identity Provider and Access Gateway done as below

Upgrading Access Manager from 5.0 to 5.0 Service Pack 1

Step 1 – Assigning node labels before upgrade

On Kubernetes master node, run below command to get the list of 5.0 Identity provider and Access Gateway pods and the nodes they are scheduled to -
sudo kubectl get pods -n development -o wide

a. Assign node labels for Identity Provider
From master node, add labels for all nodes where 5.0 Identity Providers are deployed.
sudo kubectl label node node5 nam.app/idp=true
sudo kubectl label node node4 nam.app/idp=true
sudo kubectl label node node1 nam.app/idp=true

After assigning label for Identity Provider nodes, verify the labels using command -
sudo kubectl get nodes --show-labels | grep idp

“nam.app/idp=true” can be found for all the 3 specified nodes.

b. Assign node labels for Access Gateway
From master node, add labels for all nodes where 5.0 Access Gateways are deployed.
sudo kubectl label node node6 nam.app/ag=true
sudo kubectl label node node3 nam.app/ag=true
sudo kubectl label node node4 nam.app/ag=true

After assigning label for Access Gateway nodes, verify the labels using command -
sudo kubectl get nodes --show-labels | grep ag

“nam.app/ag=true” can be found for all the 3 specified nodes.

Step 2 - Update Identity Provider and Access Gateway nodeAffinity in values.yaml file

  • Pull Access Manager 5.0 Service Pack 1 helm chart, which on extracting will give access-manager-1.0.1 folder.
  • After this, either copy access-manager-1.0.0/values.yml that was used to deploy Access Manager 5.0 to access-manager-1.0.1 folder or update access-manager-1.0.1/values.yml with appropriate values used in 5.0 deployment.
  • Now update nodeAffinity section under “am-idp” for Identity Provider and under “am-ag” for Access Gateway in access-manager-1.0.1/values.yml.

Update nodeAffinity for Identity Provider in values.yml file

Update nodeAffinity for Access Gateway in values.yml file

Step 3 - Perform Access Manager upgrade using helm upgrade command

Execute helm upgrade command to upgrade Access Manager 5.0 pods to 5.0 Service Pack 1 -
sudo helm upgrade --namespace development access-manager access-manager-1.0.1 --reuse-values

Once helm upgrade command is successfully executed, validate that all Identity Provider and Access Gateway pods are getting scheduled to same nodes as before.
To validate pod scheduling execute command - sudo kubectl get pods -n development -o wide

Once all pods are up and running and in ‘Ready’ state, validate helm history is having expected release history. To validate this, execute command - sudo helm history access-manager -n development

Also access Administration Console to validate Identity Provider and Access Gateway pod scheduling is intact in user interface as well. Also validate that upgraded version is reflected in Administration Console.

NOTES

1. If replica count is decreased before upgrade, then it is recommended to remove node label as well, before performing helm upgrade.

2. Before upgrade, if replica count is increased, then it is recommended to add node label before performing helm upgrade.

Labels:

How To-Best Practice
Other
Comment List
Anonymous
Related Discussions
Recommended