kubectl not working, unauthorized access

Any kind of kubectl won't work and have an error such as  error: You must be logged in to the server (Unauthorized)

[root@VM2SMk8sMaster ~]# kubectl top nodes
error: You must be logged in to the server (Unauthorized)

[root@VM2SMk8sMaster ssl]# kubectl get nodes
error: the server doesn't have a resource type "nodes"

[root@VM2SMk8sMaster ssl]# kubectl get pods --all-namespaces -o wide
error: the server doesn't have a resource type "pods"

 

  • Is this for a AWS install ? If I Google the error messages, I hit a lot of AWS configuration issues and a couple  due to changes in kubernetes version or new certs.

  • Hi, this is a On-premise installed server
    currently we are lockout on SMAX kubernetes can;t perfrom any kubectl commands.
  • Hi,

    the Role based access control (RBAC) of kubernetes is not allowing you server to execute the kubectl command. 

    depending on the version your are using I would suggest to check your internal certificates. 

    Run these commands from the master node. Depending on your CDF version the first command should show success if internal certificates are OK.  The second may not work if you are using an old version but at least the first one should help. 

    #/opt/kubernetes/scripts/certCheck -ca /opt/kubernetes/ssl/ca.crt -cert /opt/kubernetes/ssl/server.crt -key /opt/kubernetes/ssl/server.key -host FQDN_of_master

    #/opt/kubernetes/scripts/certCheck -ca /opt/kubernetes/ssl/ca.crt -cert /opt/kubernetes/ssl/kubernetes.crt -key /opt/kubernetes/ssl/kubernetes.key -host FQDN_of_master

     

    FQDN_of_master = your master FQDN

  • [root@VM2SMk8sMaster ~]# hostname
    VM2SMk8sMaster
    [root@VM2SMk8sMaster ~]# /opt/kubernetes/scripts/certCheck -ca /opt/kubernetes/ssl/ca.crt -cert /opt/kubernetes/ssl/server.crt -key /opt/kubernetes/ssl/server.key -host VM2SMk8sMaster


    success
    0


    [root@VM2SMk8sMaster ~]# /opt/kubernetes/scripts/certCheck -ca /opt/kubernetes/ssl/ca.crt -cert /opt/kubernetes/ssl/kubernetes.crt -key /opt/kubernetes/ssl/kubernetes.key -host VM2SMk8sMaster


    Server certificate is expired or soon
    301


    [root@VM2SMk8sMaster ~]#

  • well it said you kubernetes certificates (kubernetes.crt and kubernetes.key) are expired

    You can navigate to /opt/kubernetes/ssl

    and run this command to check the date:

    openssl x509 -in kubernetes.crt -text -noout

    Validity
    Not Before:
    Not After :

    What is your CDF version ? 

  • Hi here is the result, currently on 2019.08

     

    p3.PNG

    But when using kube status script, it shows that we still have 9 years left.

    p4.PNG

     

     

  • The Kube-status.sh output most probably referring to the server.key and server.crt certificates to confirm run this command from /opt/Kubernetes/ssl/:
    #openssl x509 -in server.crt -text -noout

    The kubernetes internal client certificates can also be verified with:
    #openssl verify -CAfile /opt/kubernetes/ssl/ca.crt /opt/kubernetes/ssl/kubernetes.crt

    Your client certificate (/opt/Kubernetes/ssl/Kubernetes.crt) is the one which is used by kubectl to authenticate against the kubernetes api-server. No kubectl command will work if those certificates are expired

    As a workaround (with no guarantee that it will work in your case), you can backup your kubernetes.crt and kubernetes.key then use the server certificate as client certificate:

    #cp /opt/Kubernetes/ssl/server.crt /opt/Kubernetes/ssl/kubernetes.crt
    and
    #cp /opt/Kubernetes/ssl/server.key /opt/Kubernetes/ssl/kubernetes.key
    The better solution would be to create a csr and sign it with the ca.key located in /opt/kubernetes/ssl

    Or simple follow the guide to renew all certificates and the dates will be aligned for all. You can open a support case to get help in renewing those certificates

  • Hi thanks for your reply, Is there a downtime on SMAX, if we would copy the certificate the Server cert to Kubernetes cert?
  • The copy command is going to update the contains of your kubernetes.* with the contains of the server.* and since the server.* are still valid the k8s api-server may accept it.
    So we re just updating the kubernetes.* which are expired anyway. When executing the copy commands ,I would really expect the status of the suite (SMAX) to be what it is now 

    Again, please first backup your kubernetes.crt and kubernetes.key
    on the Masters
    #cp /opt/kubernetes/ssl/server.crt /opt/kubernetes/ssl/kubernetes.crt
    #cp /opt/kubernetes/ssl/server.key /opt/kubernetes/ssl/kubernetes.key

    on the workers
    #cp /opt/kubernetes/ssl/client.crt /opt/kubernetes/ssl/kubernetes.crt
    #cp /opt/kubernetes/ssl/client.key /opt/kubernetes/ssl/kubernetes.key

    I have to repeat this:
    It is a workaround with no guarantee that it will work in your case. For my test system where I did the test it was oK. The better solution would be to create a csr and sign it with the ca.key located in /opt/kubernetes/ssl or simple follow the guide to renew all certificates and the dates will be aligned for all. You can open a support case to get help in renewing those certificates

     

  • Tested it on our test server, but got this error, We've already got a support ticket but no one is replying 

    c1.PNG