Application Delivery Management
Application Modernization & Connectivity
CyberRes
IT Operations Management
Kubernetes describes two basic mechanisms for checking the health of a container:
Readiness probes are intended to let Kubernetes know when the app is ready to serve traffic. Kubernetes makes sure the readiness probe passes before allowing a service to forward traffic to the pod. Kubernetes don't route work to a container with a failing readiness probe.
A readiness probe can fail if a service is busy, doesn't finish initializing, is overloaded, or unable to process requests.
Liveness probes let Kubernetes know if the app is alive or dead. If your app is alive, then Kubernetes leaves it alone. Kubernetes stops and restarts a container with a failing liveness probe to ensure that Pods in a defunct state are terminated and replaced.
A liveness probe fails if the service is in an unrecoverable state, for example, if an out of memory condition happens.
The probes (Readiness and Liveness) can be achieved by running a command, checking a TCP endpoint for connectivity, or performing an HTTP invocation.
Both readiness and liveness probe run in parallel throughout the life of a container but the probes impact the system in different ways.
When a container starts, the readiness state is initially negative, and it enters a positive state only after the container is working correctly. A liveness check starts in a positive state, and it enters a negative state only when the process becomes inoperative.
Both probes use the same configuration parameters:
Let us understand the configuration of the probes with an example
livenessProbe:
httpGet:
path: /livenessProbe
port: http
initialDelaySeconds: 180
periodSeconds: 60
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /readinessProbe
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 25
Explanation of the values set in the above configuration:
Let me touch upon one more probe in addition to liveness and readiness probes before we look at the debugging section, Startup Probe probe indicates whether the application within the container is started. If this probe is configured, it disables liveness and readiness checks until it succeeds, making sure those probes don't conflict with the application startup.
We can use the following kubectl command to check probe status for a pod
kubectl describe pod <pod-name>
If you are wondering how to get the pod name, you can use kubectl get pods command.
I have taken out errors from NetIQ Access Manager docker deployment for better understanding but these are just for illustration as the docker support is under heavy development.
If there is no error in the probes, you won't see anything under "Events" after you run kubectl describe command.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 7s kubelet, namk8s-w3 Liveness probe failed: IDP Health Check: Waiting to establish connection
Warning Unhealthy 4s (x11 over 19h) kubelet, namk8s-w3 Readiness probe failed: IDP Health Check: Waiting to establish connection
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 115s kubelet, namk8s-w1 Readiness probe failed: AG Tomcat Health Check: Waiting to establish connection
AG JCC Health Check: Waiting to establish connection
Warning Unhealthy 20s (x2 over 80s) kubelet, namk8s-w1 Liveness probe failed: AG Tomcat Health Check: Waiting to establish connection
AG HTTPD Health Check: Waiting to establish connection
AG Activemq Health Check: Waiting to establish connection
AG JCC Health Check: Waiting to establish connection
Warning Unhealthy 5s (x35 over 19h) kubelet, namk8s-w1 Readiness probe failed: AG Tomcat Health Check: Waiting to establish connection
AG HTTPD Health Check: Waiting to establish connection
AG Activemq Health Check: Waiting to establish connection
AG JCC Health Check: Waiting to establish connection
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 4s (x3 over 2m4s) kubelet, namk8s-w3 Liveness probe failed: IDP Health Check: Waiting to establish connection
Normal Killing 4s kubelet, namk8s-w3 Container am-idp failed liveness probe, will be restarted
Warning Unhealthy 1s (x23 over 19h) kubelet, namk8s-w3 Readiness probe failed: IDP Health Check: Waiting to establish connection
Both liveness & readiness probes are used to control the health of an application. Failing a liveness probe will restart the container, whereas a failing readiness probe will stop the application from serving traffic.
Here is the Kubernetes documentation for the reference.