K8s | Probes Lab

Problem Statement:-

In this lab, we will use liveness and readiness probes to ensure that k8s to handle the availability of microservices. Implement readiness & liveness for all of our microservice:

• Salary, Employee, Attendance, Gateway
• Identify the difference in deployment with and without readiness probe

By the end of this lab, you will have a clear understanding of probes and how important a role they play in a fully functional app.

What you will not have yet is control in terms of allocation of pods in your k8s cluster

Elasticsearch

So now let's deploy the elasticsearch and it's service to make this work.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: empms-es
  name: empms-es
spec:
  replicas: 1
  selector:
    matchLabels:
      app: empms-es
  template:
    metadata:
      labels:
        app: empms-es
    spec:
      containers:
      - image: opstree/empms-es:1.0
        imagePullPolicy: Always
        name: empms-es
        ports:
        - containerPort: 9200

kubectl apply -f elastic-deployment.yaml

Once after that, we need to create the service of elasticsearch

---
kind: Service
apiVersion: v1
metadata:
  name: empms-es
spec:
  type: ClusterIP
  selector:
    app: empms-es
  ports:
  - protocol: TCP
    port: 9200

kubectl apply -f elastic-service.yaml

Salary

We have to create the salary component with readiness and liveness probe.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: empms-salary
  name: empms-salary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: empms-salary
  template:
    metadata:
      labels:
        app: empms-salary
    spec:
      containers:
      - image: opstree/empms-salary:1.0
        imagePullPolicy: Always
        name: empms-salary
        env:
        - name: DELAY_TIME
          value: "5"
        ports:
        - containerPort: 8082
          name: client
        livenessProbe:
          httpGet:
            path: /salary/healthz
            port: 8082
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /salary/healthz
            port: 8082
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 10

Create the salary deployment

kubectl apply -f salary-deployment.yaml

Liveness Test

Let's test the liveness for salary application.

Exec inside the salary pod by kubectl exec.

kubectl exec -it empms-salary-xxx bash

Once you are inside the salary pod, put this curl request

curl -XPOST http://localhost:8082/salary/configure/liveness?delay_time=100

This will delay the response of health check

Now if you see that Kubernetes will restart the salary application because of delay in response.

You can watch it by

kubectl get pod -w

kubectl describe pod empms-salary-xxx

Attendance

Deployment Without Probes

Update the attendance Deployment manifest.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: empms-attendance
  name: empms-attendance
spec:
  replicas: 1
  selector:
    matchLabels:
      app: empms-attendance
  template:
    metadata:
      labels:
        app: empms-attendance
    spec:
      containers:
      - image: opstree/empms-attendance:1.0
        imagePullPolicy: Always
        name: empms-attendance
        ports:
        - containerPort: 8081

Create a resource from this manifest.

kubectl apply -f attendance-deployment.yaml

Validate the service

kubectl get deployments
kubectl get pods

Make sure empms-db is deployed otherwise readiness check will fail. You can refer deployment and service

Now let's try to update the image version of the attendance

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: empms-attendance
  name: empms-attendance
spec:
  replicas: 1
  selector:
    matchLabels:
      app: empms-attendance
  template:
    metadata:
      labels:
        app: empms-attendance
    spec:
      containers:
      - image: opstree/empms-attendance:2.0
        imagePullPolicy: Always
        name: empms-attendance
        ports:
        - containerPort: 8081

kubectl apply -f attendance-deployment.yaml

To watch the changes for deployment

kubectl get pods -w

Now if you see the second version image pod comes up without any validation.

Deployment with Probe

Update the attendance Deployment manifest.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: empms-attendance
  name: empms-attendance
spec:
  replicas: 1
  selector:
    matchLabels:
      app: empms-attendance
  template:
    metadata:
      labels:
        app: empms-attendance
    spec:
      containers:
      - image: opstree/empms-attendance:1.0
        imagePullPolicy: Always
        name: empms-attendance
        ports:
        - containerPort: 8081
        livenessProbe:
          httpGet:
            path: /attendance/healthz
            port: 8081
          initialDelaySeconds: 3
          periodSeconds: 3
          successThreshold: 1
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /attendance/healthz
            port: 8081
          initialDelaySeconds: 3
          periodSeconds: 3
          successThreshold: 1
          timeoutSeconds: 10

Create a resource from this manifest.

kubectl apply -f attendance-deployment.yaml

Validate the service

kubectl get deployments
kubectl get pods

To watch the changes for deployment

kubectl get pods -w

kubectl describe pod empms-db-xxx

In this case, the application is waiting for few minutes for readiness and liveness check to pass and once the status is the success then it deletes the previous version, but if the check is failing it will not live the second version of the image.

Let's put MySQL again.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: empms-db
  name: empms-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: empms-db
  template:
    metadata:
      labels:
        app: empms-db
    spec:
      containers:
      - image: opstree/empms-db:1.0
        imagePullPolicy: Always
        name: empms-db
        ports:
        - containerPort: 3306

---
kind: Service
apiVersion: v1
metadata:
  name: empms-db
spec:
  type: ClusterIP
  selector:
    app: empms-db
  ports:
  - protocol: TCP
    port: 3306

Create a resource from this manifest.

kubectl apply -f mysql-deployment.yaml
kubectl apply -f mysql-service.yaml

Validate the service

kubectl get deployments
kubectl get pods

To watch the changes for deployment

kubectl get pods -w

Now if you see, after deploying MySQL, attendance service is up again