Linkerd with Nginx Ingress

Azure Kubernetes Service

I installed Linkerd and Nginx on an AKS cluster. You will need an Azure subscription and deploy a basic cluster.

Install Linkerd CLI

Install the Linkerd CLI: see https://linkerd.io/2.11/getting-started/#step-1-install-the-cli. You can also install the cli with brew:

brew install linkerd

On Windows, use Chocolatey:

choco install linkerd

⚠️ Important: in production, use the Helm chart and generate your own certificates. See https://linkerd.io/2.10/tasks/install-helm/

Install Linkerd on the cluster

Use the following commands:

linkerd check --pre
linkerd install | kubectl apply -f -
linkerd check

The first command checks if your cluster meets the pre-requisites. The second command installs Linkerd in the linkerd namespace. The third command checks if Linkerd is installed correctly.

⚠️ Note: it might take some time for the Linkerd pods to be ready in the linkerd namespace. The linkerd check command might time out. Check the pods manually with kubectl get pods -n linkerd and run linkerd check again.

Install the viz extension

The viz extension installs a metric stack on your cluster with Prometheus and Grafana. Use the following commands:

linkerd viz install | kubectl apply -f -
linkerd viz check

Install a sample application

Apply the following YAML to your cluster:

apiVersion: v1
kind: Namespace
metadata:
  name:  linkerdapp
---
kind: Service
apiVersion: v1
metadata:
  name:  superapi
  namespace: linkerdapp
spec:
  selector:
    app:  superapi
  type:  ClusterIP
  ports:
  - name:  http
    port:  80
    targetPort:  8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: superapi
spec:
  replicas: 2
  selector:
    matchLabels:
      app: superapi
  template:
    metadata:
      labels:
        app: superapi
      annotations:
        linkerd.io/inject: "enabled"
    spec:
      containers:
      - name: superapi
        image: ghcr.io/gbaeke/super:1.0.7
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "50m"
        env:
          - name: IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: WELCOME
            value: Welcome from $(IP)
        ports:
        - containerPort: 8080

Save the above to a file and then run kubectl apply -f <file>.

The super-api pods will be meshed because of the linkerd annotation in the deployment's pod template metadata:

annotations:
  linkerd.io/inject: "enabled"

Open the Linkerd dashboard

Start the dashboard with linkerd viz dashboard. A browser will open and you will see the dashboard. If the browser does not open, use the displayed localhost link. You should see all namespaces. The following namespaces should be meshed:

• linkerdapp
• linkerd
• linkerd-viz

Deploy meshed Nginx Ingress Controller

Save the following to a file called values.yaml:

controller:
  podAnnotations:
    linkerd.io/inject: enabled

From the folder containing values.yaml, run the following command:

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --values values.yaml

The above command installs Nginx Ingress Controller in the ingress-nginx namespace. It adds the linkerd.io/inject annotation to the ingress controller pods.

In the Linkerd UI, the namespace ingress-nginx should be meshed.

Because the ingress contoller is meshed, the following happens:

• golden metrics are provided for the ingress controller (requests per second, etc.)
• traffic between the ingress controller pods and meshed application pods is encrypted (and mutually authenticated)
• HTTP traffic can be seen
• When applications return errors (e.g. 5xx HTTP status codes), this will be visible in the Linkerd UI for both the applications but also nginx ingress controller as it returns the error codes to the client.

Adding an ingress to reach the super-api application

Add the following ingress.yaml to the linkerdapp namespace:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: superapi-ingress
  labels:
    name: superapi-ingress
  namespace: linkerdapp
  annotations:
    # add below line of nginx is meshed
    nginx.ingress.kubernetes.io/service-upstream: "true"
    # nginx.ingress.kubernetes.io/affinity: "cookie"
    # nginx.ingress.kubernetes.io/affinity-mode: "persistent"
spec:
  ingressClassName: nginx
  rules:
  # update IP with your own IP used by Ingress Controller
  - host: linkerd.REPLACE-WITH-YOUR-IP.nip.io
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: superapi
            port: 
              number: 80

⚠️ Important notes:

• nip.io is used to provide a custom domain name that resolves to the IP address of the Ingress Controller. You need to retrieve the IP address of the Ingress Controller from the ingress-nginx namespace. Use kubectl get svc -n ingress-nginx and find the external IP. Replace the IP in the hostname above. This IP address is actually a front-end IP on the kubernetes Azure public load balancer in the MC_ group of your AKS cluster.
• the annotation nginx.ingress.kubernetes.io/service-upstream: "true" is used to tell Nginx Ingress Controller to route traffic to the service of the meshed application instead of directly to the pods. By default, Ingress Controllers merely query the endpoints of their target service to retrieve the IP addresses of the pods behind the service. They route traffic to the pods directly. By sending traffic to the service, Linkerd features such as load balancing and traffic splitting are enabled.
• when you use the annotation above, you cannot have sticky sessions with cookie-based affinity

Hit the service with traffic and introduce errors

You can use hey to send traffic to the super-api application via the ingress:

watch hey -n 1500 -c 50 http://linkerd.YOURIP.nip.io/flaky

Above, replace YOURIP with the IP address you used in the Ingress host. The /flaky path is used to introduce errors. It returns a 5xx HTTP status code for about 10% of the requests.

From the Linkerd UI, check the traffic. You should see lots of red because you are not at 100% of the traffic.

Bonus: load testing with Azure

You can use Azure Load Testing to hit the Ingress controller with traffic. Use the following file to configure the load test:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.4.1">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Azure Load Testing Quickstart" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">250</stringProp>
        <stringProp name="ThreadGroup.ramp_time">10</stringProp>
        <boolProp name="ThreadGroup.scheduler">true</boolProp>
        <stringProp name="ThreadGroup.duration">120</stringProp>
        <stringProp name="ThreadGroup.delay">5</stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="Homepage" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">linkerd.20.23.53.238.nip.io</stringProp>
          <stringProp name="HTTPSampler.port">80</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/flaky</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
        </HTTPSamplerProxy>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Above, modify HTTPSampler.domain to the host you configured in the Ingress resource. Modify HTTPSampler.path to the path you want to hit. You can use / to not return errors or /flaky to return errors.