Skip to content

Instantly share code, notes, and snippets.

@lusoal

lusoal/runbook-move-pods.md

Created October 27, 2022 18:18
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save lusoal/c5257aa801b729f2fd9dedf6c3fd2c2e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save lusoal/c5257aa801b729f2fd9dedf6c3fd2c2e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
runbook-move-pods.md

Runbook to create New Nodes, Cordon and Drain

Creating NodeGroups

First of all we are going to create 3 different NodeGroups in 3 different AZs (Availability Zone). In that way the AutoScaling Group will not try to rebalance the Nodes to place the same amount of Nodes in different AZs.

Follow that documentation in order to create the different NodeGroups.

https://docs.aws.amazon.com/eks/latest/userguide/create-managed-node-group.html

For Desired, Minimum and Maximum Size:

Minimum size – Specify the minimum number of nodes that the managed node group can scale in to. (You can copy that from your default NodeGroup)

Note: If you minimum for one Node is for example 4, and you are going to have 3 different NodeGroups right now instead of one, you can create two NodeGroups with minimum of 1, and another one with minimum of 4, that will give you 4 Nodes minimum if we combine the 3 NodeGroups.

Maximum size – Specify the maximum number of nodes that the managed node group can scale out to.

Desired size – Specify the current number of nodes that the managed node group should maintain at launch.

Note: For desired size, check how many Nodes do you have in your cluster right now, and define the desired of the 3 NodeGroups to match the same amount of Instances, the same that we did for minimum.

Note: Cluster Austoscaler will be the responsible to change the desired nodes, to Guaruntee on the moving proccess from your old Nodes to New ones, you can set the miminum == desired and after you migrate you can change the minimum to you previous state.

On the Specify networking page: Select just one subnet per NodeGroup, in that way we guarantee that the ASG will be created and place instances in just one subnet (One AZ).

Cordon Old Nodes

Cordon: Marking a node as unschedulable prevents the scheduler from placing new pods onto that Node but does not affect existing Pods on the Node. This is useful as a preparatory step before a node reboot or other maintenance.

In Order to the Cordon work properly and Cluster Autoscaler doesn't try to scale-in or scale-out the Nodes on the moving proccess, you can set the minimum, and max size of you Old NodeGroup the same, so for example, if you have 4 as desired, set 4 as your minimum and maximum size.

Get the Nodes that you wanna Cordon.

kubectl get pod -o  wide -n your-namespace

this will give you the VM on which your pod is running, then execute.

kubectl cordon {name_of_that_node_in_which_POD_is_running}

That will prevent that new Pods are going to be placed onto that Node.

Do that for all the Nodes that you don't want to use anymore. (Nodes from your old NodeGroup)

Moving Pods from One Node to Another

There are many ways that we can do that, what I do recommend is, in order to don't have unavailability of your workload, what you can do is:

Scaling you application Pods to 2 times the size that they are right now:

kubectl scale deployment/deployment-name -n namespace --replicas 2times-your-replicas

Note: HPA could impact this, so maybe stoping HPA at that time is recommended.

Since we marked our old Nodes as unschedulable, the new Pods will be placed onto your Nodes that we've just created.

Once the Pods are Up and Running onto the New Nodes, we can Drain the Old Nodes (That are marked as unschedulable), to remove everything from them.

kubectl drain <node name>

After draining, since we have scaled our deployment, the Deployment Controller will place the drained replicas into the new Nodes, now its time to scale down the replicas again to the same amout that you used to have.

kubectl scale deployment/deployment-name -n namespace --replicas original-size

After that, you pods will be Up and Running on the new Nodes, you can delete your Old AutoScaling Group.

Here is the official Kubernetes Documentation about Draining: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/

It will explain you with much more details about that proccess in case you are interested.

Conclusion

Now that we have One NodeGroup per Availability Zone, we don't need to worry about the AutoScaling Group Rebalance, since it will not try to rebalance the ASG because we have one ASG per Availability Zone.

Remember to define the terminationGracePeriodSeconds and safe-to-evict annotation.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-application
  name: my-application
  namespace: scaleout
spec:
  replicas: 20
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
      annotations:
         "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
    spec:
      containers:
      - image: brentley/ecsdemo-nodejs:latest
        name: my-application
        resources:
          requests:
            memory: "2278Mi"
            cpu: "450m"
          limits:
            memory: "2278Mi"
            cpu: "450m"
      terminationGracePeriodSeconds: 14400

My termination logs in case you wanna check that, more than 2 hours in the terminating proccess.

my-application-d94594766-4mdzr Time:7m37s my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:7m44s
my-application-d94594766-4mdzr Time:8m38s my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:8m45s
my-application-d94594766-4mdzr Time:9m39s my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:9m46s
my-application-d94594766-4mdzr Time:10m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:10m
my-application-d94594766-4mdzr Time:11m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:11m
my-application-d94594766-4mdzr Time:12m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:12m
my-application-d94594766-4mdzr Time:13m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:13m
my-application-d94594766-4mdzr Time:14m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:14m
my-application-d94594766-4mdzr Time:15m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:15m
my-application-d94594766-4mdzr Time:16m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:16m
my-application-d94594766-4mdzr Time:17m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:17m
my-application-d94594766-4mdzr Time:18m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:18m
my-application-d94594766-4mdzr Time:19m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:19m
my-application-d94594766-4mdzr Time:20m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:20m
my-application-d94594766-4mdzr Time:21m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:21m
my-application-d94594766-4mdzr Time:22m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:22m
my-application-d94594766-4mdzr Time:23m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:23m
my-application-d94594766-4mdzr Time:24m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:24m
my-application-d94594766-4mdzr Time:25m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:25m
my-application-d94594766-4mdzr Time:26m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:27m
my-application-d94594766-4mdzr Time:27m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:28m
my-application-d94594766-4mdzr Time:28m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:29m
my-application-d94594766-4mdzr Time:29m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:30m
my-application-d94594766-4mdzr Time:30m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:31m
my-application-d94594766-4mdzr Time:31m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:32m
my-application-d94594766-4mdzr Time:32m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:33m
my-application-d94594766-4mdzr Time:33m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:34m
my-application-d94594766-4mdzr Time:34m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:35m
my-application-d94594766-4mdzr Time:36m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:36m
my-application-d94594766-4mdzr Time:37m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:37m
my-application-d94594766-4mdzr Time:38m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:38m
my-application-d94594766-4mdzr Time:39m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:39m
my-application-d94594766-4mdzr Time:40m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:40m
my-application-d94594766-4mdzr Time:41m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:41m
my-application-d94594766-4mdzr Time:42m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:42m
my-application-d94594766-4mdzr Time:43m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:43m
my-application-d94594766-4mdzr Time:44m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:44m
my-application-d94594766-4mdzr Time:45m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:45m
my-application-d94594766-4mdzr Time:46m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:46m
my-application-d94594766-4mdzr Time:47m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:47m
my-application-d94594766-4mdzr Time:48m my-application-d94594766-6n8nw Time:(39h my-application-d94594766-r8vgv Time:48m
my-application-d94594766-4mdzr Time:49m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:49m
my-application-d94594766-4mdzr Time:50m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:50m
my-application-d94594766-4mdzr Time:51m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:51m
my-application-d94594766-4mdzr Time:52m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:52m
my-application-d94594766-4mdzr Time:53m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:53m
my-application-d94594766-4mdzr Time:54m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:54m
my-application-d94594766-4mdzr Time:55m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:55m
my-application-d94594766-4mdzr Time:56m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:56m
my-application-d94594766-4mdzr Time:57m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:57m
my-application-d94594766-4mdzr Time:58m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:58m
my-application-d94594766-4mdzr Time:59m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:59m
my-application-d94594766-4mdzr Time:60m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:60m
my-application-d94594766-4mdzr Time:61m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:61m
my-application-d94594766-4mdzr Time:62m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:62m
my-application-d94594766-4mdzr Time:63m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:63m
my-application-d94594766-4mdzr Time:64m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:64m
my-application-d94594766-4mdzr Time:65m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:65m
my-application-d94594766-4mdzr Time:66m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:66m
my-application-d94594766-4mdzr Time:67m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:67m
my-application-d94594766-4mdzr Time:68m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:68m
my-application-d94594766-4mdzr Time:69m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:69m
my-application-d94594766-4mdzr Time:70m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:70m
my-application-d94594766-4mdzr Time:71m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:71m
my-application-d94594766-4mdzr Time:72m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:72m
my-application-d94594766-4mdzr Time:73m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:73m
my-application-d94594766-4mdzr Time:74m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:74m
my-application-d94594766-4mdzr Time:75m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:75m
my-application-d94594766-4mdzr Time:76m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:76m
my-application-d94594766-4mdzr Time:77m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:77m
my-application-d94594766-4mdzr Time:78m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:78m
my-application-d94594766-4mdzr Time:79m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:79m
my-application-d94594766-4mdzr Time:80m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:80m
my-application-d94594766-4mdzr Time:81m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:81m
my-application-d94594766-4mdzr Time:82m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:82m
my-application-d94594766-4mdzr Time:83m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:83m
my-application-d94594766-4mdzr Time:84m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:84m
my-application-d94594766-4mdzr Time:85m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:85m
my-application-d94594766-4mdzr Time:86m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:86m
my-application-d94594766-4mdzr Time:87m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:87m
my-application-d94594766-4mdzr Time:88m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:88m
my-application-d94594766-4mdzr Time:89m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:89m
my-application-d94594766-4mdzr Time:90m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:90m
my-application-d94594766-4mdzr Time:91m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:91m
my-application-d94594766-4mdzr Time:92m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:92m
my-application-d94594766-4mdzr Time:93m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:93m
my-application-d94594766-4mdzr Time:94m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:94m
my-application-d94594766-4mdzr Time:95m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:95m
my-application-d94594766-4mdzr Time:96m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:96m
my-application-d94594766-4mdzr Time:97m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:97m
my-application-d94594766-4mdzr Time:98m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:98m
my-application-d94594766-4mdzr Time:99m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:99m
my-application-d94594766-4mdzr Time:100m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:100m
my-application-d94594766-4mdzr Time:101m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:101m
my-application-d94594766-4mdzr Time:102m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:103m
my-application-d94594766-4mdzr Time:103m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:104m
my-application-d94594766-4mdzr Time:104m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:105m
my-application-d94594766-4mdzr Time:105m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:106m
my-application-d94594766-4mdzr Time:106m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:107m
my-application-d94594766-4mdzr Time:107m my-application-d94594766-6n8nw Time:(40h my-application-d94594766-r8vgv Time:108m
my-application-d94594766-4mdzr Time:108m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:109m
my-application-d94594766-4mdzr Time:109m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:110m
my-application-d94594766-4mdzr Time:110m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:111m
my-application-d94594766-4mdzr Time:112m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:112m
my-application-d94594766-4mdzr Time:113m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:113m
my-application-d94594766-4mdzr Time:114m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:114m
my-application-d94594766-4mdzr Time:115m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:115m
my-application-d94594766-4mdzr Time:116m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:116m
my-application-d94594766-4mdzr Time:117m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:117m
my-application-d94594766-4mdzr Time:118m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:118m
my-application-d94594766-4mdzr Time:119m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:119m
my-application-d94594766-4mdzr Time:120m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:120m
my-application-d94594766-4mdzr Time:121m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:121m
my-application-d94594766-4mdzr Time:122m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:122m
my-application-d94594766-4mdzr Time:123m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:123m
my-application-d94594766-4mdzr Time:124m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:124m
my-application-d94594766-4mdzr Time:125m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:125m
my-application-d94594766-4mdzr Time:126m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:126m
my-application-d94594766-4mdzr Time:127m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:127m
my-application-d94594766-4mdzr Time:128m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:128m
my-application-d94594766-4mdzr Time:129m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:129m
my-application-d94594766-4mdzr Time:130m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:130m
my-application-d94594766-4mdzr Time:131m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:131m
my-application-d94594766-4mdzr Time:132m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:132m
my-application-d94594766-4mdzr Time:133m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:133m
my-application-d94594766-4mdzr Time:134m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:134m
my-application-d94594766-4mdzr Time:135m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:135m
my-application-d94594766-4mdzr Time:136m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:136m
my-application-d94594766-4mdzr Time:137m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:137m
my-application-d94594766-4mdzr Time:138m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:138m
my-application-d94594766-4mdzr Time:139m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:139m
my-application-d94594766-4mdzr Time:140m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:140m
my-application-d94594766-4mdzr Time:141m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:141m
my-application-d94594766-4mdzr Time:142m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:142m
my-application-d94594766-4mdzr Time:143m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:143m
my-application-d94594766-4mdzr Time:144m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:144m
my-application-d94594766-4mdzr Time:145m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:145m
my-application-d94594766-4mdzr Time:146m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:146m
my-application-d94594766-4mdzr Time:147m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:147m
my-application-d94594766-4mdzr Time:148m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:148m
my-application-d94594766-4mdzr Time:149m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:149m
my-application-d94594766-4mdzr Time:150m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:150m
my-application-d94594766-4mdzr Time:151m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:151m
my-application-d94594766-4mdzr Time:152m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:152m
my-application-d94594766-4mdzr Time:153m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:153m
my-application-d94594766-4mdzr Time:154m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:154m
my-application-d94594766-4mdzr Time:155m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:155m
my-application-d94594766-4mdzr Time:156m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:156m
my-application-d94594766-4mdzr Time:157m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:157m
my-application-d94594766-4mdzr Time:158m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:158m
my-application-d94594766-4mdzr Time:159m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:159m
my-application-d94594766-4mdzr Time:160m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:160m
my-application-d94594766-4mdzr Time:161m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:161m
my-application-d94594766-4mdzr Time:162m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:162m
my-application-d94594766-4mdzr Time:163m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:163m
my-application-d94594766-4mdzr Time:164m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:164m
my-application-d94594766-4mdzr Time:165m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:165m
my-application-d94594766-4mdzr Time:166m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:166m
my-application-d94594766-4mdzr Time:167m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:167m
my-application-d94594766-4mdzr Time:168m my-application-d94594766-6n8nw Time:(41h my-application-d94594766-r8vgv Time:168m
my-application-d94594766-4mdzr Time:169m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:169m
my-application-d94594766-4mdzr Time:170m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:170m
my-application-d94594766-4mdzr Time:171m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:171m
my-application-d94594766-4mdzr Time:172m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:172m
my-application-d94594766-4mdzr Time:173m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:173m
my-application-d94594766-4mdzr Time:174m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:174m
my-application-d94594766-4mdzr Time:175m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:175m
my-application-d94594766-4mdzr Time:176m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:176m
my-application-d94594766-4mdzr Time:177m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:177m
my-application-d94594766-4mdzr Time:178m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:179m
my-application-d94594766-4mdzr Time:179m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:3h
my-application-d94594766-4mdzr Time:3h my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:3h1m
my-application-d94594766-4mdzr Time:3h1m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:3h2m
my-application-d94594766-4mdzr Time:3h2m my-application-d94594766-6n8nw Time:(42h my-application-d94594766-r8vgv Time:3h3m
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment