Scenario 1

Application with anti-affinity defined, to scale one replica only per Node.

ASG multi-az: us-east-2a, us-east-2b, us-east-2c

Capacity rebalance enabled

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-application
  name: my-application
  namespace: scaleout
spec:
  replicas: 4
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
      annotations:
         "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
    spec:
      containers:
      - image: brentley/ecsdemo-nodejs:latest
        name: my-application
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
      tolerations:
      - key: "type"
        operator: "Equal"
        value: "multiaz"
        effect: "NoSchedule"
      terminationGracePeriodSeconds: 14400
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - my-application
            topologyKey: "kubernetes.io/hostname"

Scaled from 2 to 4 Nodes to match podAntiAffinity configuration.

Cluster Autoscaler Log:

I1025 14:00:24.295652       1 scale_down.go:509] Scale-down calculation: ignoring 4 nodes unremovable in the last 5m0s

CA is ignoring those Nodes because the Pods still running in there and safe-to-evict flag is defined.

Termination Logs

After more than 20 minutes, the Pods still in Terminating state, thanks to terminationGracePeriodSeconds defined

my-application-797674c4c8-j6bjq Time:6m22s my-application-797674c4c8-qhlp7 Time:6m22s
my-application-797674c4c8-j6bjq Time:7m23s my-application-797674c4c8-qhlp7 Time:7m23s
my-application-797674c4c8-j6bjq Time:8m24s my-application-797674c4c8-qhlp7 Time:8m24s
my-application-797674c4c8-j6bjq Time:9m25s my-application-797674c4c8-qhlp7 Time:9m25s
my-application-797674c4c8-j6bjq Time:10m my-application-797674c4c8-qhlp7 Time:10m
my-application-797674c4c8-j6bjq Time:11m my-application-797674c4c8-qhlp7 Time:11m
my-application-797674c4c8-j6bjq Time:12m my-application-797674c4c8-qhlp7 Time:12m
my-application-797674c4c8-j6bjq Time:13m my-application-797674c4c8-qhlp7 Time:13m
my-application-797674c4c8-j6bjq Time:14m my-application-797674c4c8-qhlp7 Time:14m
my-application-797674c4c8-j6bjq Time:15m my-application-797674c4c8-qhlp7 Time:15m
my-application-797674c4c8-j6bjq Time:16m my-application-797674c4c8-qhlp7 Time:16m
my-application-797674c4c8-j6bjq Time:17m my-application-797674c4c8-qhlp7 Time:17m
my-application-797674c4c8-j6bjq Time:18m my-application-797674c4c8-qhlp7 Time:18m
my-application-797674c4c8-j6bjq Time:19m my-application-797674c4c8-qhlp7 Time:19m
my-application-797674c4c8-j6bjq Time:20m my-application-797674c4c8-qhlp7 Time:20m
my-application-797674c4c8-j6bjq Time:21m my-application-797674c4c8-qhlp7 Time:21m
my-application-797674c4c8-j6bjq Time:22m my-application-797674c4c8-qhlp7 Time:22m
my-application-797674c4c8-j6bjq Time:23m my-application-797674c4c8-qhlp7 Time:23m
my-application-797674c4c8-j6bjq Time:24m my-application-797674c4c8-qhlp7 Time:24m
my-application-797674c4c8-j6bjq Time:25m my-application-797674c4c8-qhlp7 Time:25m
my-application-797674c4c8-j6bjq Time:26m my-application-797674c4c8-qhlp7 Time:26m
my-application-797674c4c8-j6bjq Time:27m my-application-797674c4c8-qhlp7 Time:27m
my-application-797674c4c8-j6bjq Time:28m my-application-797674c4c8-qhlp7 Time:28m
my-application-797674c4c8-j6bjq Time:29m my-application-797674c4c8-qhlp7 Time:29m

ASG Logs

No useful logs from Autoscaling Group

Scenario 2

Application with increased amount of resource requests defined.

ASG multi-az: us-east-2a, us-east-2b, us-east-2c

Capacity rebalance enabled

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-application
  name: my-application
  namespace: scaleout
spec:
  replicas: 20
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
      annotations:
         "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
    spec:
      containers:
      - image: brentley/ecsdemo-nodejs:latest
        name: my-application
        resources:
          requests:
            memory: "2278Mi"
            cpu: "450m"
          limits:
            memory: "2278Mi"
            cpu: "450m"
      tolerations:
      - key: "type"
        operator: "Equal"
        value: "multiaz"
        effect: "NoSchedule"
      terminationGracePeriodSeconds: 14400

Scale-in from 20 to 4 to check terminationGracePeriodSeconds and safe-to-evict.

kubectl scale deployment/my-application --replicas=4 -nscaleout

Cluster Autoscaler Log:

I1025 14:38:57.453292       1 scale_down.go:444] Node ip-10-0-12-61.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 14:38:57.453318       1 scale_down.go:444] Node ip-10-0-11-177.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.661151)
I1025 14:38:57.453339       1 scale_down.go:444] Node ip-10-0-12-155.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 14:38:57.453364       1 scale_down.go:444] Node ip-10-0-11-81.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 14:38:57.453388       1 scale_down.go:444] Node ip-10-0-10-140.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 14:38:57.453409       1 scale_down.go:444] Node ip-10-0-10-24.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 14:38:57.453433       1 scale_down.go:444] Node ip-10-0-12-170.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 14:38:57.453468       1 static_autoscaler.go:520] Scale down status: unneededOnly=false lastScaleUpTime=2022-10-25 14:26:33.054050582 +0000 UTC m=+3628.090764250 lastScaleDownDeleteTime=2022-10-25 12:26:23.59933461 +0000 UTC m=-3581.363951715 lastScaleDownFailTime=2022-10-25 12:26:23.59933461 +0000 UTC m=-3581.363951715 scaleDownForbidden=false isDeleteInProgress=false scaleDownInCooldown=false
I1025 14:38:57.453499       1 static_autoscaler.go:533] Starting scale down
I1025 14:38:57.453553       1 scale_down.go:918] No candidates for scale down

Cluster Autoscaler is not scaling down the Nodes because resources are in use on that Node.

Termination Logs

After more than 20 minutes, the Pods still in Terminating state, thanks to terminationGracePeriodSeconds defined

my-application-d94594766-b2dqv Time:14m my-application-d94594766-drnxx Time:14m my-application-d94594766-fctgp Time:11m my-application-d94594766-g659w Time:14m my-application-d94594766-h2fnq Time:14m my-application-d94594766-hfrkg Time:14m my-application-d94594766-hql4r Time:11m my-application-d94594766-jcwbq Time:14m my-application-d94594766-kgmzc Time:11m my-application-d94594766-rmcdz Time:11m my-application-d94594766-swsn7 Time:14m my-application-d94594766-t5kjr Time:14m my-application-d94594766-t9ft9 Time:11m my-application-d94594766-vbnxs Time:11m my-application-d94594766-x9v64 Time:14m my-application-d94594766-z22df Time:14m
my-application-d94594766-b2dqv Time:15m my-application-d94594766-drnxx Time:15m my-application-d94594766-fctgp Time:12m my-application-d94594766-g659w Time:15m my-application-d94594766-h2fnq Time:15m my-application-d94594766-hfrkg Time:15m my-application-d94594766-hql4r Time:12m my-application-d94594766-jcwbq Time:15m my-application-d94594766-kgmzc Time:12m my-application-d94594766-rmcdz Time:12m my-application-d94594766-swsn7 Time:15m my-application-d94594766-t5kjr Time:15m my-application-d94594766-t9ft9 Time:12m my-application-d94594766-vbnxs Time:12m my-application-d94594766-x9v64 Time:15m my-application-d94594766-z22df Time:15m
my-application-d94594766-b2dqv Time:16m my-application-d94594766-drnxx Time:16m my-application-d94594766-fctgp Time:13m my-application-d94594766-g659w Time:16m my-application-d94594766-h2fnq Time:16m my-application-d94594766-hfrkg Time:16m my-application-d94594766-hql4r Time:13m my-application-d94594766-jcwbq Time:16m my-application-d94594766-kgmzc Time:13m my-application-d94594766-rmcdz Time:13m my-application-d94594766-swsn7 Time:16m my-application-d94594766-t5kjr Time:16m my-application-d94594766-t9ft9 Time:13m my-application-d94594766-vbnxs Time:13m my-application-d94594766-x9v64 Time:16m my-application-d94594766-z22df Time:16m
my-application-d94594766-b2dqv Time:17m my-application-d94594766-drnxx Time:17m my-application-d94594766-fctgp Time:14m my-application-d94594766-g659w Time:17m my-application-d94594766-h2fnq Time:17m my-application-d94594766-hfrkg Time:17m my-application-d94594766-hql4r Time:14m my-application-d94594766-jcwbq Time:17m my-application-d94594766-kgmzc Time:14m my-application-d94594766-rmcdz Time:14m my-application-d94594766-swsn7 Time:17m my-application-d94594766-t5kjr Time:17m my-application-d94594766-t9ft9 Time:14m my-application-d94594766-vbnxs Time:14m my-application-d94594766-x9v64 Time:17m my-application-d94594766-z22df Time:17m
my-application-d94594766-b2dqv Time:18m my-application-d94594766-drnxx Time:18m my-application-d94594766-fctgp Time:15m my-application-d94594766-g659w Time:18m my-application-d94594766-h2fnq Time:18m my-application-d94594766-hfrkg Time:18m my-application-d94594766-hql4r Time:15m my-application-d94594766-jcwbq Time:18m my-application-d94594766-kgmzc Time:15m my-application-d94594766-rmcdz Time:15m my-application-d94594766-swsn7 Time:18m my-application-d94594766-t5kjr Time:18m my-application-d94594766-t9ft9 Time:15m my-application-d94594766-vbnxs Time:15m my-application-d94594766-x9v64 Time:18m my-application-d94594766-z22df Time:18m
my-application-d94594766-b2dqv Time:19m my-application-d94594766-drnxx Time:19m my-application-d94594766-fctgp Time:16m my-application-d94594766-g659w Time:19m my-application-d94594766-h2fnq Time:19m my-application-d94594766-hfrkg Time:19m my-application-d94594766-hql4r Time:16m my-application-d94594766-jcwbq Time:19m my-application-d94594766-kgmzc Time:16m my-application-d94594766-rmcdz Time:16m my-application-d94594766-swsn7 Time:19m my-application-d94594766-t5kjr Time:19m my-application-d94594766-t9ft9 Time:16m my-application-d94594766-vbnxs Time:16m my-application-d94594766-x9v64 Time:19m my-application-d94594766-z22df Time:19m
my-application-d94594766-b2dqv Time:20m my-application-d94594766-drnxx Time:20m my-application-d94594766-fctgp Time:17m my-application-d94594766-g659w Time:20m my-application-d94594766-h2fnq Time:20m my-application-d94594766-hfrkg Time:20m my-application-d94594766-hql4r Time:17m my-application-d94594766-jcwbq Time:20m my-application-d94594766-kgmzc Time:17m my-application-d94594766-rmcdz Time:17m my-application-d94594766-swsn7 Time:20m my-application-d94594766-t5kjr Time:20m my-application-d94594766-t9ft9 Time:17m my-application-d94594766-vbnxs Time:17m my-application-d94594766-x9v64 Time:20m my-application-d94594766-z22df Time:20m
my-application-d94594766-b2dqv Time:21m my-application-d94594766-drnxx Time:21m my-application-d94594766-fctgp Time:18m my-application-d94594766-g659w Time:21m my-application-d94594766-h2fnq Time:21m my-application-d94594766-hfrkg Time:21m my-application-d94594766-hql4r Time:18m my-application-d94594766-jcwbq Time:21m my-application-d94594766-kgmzc Time:18m my-application-d94594766-rmcdz Time:18m my-application-d94594766-swsn7 Time:21m my-application-d94594766-t5kjr Time:21m my-application-d94594766-t9ft9 Time:18m my-application-d94594766-vbnxs Time:18m my-application-d94594766-x9v64 Time:21m my-application-d94594766-z22df Time:21m
my-application-d94594766-b2dqv Time:22m my-application-d94594766-drnxx Time:22m my-application-d94594766-fctgp Time:19m my-application-d94594766-g659w Time:22m my-application-d94594766-h2fnq Time:22m my-application-d94594766-hfrkg Time:22m my-application-d94594766-hql4r Time:19m my-application-d94594766-jcwbq Time:22m my-application-d94594766-kgmzc Time:19m my-application-d94594766-rmcdz Time:19m my-application-d94594766-swsn7 Time:22m my-application-d94594766-t5kjr Time:22m my-application-d94594766-t9ft9 Time:19m my-application-d94594766-vbnxs Time:19m my-application-d94594766-x9v64 Time:22m my-application-d94594766-z22df Time:22m
my-application-d94594766-b2dqv Time:23m my-application-d94594766-drnxx Time:23m my-application-d94594766-fctgp Time:20m my-application-d94594766-g659w Time:23m my-application-d94594766-h2fnq Time:23m my-application-d94594766-hfrkg Time:23m my-application-d94594766-hql4r Time:20m my-application-d94594766-jcwbq Time:23m my-application-d94594766-kgmzc Time:20m my-application-d94594766-rmcdz Time:20m my-application-d94594766-swsn7 Time:23m my-application-d94594766-t5kjr Time:23m my-application-d94594766-t9ft9 Time:20m my-application-d94594766-vbnxs Time:20m my-application-d94594766-x9v64 Time:23m my-application-d94594766-z22df Time:23m
my-application-d94594766-b2dqv Time:24m my-application-d94594766-drnxx Time:24m my-application-d94594766-fctgp Time:21m my-application-d94594766-g659w Time:24m my-application-d94594766-h2fnq Time:24m my-application-d94594766-hfrkg Time:24m my-application-d94594766-hql4r Time:21m my-application-d94594766-jcwbq Time:24m my-application-d94594766-kgmzc Time:21m my-application-d94594766-rmcdz Time:21m my-application-d94594766-swsn7 Time:24m my-application-d94594766-t5kjr Time:24m my-application-d94594766-t9ft9 Time:21m my-application-d94594766-vbnxs Time:21m my-application-d94594766-x9v64 Time:24m my-application-d94594766-z22df Time:24m
my-application-d94594766-b2dqv Time:25m my-application-d94594766-drnxx Time:25m my-application-d94594766-fctgp Time:22m my-application-d94594766-g659w Time:25m my-application-d94594766-h2fnq Time:25m my-application-d94594766-hfrkg Time:25m my-application-d94594766-hql4r Time:22m my-application-d94594766-jcwbq Time:25m my-application-d94594766-kgmzc Time:22m my-application-d94594766-rmcdz Time:22m my-application-d94594766-swsn7 Time:25m my-application-d94594766-t5kjr Time:25m my-application-d94594766-t9ft9 Time:22m my-application-d94594766-vbnxs Time:22m my-application-d94594766-x9v64 Time:25m my-application-d94594766-z22df Time:25m
my-application-d94594766-b2dqv Time:26m my-application-d94594766-drnxx Time:26m my-application-d94594766-fctgp Time:23m my-application-d94594766-g659w Time:26m my-application-d94594766-h2fnq Time:26m my-application-d94594766-hfrkg Time:26m my-application-d94594766-hql4r Time:23m my-application-d94594766-jcwbq Time:26m my-application-d94594766-kgmzc Time:23m my-application-d94594766-rmcdz Time:23m my-application-d94594766-swsn7 Time:26m my-application-d94594766-t5kjr Time:26m my-application-d94594766-t9ft9 Time:23m my-application-d94594766-vbnxs Time:23m my-application-d94594766-x9v64 Time:26m my-application-d94594766-z22df Time:26m
my-application-d94594766-b2dqv Time:27m my-application-d94594766-drnxx Time:27m my-application-d94594766-fctgp Time:24m my-application-d94594766-g659w Time:27m my-application-d94594766-h2fnq Time:27m my-application-d94594766-hfrkg Time:27m my-application-d94594766-hql4r Time:24m my-application-d94594766-jcwbq Time:27m my-application-d94594766-kgmzc Time:24m my-application-d94594766-rmcdz Time:24m my-application-d94594766-swsn7 Time:27m my-application-d94594766-t5kjr Time:27m my-application-d94594766-t9ft9 Time:24m my-application-d94594766-vbnxs Time:24m my-application-d94594766-x9v64 Time:27m my-application-d94594766-z22df Time:27m
my-application-d94594766-b2dqv Time:28m my-application-d94594766-drnxx Time:28m my-application-d94594766-fctgp Time:25m my-application-d94594766-g659w Time:28m my-application-d94594766-h2fnq Time:28m my-application-d94594766-hfrkg Time:28m my-application-d94594766-hql4r Time:25m my-application-d94594766-jcwbq Time:28m my-application-d94594766-kgmzc Time:25m my-application-d94594766-rmcdz Time:25m my-application-d94594766-swsn7 Time:28m my-application-d94594766-t5kjr Time:28m my-application-d94594766-t9ft9 Time:25m my-application-d94594766-vbnxs Time:25m my-application-d94594766-x9v64 Time:28m my-application-d94594766-z22df Time:28m
my-application-d94594766-b2dqv Time:29m my-application-d94594766-drnxx Time:29m my-application-d94594766-fctgp Time:26m my-application-d94594766-g659w Time:29m my-application-d94594766-h2fnq Time:29m my-application-d94594766-hfrkg Time:29m my-application-d94594766-hql4r Time:26m my-application-d94594766-jcwbq Time:29m my-application-d94594766-kgmzc Time:26m my-application-d94594766-rmcdz Time:26m my-application-d94594766-swsn7 Time:29m my-application-d94594766-t5kjr Time:29m my-application-d94594766-t9ft9 Time:26m my-application-d94594766-vbnxs Time:26m my-application-d94594766-x9v64 Time:29m my-application-d94594766-z22df Time:29m
my-application-d94594766-b2dqv Time:30m my-application-d94594766-drnxx Time:30m my-application-d94594766-fctgp Time:27m my-application-d94594766-g659w Time:30m my-application-d94594766-h2fnq Time:30m my-application-d94594766-hfrkg Time:30m my-application-d94594766-hql4r Time:27m my-application-d94594766-jcwbq Time:30m my-application-d94594766-kgmzc Time:27m my-application-d94594766-rmcdz Time:27m my-application-d94594766-swsn7 Time:30m my-application-d94594766-t5kjr Time:30m my-application-d94594766-t9ft9 Time:27m my-application-d94594766-vbnxs Time:27m my-application-d94594766-x9v64 Time:30m my-application-d94594766-z22df Time:30m
my-application-d94594766-b2dqv Time:32m my-application-d94594766-drnxx Time:32m my-application-d94594766-fctgp Time:28m my-application-d94594766-g659w Time:32m my-application-d94594766-h2fnq Time:32m my-application-d94594766-hfrkg Time:32m my-application-d94594766-hql4r Time:28m my-application-d94594766-jcwbq Time:32m my-application-d94594766-kgmzc Time:28m my-application-d94594766-rmcdz Time:28m my-application-d94594766-swsn7 Time:32m my-application-d94594766-t5kjr Time:32m my-application-d94594766-t9ft9 Time:28m my-application-d94594766-vbnxs Time:28m my-application-d94594766-x9v64 Time:32m my-application-d94594766-z22df Time:32m
my-application-d94594766-b2dqv Time:33m my-application-d94594766-drnxx Time:33m my-application-d94594766-fctgp Time:29m my-application-d94594766-g659w Time:33m my-application-d94594766-h2fnq Time:33m my-application-d94594766-hfrkg Time:33m my-application-d94594766-hql4r Time:29m my-application-d94594766-jcwbq Time:33m my-application-d94594766-kgmzc Time:29m my-application-d94594766-rmcdz Time:29m my-application-d94594766-swsn7 Time:33m my-application-d94594766-t5kjr Time:33m my-application-d94594766-t9ft9 Time:29m my-application-d94594766-vbnxs Time:29m my-application-d94594766-x9v64 Time:33m my-application-d94594766-z22df Time:33m
my-application-d94594766-b2dqv Time:34m my-application-d94594766-drnxx Time:34m my-application-d94594766-fctgp Time:30m my-application-d94594766-g659w Time:34m my-application-d94594766-h2fnq Time:34m my-application-d94594766-hfrkg Time:34m my-application-d94594766-hql4r Time:30m my-application-d94594766-jcwbq Time:34m my-application-d94594766-kgmzc Time:30m my-application-d94594766-rmcdz Time:30m my-application-d94594766-swsn7 Time:34m my-application-d94594766-t5kjr Time:34m my-application-d94594766-t9ft9 Time:30m my-application-d94594766-vbnxs Time:30m my-application-d94594766-x9v64 Time:34m my-application-d94594766-z22df Time:34m
my-application-d94594766-b2dqv Time:35m my-application-d94594766-drnxx Time:35m my-application-d94594766-fctgp Time:31m my-application-d94594766-g659w Time:35m my-application-d94594766-h2fnq Time:35m my-application-d94594766-hfrkg Time:35m my-application-d94594766-hql4r Time:31m my-application-d94594766-jcwbq Time:35m my-application-d94594766-kgmzc Time:31m my-application-d94594766-rmcdz Time:31m my-application-d94594766-swsn7 Time:35m my-application-d94594766-t5kjr Time:35m my-application-d94594766-t9ft9 Time:31m my-application-d94594766-vbnxs Time:31m my-application-d94594766-x9v64 Time:35m my-application-d94594766-z22df Time:35m
my-application-d94594766-b2dqv Time:36m my-application-d94594766-drnxx Time:36m my-application-d94594766-fctgp Time:32m my-application-d94594766-g659w Time:36m my-application-d94594766-h2fnq Time:36m my-application-d94594766-hfrkg Time:36m my-application-d94594766-hql4r Time:32m my-application-d94594766-jcwbq Time:36m my-application-d94594766-kgmzc Time:32m my-application-d94594766-rmcdz Time:32m my-application-d94594766-swsn7 Time:36m my-application-d94594766-t5kjr Time:36m my-application-d94594766-t9ft9 Time:32m my-application-d94594766-vbnxs Time:32m my-application-d94594766-x9v64 Time:36m my-application-d94594766-z22df Time:36m
my-application-d94594766-b2dqv Time:37m my-application-d94594766-drnxx Time:37m my-application-d94594766-fctgp Time:33m my-application-d94594766-g659w Time:37m my-application-d94594766-h2fnq Time:37m my-application-d94594766-hfrkg Time:37m my-application-d94594766-hql4r Time:33m my-application-d94594766-jcwbq Time:37m my-application-d94594766-kgmzc Time:33m my-application-d94594766-rmcdz Time:33m my-application-d94594766-swsn7 Time:37m my-application-d94594766-t5kjr Time:37m my-application-d94594766-t9ft9 Time:33m my-application-d94594766-vbnxs Time:33m my-application-d94594766-x9v64 Time:37m my-application-d94594766-z22df Time:37m
my-application-d94594766-b2dqv Time:38m my-application-d94594766-drnxx Time:38m my-application-d94594766-fctgp Time:34m my-application-d94594766-g659w Time:38m my-application-d94594766-h2fnq Time:38m my-application-d94594766-hfrkg Time:38m my-application-d94594766-hql4r Time:34m my-application-d94594766-jcwbq Time:38m my-application-d94594766-kgmzc Time:34m my-application-d94594766-rmcdz Time:34m my-application-d94594766-swsn7 Time:38m my-application-d94594766-t5kjr Time:38m my-application-d94594766-t9ft9 Time:34m my-application-d94594766-vbnxs Time:34m my-application-d94594766-x9v64 Time:38m my-application-d94594766-z22df Time:38m
my-application-d94594766-b2dqv Time:39m my-application-d94594766-drnxx Time:39m my-application-d94594766-fctgp Time:35m my-application-d94594766-g659w Time:39m my-application-d94594766-h2fnq Time:39m my-application-d94594766-hfrkg Time:39m my-application-d94594766-hql4r Time:35m my-application-d94594766-jcwbq Time:39m my-application-d94594766-kgmzc Time:35m my-application-d94594766-rmcdz Time:35m my-application-d94594766-swsn7 Time:39m my-application-d94594766-t5kjr Time:39m my-application-d94594766-t9ft9 Time:35m my-application-d94594766-vbnxs Time:35m my-application-d94594766-x9v64 Time:39m my-application-d94594766-z22df Time:39m
my-application-d94594766-b2dqv Time:40m my-application-d94594766-drnxx Time:40m my-application-d94594766-fctgp Time:36m my-application-d94594766-g659w Time:40m my-application-d94594766-h2fnq Time:40m my-application-d94594766-hfrkg Time:40m my-application-d94594766-hql4r Time:36m my-application-d94594766-jcwbq Time:40m my-application-d94594766-kgmzc Time:36m my-application-d94594766-rmcdz Time:36m my-application-d94594766-swsn7 Time:40m my-application-d94594766-t5kjr Time:40m my-application-d94594766-t9ft9 Time:36m my-application-d94594766-vbnxs Time:36m my-application-d94594766-x9v64 Time:40m my-application-d94594766-z22df Time:40m
my-application-d94594766-b2dqv Time:41m my-application-d94594766-drnxx Time:41m my-application-d94594766-fctgp Time:37m my-application-d94594766-g659w Time:41m my-application-d94594766-h2fnq Time:41m my-application-d94594766-hfrkg Time:41m my-application-d94594766-hql4r Time:37m my-application-d94594766-jcwbq Time:41m my-application-d94594766-kgmzc Time:37m my-application-d94594766-rmcdz Time:37m my-application-d94594766-swsn7 Time:41m my-application-d94594766-t5kjr Time:41m my-application-d94594766-t9ft9 Time:37m my-application-d94594766-vbnxs Time:37m my-application-d94594766-x9v64 Time:41m my-application-d94594766-z22df Time:41m
my-application-d94594766-b2dqv Time:42m my-application-d94594766-drnxx Time:42m my-application-d94594766-fctgp Time:38m my-application-d94594766-g659w Time:42m my-application-d94594766-h2fnq Time:42m my-application-d94594766-hfrkg Time:42m my-application-d94594766-hql4r Time:38m my-application-d94594766-jcwbq Time:42m my-application-d94594766-kgmzc Time:38m my-application-d94594766-rmcdz Time:38m my-application-d94594766-swsn7 Time:42m my-application-d94594766-t5kjr Time:42m my-application-d94594766-t9ft9 Time:38m my-application-d94594766-vbnxs Time:38m my-application-d94594766-x9v64 Time:42m my-application-d94594766-z22df Time:42m
my-application-d94594766-b2dqv Time:43m my-application-d94594766-drnxx Time:43m my-application-d94594766-fctgp Time:39m my-application-d94594766-g659w Time:43m my-application-d94594766-h2fnq Time:43m my-application-d94594766-hfrkg Time:43m my-application-d94594766-hql4r Time:39m my-application-d94594766-jcwbq Time:43m my-application-d94594766-kgmzc Time:39m my-application-d94594766-rmcdz Time:39m my-application-d94594766-swsn7 Time:43m my-application-d94594766-t5kjr Time:43m my-application-d94594766-t9ft9 Time:39m my-application-d94594766-vbnxs Time:39m my-application-d94594766-x9v64 Time:43m my-application-d94594766-z22df Time:43m
my-application-d94594766-b2dqv Time:44m my-application-d94594766-drnxx Time:44m my-application-d94594766-fctgp Time:40m my-application-d94594766-g659w Time:44m my-application-d94594766-h2fnq Time:44m my-application-d94594766-hfrkg Time:44m my-application-d94594766-hql4r Time:40m my-application-d94594766-jcwbq Time:44m my-application-d94594766-kgmzc Time:40m my-application-d94594766-rmcdz Time:40m my-application-d94594766-swsn7 Time:44m my-application-d94594766-t5kjr Time:44m my-application-d94594766-t9ft9 Time:40m my-application-d94594766-vbnxs Time:40m my-application-d94594766-x9v64 Time:44m my-application-d94594766-z22df Time:44m
my-application-d94594766-b2dqv Time:45m my-application-d94594766-drnxx Time:45m my-application-d94594766-fctgp Time:41m my-application-d94594766-g659w Time:45m my-application-d94594766-h2fnq Time:45m my-application-d94594766-hfrkg Time:45m my-application-d94594766-hql4r Time:41m my-application-d94594766-jcwbq Time:45m my-application-d94594766-kgmzc Time:41m my-application-d94594766-rmcdz Time:41m my-application-d94594766-swsn7 Time:45m my-application-d94594766-t5kjr Time:45m my-application-d94594766-t9ft9 Time:41m my-application-d94594766-vbnxs Time:41m my-application-d94594766-x9v64 Time:45m my-application-d94594766-z22df Time:45m
my-application-d94594766-b2dqv Time:46m my-application-d94594766-drnxx Time:46m my-application-d94594766-fctgp Time:42m my-application-d94594766-g659w Time:46m my-application-d94594766-h2fnq Time:46m my-application-d94594766-hfrkg Time:46m my-application-d94594766-hql4r Time:42m my-application-d94594766-jcwbq Time:46m my-application-d94594766-kgmzc Time:42m my-application-d94594766-rmcdz Time:42m my-application-d94594766-swsn7 Time:46m my-application-d94594766-t5kjr Time:46m my-application-d94594766-t9ft9 Time:42m my-application-d94594766-vbnxs Time:42m my-application-d94594766-x9v64 Time:46m my-application-d94594766-z22df Time:46m

Pods did not get deleted and respected the terminationGracePeriodSeconds, no error logs on Cluster Autoscaler and no useful logs of ASG.

Scenario 3

Application with increased amount of resource requests defined.

ASG single-az: us-east-2a

Capacity rebalance enabled (It will not have effect)

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-application
  name: my-application
  namespace: scaleout
spec:
  replicas: 20
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
      annotations:
         "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
    spec:
      containers:
      - image: brentley/ecsdemo-nodejs:latest
        name: my-application
        resources:
          requests:
            memory: "2278Mi"
            cpu: "450m"
          limits:
            memory: "2278Mi"
            cpu: "450m"
      tolerations:
      - key: "type"
        operator: "Equal"
        value: "multiaz"
        effect: "NoSchedule"
      terminationGracePeriodSeconds: 14400

Applied the manifest, CA -- > ASG spined-up 7 nodes to accomodate the pending pods in only one az.

CA Logs

I1025 15:43:18.164280       1 scale_down.go:444] Node ip-10-0-0-149.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:43:18.164305       1 scale_down.go:444] Node ip-10-0-0-101.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:43:18.164326       1 scale_down.go:444] Node ip-10-0-0-167.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:43:18.164346       1 scale_down.go:444] Node ip-10-0-0-11.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:43:18.164363       1 scale_down.go:444] Node ip-10-0-0-232.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.661151)
I1025 15:43:18.164380       1 scale_down.go:444] Node ip-10-0-0-236.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:43:18.164399       1 scale_down.go:444] Node ip-10-0-0-93.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)

Pods are using Node resources, no scale-in occurred.

Scaling down the deployment replicas from 20 to 4.

Even after scaling down, the CA logs still the same.

I1025 15:45:38.875143       1 scale_down.go:444] Node ip-10-0-0-232.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.661151)
I1025 15:45:38.875175       1 scale_down.go:444] Node ip-10-0-0-236.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:45:38.875196       1 scale_down.go:444] Node ip-10-0-0-93.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:45:38.875214       1 scale_down.go:444] Node ip-10-0-0-167.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:45:38.875233       1 scale_down.go:444] Node ip-10-0-0-11.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:45:38.875250       1 scale_down.go:444] Node ip-10-0-0-149.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)
I1025 15:45:38.875267       1 scale_down.go:444] Node ip-10-0-0-101.us-east-2.compute.internal is not suitable for removal - memory utilization too big (0.983243)

Enabling Multi-Az on ASG

After enabling multi-az on ASG, ASG will try to rebalance the Nodes to split Nodes deploy in 3 AZs.

ASG Logs

At 2022-10-25T15:47:56Z instances were launched to balance instances in zones us-east-2a us-east-2b us-east-2c with other zones resulting in more than desired number of instances in the group. At 2022-10-25T15:48:08Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 11 to 9. At 2022-10-25T15:48:08Z instance i-053fb42ec14a3b292 was selected for termination. At 2022-10-25T15:48:08Z instance i-0ef64b419983d893b was selected for termination.

Since the AZs are not balanced, the rebalance recommendation happens and the ASG spin-up instances to balance into multiple AZs. Cluster Austoscaler still respcts the memory utilization of the Terminating Pods. But since is the ASG that is terminating the instances and ASG does not have visibility into terminationGracePeriodSeconds on the Deployment manifest. It will follow the ASG lifecycle hook, terminating the EC2 instances anyways after 1800 secs.

And then the logs of the Cluster Autoscaler will be the follow:

I1025 00:30:34.448463 1 scale_down.go:667] Can't retrieve node ip-10-0-10-141.us-east-2.compute.internal from snapshot, removing from unremovable map, err: node not found

Since was not Cluster Autoscaler who scaled down that Node, it was ASG rebalance lifecycle hook.