Skip to content

Instantly share code, notes, and snippets.

@YangLu1031

YangLu1031/pods.yaml

Last active November 20, 2020 17:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save YangLu1031/a318ad5e92ae1e61102801fdb9109788 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save YangLu1031/a318ad5e92ae1e61102801fdb9109788 to your computer and use it in GitHub Desktop.
Download ZIP
Scripts to reproduce PLEG unhealthy issue
Raw
pods.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
# Do not automount default service account, to eliminate its impact.
automountServiceAccountToken: false
nodeSelector:
kubernetes.io/os: windows
containers:
- image: gcr.io/gke-release/pause-win:1.0.0
imagePullPolicy: IfNotPresent
name: pause-win
ports:
# Add not-ready/unreachable tolerations for 15 minutes so that node
# failure doesn't trigger pod deletion.
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 900
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 900
Raw
script.sh
#!/bin/bash
# Create namespaces
for pod_number in {1..30}; do
namespace=test-pod-${pod_number}
kubectl create namespace ${namespace}
done
# # Create test pods
for pod_number in {1..30}; do
namespace=test-pod-${pod_number}
kubectl apply -f pods.yaml --namespace=${namespace}
sleep 1s
done
# Delete all test pods
# for pod_number in {1..30}; do
# namespace=test-pod-${pod_number}
# kubectl delete po test-pod --namespace=${namespace}
# done
Raw
windows node status log
$ kubectl describe nodes e2e-test-yluu-windows-node-group-p7r3
Name: e2e-test-yluu-windows-node-group-p7r3
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=n1-standard-8
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=us-central1
failure-domain.beta.kubernetes.io/zone=us-central1-b
kubernetes.io/arch=amd64
kubernetes.io/hostname=e2e-test-yluu-windows-node-group-p7r3
kubernetes.io/os=windows
node.kubernetes.io/instance-type=n1-standard-8
node.kubernetes.io/windows-build=10.0.17763
topology.kubernetes.io/region=us-central1
topology.kubernetes.io/zone=us-central1-b
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 04 Feb 2020 19:11:39 -0800
Taints: node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: e2e-test-yluu-windows-node-group-p7r3
AcquireTime: <unset>
RenewTime: Fri, 07 Feb 2020 08:04:17 -0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Tue, 04 Feb 2020 19:11:39 -0800 Tue, 04 Feb 2020 19:11:39 -0800 RouteCreated NodeController create implicit route
MemoryPressure False Fri, 07 Feb 2020 08:04:21 -0800 Wed, 05 Feb 2020 06:03:47 -0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 07 Feb 2020 08:04:21 -0800 Wed, 05 Feb 2020 06:03:47 -0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 07 Feb 2020 08:04:21 -0800 Wed, 05 Feb 2020 06:03:47 -0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Fri, 07 Feb 2020 08:04:21 -0800 Fri, 07 Feb 2020 08:04:21 -0800 KubeletNotReady PLEG is not healthy: pleg was last seen active 3m8.068354s ago; threshold is 3m0s
Addresses:
InternalIP: 10.40.0.3
ExternalIP: 35.202.225.199
InternalDNS: e2e-test-yluu-windows-node-group-p7r3.c.yluu-gke-dev.internal
Hostname: e2e-test-yluu-windows-node-group-p7r3.c.yluu-gke-dev.internal
Capacity:
attachable-volumes-gce-pd: 127
cpu: 8
ephemeral-storage: 104855548Ki
memory: 31456872Ki
pods: 110
Allocatable:
attachable-volumes-gce-pd: 127
cpu: 8
ephemeral-storage: 96634872877
memory: 31200872Ki
pods: 110
System Info:
Machine ID: e2e-test-yluu-windows-node-group-p7r3
System UUID: DD82CDB7-1427-1EAD-A326-6747F5B1CED7
Boot ID:
Kernel Version: 10.0.17763.615
OS Image: Windows Server Datacenter
Operating System: windows
Architecture: amd64
Container Runtime Version: docker://18.9.7
Kubelet Version: v1.18.0-alpha.1.59+06f9227ae08821-dirty
Kube-Proxy Version: v1.18.0-alpha.1.59+06f9227ae08821-dirty
PodCIDR: 10.64.1.0/24
PodCIDRs: 10.64.1.0/24
ProviderID: gce://yluu-gke-dev/us-central1-b/e2e-test-yluu-windows-node-group-p7r3
Non-terminated Pods: (28 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
test-pod-1 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m36s
test-pod-10 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m24s
test-pod-11 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m22s
test-pod-12 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m21s
test-pod-13 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m20s
test-pod-14 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m18s
test-pod-15 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m17s
test-pod-16 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m15s
test-pod-17 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m14s
test-pod-18 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m12s
test-pod-19 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m10s
test-pod-2 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m35s
test-pod-20 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m9s
test-pod-21 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m7s
test-pod-22 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m6s
test-pod-24 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m3s
test-pod-26 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m42s
test-pod-27 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m41s
test-pod-28 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m39s
test-pod-29 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m38s
test-pod-3 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m34s
test-pod-30 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m37s
test-pod-4 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m32s
test-pod-5 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m31s
test-pod-6 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m29s
test-pod-7 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m28s
test-pod-8 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m27s
test-pod-9 test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m25s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-gce-pd 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ContainerGCFailed 38m (x8 over 2d1h) kubelet, e2e-test-yluu-windows-node-group-p7r3 rpc error: code = DeadlineExceeded desc = context deadline exceeded
Normal NodeReady 34m (x24 over 2d2h) kubelet, e2e-test-yluu-windows-node-group-p7r3 Node e2e-test-yluu-windows-node-group-p7r3 status is now: NodeReady
Normal NodeNotReady 3s (x24 over 2d1h) kubelet, e2e-test-yluu-windows-node-group-p7r3 Node e2e-test-yluu-windows-node-group-p7r3 status is now: NodeNotReady
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment