Skip to content

Instantly share code, notes, and snippets.

@venkatsc

venkatsc/CSI-E2E-test-fail.log

Created July 13, 2020 14:29
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save venkatsc/00828baf4e5fe6165c76d53fdf3cbc66 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save venkatsc/00828baf4e5fe6165c76d53fdf3cbc66 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
CSI-E2E-test-fail.log
External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
Verify if offline PVC expansion works
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:159
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:101
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:90
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:151
STEP: Creating a kubernetes client
Jul 13 13:30:13.574: INFO: >>> kubeConfig: /home/vagrant/.kube/config
STEP: Building a namespace api object, basename volume-expand
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in volume-expand-1006
STEP: Waiting for a default service account to be provisioned in namespace
[It] Verify if offline PVC expansion works
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:159
Jul 13 13:30:14.209: INFO: Test running for native CSI Driver, not checking metrics
Jul 13 13:30:14.209: INFO: Creating resource for dynamic PV
Jul 13 13:30:14.209: INFO: Using claimSize:1Mi, test suite supported size:{ 1Mi}, driver(csi.quobyte.com) supported size:{ 1Mi}
STEP: creating a StorageClass quobyte-csi-volume-expand-1006-bncpl
STEP: creating a claim
Jul 13 13:30:14.218: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Jul 13 13:30:14.228: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi.quobyte.com44vwf] to have phase Bound
Jul 13 13:30:14.235: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found but phase is Pending instead of Bound.
Jul 13 13:30:16.238: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found but phase is Pending instead of Bound.
Jul 13 13:30:18.254: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found but phase is Pending instead of Bound.
Jul 13 13:30:20.268: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found and phase=Bound (6.040557515s)
STEP: Creating a pod with dynamically provisioned volume
STEP: Deleting the previously created pod
Jul 13 13:30:22.338: INFO: Deleting pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" in namespace "volume-expand-1006"
Jul 13 13:30:22.360: INFO: Wait up to 5m0s for pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" to be fully deleted
STEP: Expanding current pvc
Jul 13 13:30:36.372: INFO: currentPvcSize {{1048576 0} {<nil>} 1Mi BinarySI}, newSize {{1074790400 0} {<nil>} BinarySI}
STEP: Waiting for cloudprovider resize to finish
STEP: Checking for conditions on pvc
STEP: Creating a new pod with same volume
STEP: Waiting for file system resize to finish
Jul 13 13:30:40.450: INFO: Deleting pod "security-context-c1add684-c2d6-43e1-85f1-c98a9090250c" in namespace "volume-expand-1006"
Jul 13 13:30:40.457: INFO: Wait up to 5m0s for pod "security-context-c1add684-c2d6-43e1-85f1-c98a9090250c" to be fully deleted
Jul 13 13:30:44.466: INFO: Deleting pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" in namespace "volume-expand-1006"
STEP: Deleting pod
Jul 13 13:30:44.469: INFO: Deleting pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" in namespace "volume-expand-1006"
STEP: Deleting pod2
Jul 13 13:30:44.472: INFO: Deleting pod "security-context-c1add684-c2d6-43e1-85f1-c98a9090250c" in namespace "volume-expand-1006"
STEP: Deleting pvc
Jul 13 13:30:44.474: INFO: Deleting PersistentVolumeClaim "csi.quobyte.com44vwf"
Jul 13 13:30:44.479: INFO: Waiting up to 5m0s for PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 to get deleted
Jul 13 13:30:44.482: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Bound (2.311771ms)
Jul 13 13:30:49.494: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (5.014293413s)
Jul 13 13:30:54.508: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (10.027941582s)
Jul 13 13:30:59.522: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (15.041949873s)
Jul 13 13:31:04.535: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (20.054948498s)
Jul 13 13:31:09.546: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (25.066617486s)
Jul 13 13:31:14.560: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (30.080377493s)
Jul 13 13:31:19.567: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (35.087021627s)
Jul 13 13:31:24.575: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (40.095202744s)
Jul 13 13:31:29.593: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (45.113054681s)
Jul 13 13:31:34.610: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (50.130130446s)
Jul 13 13:31:39.621: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (55.14098436s)
Jul 13 13:31:44.624: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m0.144800009s)
Jul 13 13:31:49.643: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m5.163372482s)
Jul 13 13:31:54.646: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m10.166611749s)
Jul 13 13:32:00.163: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m15.68367416s)
Jul 13 13:32:05.175: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m20.695581007s)
Jul 13 13:32:10.185: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m25.705022276s)
Jul 13 13:32:15.195: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m30.715756976s)
Jul 13 13:32:20.208: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m35.727970127s)
Jul 13 13:32:25.212: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m40.732298336s)
Jul 13 13:32:30.215: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m45.735349723s)
Jul 13 13:32:35.225: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m50.745596133s)
Jul 13 13:32:40.236: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m55.755821762s)
Jul 13 13:32:45.245: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m0.765257272s)
Jul 13 13:32:50.256: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m5.776014162s)
Jul 13 13:32:55.259: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m10.779461019s)
Jul 13 13:33:00.263: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m15.783433251s)
Jul 13 13:33:05.265: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m20.785766731s)
Jul 13 13:33:10.277: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m25.797562069s)
Jul 13 13:33:15.284: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m30.80459502s)
Jul 13 13:33:20.298: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m35.818629586s)
Jul 13 13:33:25.308: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m40.82829748s)
Jul 13 13:33:30.321: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m45.841160505s)
Jul 13 13:33:35.326: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m50.845980634s)
Jul 13 13:33:40.337: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m55.856988685s)
Jul 13 13:33:45.350: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m0.870328902s)
Jul 13 13:33:50.366: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m5.886271883s)
Jul 13 13:33:55.378: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m10.898017478s)
Jul 13 13:34:00.389: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m15.909448667s)
Jul 13 13:34:05.401: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m20.921007385s)
Jul 13 13:34:10.412: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m25.9322858s)
Jul 13 13:34:15.421: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m30.941222938s)
Jul 13 13:34:20.429: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m35.949774337s)
Jul 13 13:34:25.442: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m40.962296852s)
Jul 13 13:34:30.445: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m45.965424441s)
Jul 13 13:34:35.465: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m50.985649916s)
Jul 13 13:34:40.479: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m55.999020108s)
Jul 13 13:34:45.491: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m1.011078711s)
Jul 13 13:34:50.503: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m6.023299061s)
Jul 13 13:34:55.515: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m11.035180659s)
Jul 13 13:35:00.523: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m16.043175186s)
Jul 13 13:35:05.528: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m21.048423038s)
Jul 13 13:35:10.537: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m26.057075107s)
Jul 13 13:35:15.551: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m31.071056575s)
Jul 13 13:35:20.562: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m36.082605232s)
Jul 13 13:35:25.576: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m41.096550091s)
Jul 13 13:35:30.588: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m46.108133959s)
Jul 13 13:35:35.598: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m51.118676807s)
Jul 13 13:35:40.604: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m56.124041072s)
STEP: Deleting sc
Jul 13 13:35:45.636: FAIL: while cleaning up resource
Unexpected error:
<errors.aggregate | len:1, cap:1>: [
[
{
error: {
cause: {
s: "PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s",
},
msg: "Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner",
},
stack: [0x378a2d9, 0x37bff49, 0x37c1b28, 0x7d1e98, 0x7d1aef, 0x7d0f94, 0x7d7ef5, 0x7d7751, 0x7dd5cf, 0x7dd0f0, 0x7dc937, 0x7def1b, 0x7e1a77, 0x7e17bd, 0x38b1baa, 0x38b5ecb, 0x5165d9, 0x462e41],
},
],
]
Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s
occurred
[AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:152
STEP: Collecting events from namespace "volume-expand-1006".
STEP: Found 16 events.
Jul 13 13:35:45.646: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {default-scheduler } Scheduled: Successfully assigned volume-expand-1006/security-context-c1add684-c2d6-43e1-85f1-c98a9090250c to k8s-1
Jul 13 13:35:45.646: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {default-scheduler } Scheduled: Successfully assigned volume-expand-1006/security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d to k8s-1
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:14 +0000 UTC - event for csi.quobyte.com44vwf: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi.quobyte.com" or manually created by system administrator
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:14 +0000 UTC - event for csi.quobyte.com44vwf: {csi.quobyte.com_quobyte-csi-controller-0_53c8d682-983a-4335-980a-17f8a9293278 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-1006/csi.quobyte.com44vwf"
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:19 +0000 UTC - event for csi.quobyte.com44vwf: {csi.quobyte.com_quobyte-csi-controller-0_53c8d682-983a-4335-980a-17f8a9293278 } ProvisioningSucceeded: Successfully provisioned volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:21 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Started: Started container write-pod
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:21 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Created: Created container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:21 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Pulled: Container image "docker.io/library/busybox:1.29" already present on machine
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:23 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Killing: Stopping container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:36 +0000 UTC - event for csi.quobyte.com44vwf: {external-resizer csi.quobyte.com } VolumeResizeSuccessful: Resize volume succeeded
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:36 +0000 UTC - event for csi.quobyte.com44vwf: {external-resizer csi.quobyte.com } Resizing: External resizer is resizing volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:36 +0000 UTC - event for csi.quobyte.com44vwf: {volume_expand } ExternalExpanding: Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:39 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Started: Started container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:39 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Pulled: Container image "docker.io/library/busybox:1.29" already present on machine
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:39 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Created: Created container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:41 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Killing: Stopping container write-pod
Jul 13 13:35:45.652: INFO: POD NODE PHASE GRACE CONDITIONS
Jul 13 13:35:45.652: INFO:
Jul 13 13:35:45.657: INFO:
Logging node info for node k8s-1
Jul 13 13:35:45.661: INFO: Node Info: &Node{ObjectMeta:{k8s-1 /api/v1/nodes/k8s-1 2558a9b7-5e65-4a8f-8fa9-26dc3bbe1400 14852 0 2020-07-13 12:32:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-1 kubernetes.io/os:linux node-role.kubernetes.io/master:] map[csi.volume.kubernetes.io/nodeid:{"csi.quobyte.com":"k8s-1"} flannel.alpha.coreos.com/backend-data:{"VtepMAC":"0a:e1:3e:8d:14:61"} flannel.alpha.coreos.com/backend-type:vxlan flannel.alpha.coreos.com/kube-subnet-manager:true flannel.alpha.coreos.com/public-ip:172.18.8.101 kubeadm.alpha.kubernetes.io/cri-socket:/var/run/dockershim.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.233.64.0/24,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.233.64.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{42927656960 0} {<nil>} 41921540Ki BinarySI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8099856384 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{7800 -3} {<nil>} 7800m DecimalSI},ephemeral-storage: {{38634891201 0} {<nil>} 38634891201 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7482998784 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2020-07-13 13:10:01 +0000 UTC,LastTransitionTime:2020-07-13 13:10:01 +0000 UTC,Reason:FlannelIsUp,Message:Flannel is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:32:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:32:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:32:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:34:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.8.101,},NodeAddress{Type:Hostname,Address:k8s-1,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:876b5f19db854cfb89eca227a0d35fdf,SystemUUID:876B5F19-DB85-4CFB-89EC-A227A0D35FDF,BootID:cf6406a5-f0ec-4dd5-9ec2-94072923bec3,KernelVersion:3.10.0-957.12.2.el7.x86_64,OSImage:CentOS Linux 7 (Core),ContainerRuntimeVersion:docker://18.9.9,KubeletVersion:v1.17.7,KubeProxyVersion:v1.17.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[nfvpe/multus:v3.4.1],SizeBytes:275759905,},ContainerImage{Names:[kubernetesui/dashboard-amd64:v2.0.0],SizeBytes:221895031,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.17.7],SizeBytes:171088667,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.17.7],SizeBytes:160979739,},ContainerImage{Names:[nginx:1.17],SizeBytes:126698063,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.17.7],SizeBytes:116521281,},ContainerImage{Names:[quay.io/quobyte/csi@sha256:e00d3ab3965671ca30fb729da885f0575ad59fd41b8d7640b35db22254d0dad5 quay.io/quobyte/csi:v1.0.5],SizeBytes:107251602,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-node-cache:1.15.12],SizeBytes:106950843,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.17.7],SizeBytes:94436123,},ContainerImage{Names:[calico/kube-controllers:v3.13.2],SizeBytes:56577289,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:daf5332100521b1256d0e3c56d697a238eaec3af48897ed9167cbadd426773b5 gcr.io/kubernetes-e2e-test-images/agnhost:2.8],SizeBytes:52800335,},ContainerImage{Names:[quay.io/coreos/flannel:v0.12.0],SizeBytes:52767393,},ContainerImage{Names:[coredns/coredns:1.6.5],SizeBytes:41578211,},ContainerImage{Names:[quay.io/coreos/etcd:v3.3.12],SizeBytes:40584227,},ContainerImage{Names:[k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1],SizeBytes:40067731,},ContainerImage{Names:[kubernetesui/metrics-scraper:v1.0.4],SizeBytes:36937728,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:9622c6a6dac7499a055a382930f4de82905a3c5735c0753f7094115c9c871309 quay.io/k8scsi/csi-node-driver-registrar:v1.3.0],SizeBytes:16322467,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jul 13 13:35:45.661: INFO:
Logging kubelet events for node k8s-1
Jul 13 13:35:45.670: INFO:
Logging pods the kubelet thinks is on node k8s-1
Jul 13 13:35:45.681: INFO: kube-flannel-7g2gj started at 2020-07-13 12:34:50 +0000 UTC (1+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Init container install-cni ready: true, restart count 1
Jul 13 13:35:45.681: INFO: Container kube-flannel ready: true, restart count 1
Jul 13 13:35:45.681: INFO: nodelocaldns-cfgvx started at 2020-07-13 12:35:09 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container node-cache ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kubernetes-dashboard-77475cf576-54lds started at 2020-07-13 12:35:10 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kubernetes-dashboard ready: true, restart count 1
Jul 13 13:35:45.681: INFO: coredns-76798d84dd-8qtv7 started at 2020-07-13 12:35:10 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container coredns ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-proxy-zzrz5 started at 2020-07-13 12:34:38 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-proxy ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-controller-manager-k8s-1 started at 2020-07-13 13:09:55 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-controller-manager ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-scheduler-k8s-1 started at 2020-07-13 13:09:55 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-scheduler ready: true, restart count 1
Jul 13 13:35:45.681: INFO: dns-autoscaler-85f898cd5c-7zbrx started at 2020-07-13 12:35:08 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container autoscaler ready: true, restart count 1
Jul 13 13:35:45.681: INFO: quobyte-csi-node-pf65l started at 2020-07-13 12:43:01 +0000 UTC (0+2 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container csi-node-driver-registrar ready: true, restart count 1
Jul 13 13:35:45.681: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-apiserver-k8s-1 started at 2020-07-13 13:09:55 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-apiserver ready: true, restart count 1
W0713 13:35:45.684739 16162 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
Jul 13 13:35:45.700: INFO:
Latency metrics for node k8s-1
Jul 13 13:35:45.700: INFO:
Logging node info for node k8s-2
Jul 13 13:35:45.703: INFO: Node Info: &Node{ObjectMeta:{k8s-2 /api/v1/nodes/k8s-2 61b9b3fa-43c3-4c66-a608-38a72da236e9 14868 0 2020-07-13 12:33:01 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-2 kubernetes.io/os:linux node-role.kubernetes.io/master:] map[csi.volume.kubernetes.io/nodeid:{"csi.quobyte.com":"k8s-2"} flannel.alpha.coreos.com/backend-data:{"VtepMAC":"1e:62:0e:23:84:97"} flannel.alpha.coreos.com/backend-type:vxlan flannel.alpha.coreos.com/kube-subnet-manager:true flannel.alpha.coreos.com/public-ip:172.18.8.102 kubeadm.alpha.kubernetes.io/cri-socket:/var/run/dockershim.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.233.65.0/24,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.233.65.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{42927656960 0} {<nil>} 41921540Ki BinarySI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8099856384 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{7800 -3} {<nil>} 7800m DecimalSI},ephemeral-storage: {{38634891201 0} {<nil>} 38634891201 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7482998784 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2020-07-13 13:10:23 +0000 UTC,LastTransitionTime:2020-07-13 13:10:23 +0000 UTC,Reason:FlannelIsUp,Message:Flannel is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:33:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:33:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:33:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:35:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.8.102,},NodeAddress{Type:Hostname,Address:k8s-2,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6b6c1bdd969646f3b9ddb1ff366d8ee5,SystemUUID:6B6C1BDD-9696-46F3-B9DD-B1FF366D8EE5,BootID:824a72c9-f28c-457d-861b-bfcf5ed83337,KernelVersion:3.10.0-957.12.2.el7.x86_64,OSImage:CentOS Linux 7 (Core),ContainerRuntimeVersion:docker://18.9.9,KubeletVersion:v1.17.7,KubeProxyVersion:v1.17.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[nfvpe/multus:v3.4.1],SizeBytes:275759905,},ContainerImage{Names:[kubernetesui/dashboard-amd64:v2.0.0],SizeBytes:221895031,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.17.7],SizeBytes:171088667,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.17.7],SizeBytes:160979739,},ContainerImage{Names:[nginx:1.17],SizeBytes:126698063,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.17.7],SizeBytes:116521281,},ContainerImage{Names:[quay.io/quobyte/csi@sha256:e00d3ab3965671ca30fb729da885f0575ad59fd41b8d7640b35db22254d0dad5 quay.io/quobyte/csi:v1.0.5],SizeBytes:107251602,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-node-cache:1.15.12],SizeBytes:106950843,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.17.7],SizeBytes:94436123,},ContainerImage{Names:[calico/kube-controllers:v3.13.2],SizeBytes:56577289,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:daf5332100521b1256d0e3c56d697a238eaec3af48897ed9167cbadd426773b5 gcr.io/kubernetes-e2e-test-images/agnhost:2.8],SizeBytes:52800335,},ContainerImage{Names:[quay.io/coreos/flannel:v0.12.0],SizeBytes:52767393,},ContainerImage{Names:[coredns/coredns:1.6.5],SizeBytes:41578211,},ContainerImage{Names:[quay.io/coreos/etcd:v3.3.12],SizeBytes:40584227,},ContainerImage{Names:[k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1],SizeBytes:40067731,},ContainerImage{Names:[kubernetesui/metrics-scraper:v1.0.4],SizeBytes:36937728,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:9622c6a6dac7499a055a382930f4de82905a3c5735c0753f7094115c9c871309 quay.io/k8scsi/csi-node-driver-registrar:v1.3.0],SizeBytes:16322467,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jul 13 13:35:45.703: INFO:
Logging kubelet events for node k8s-2
Jul 13 13:35:45.709: INFO:
Logging pods the kubelet thinks is on node k8s-2
Jul 13 13:35:45.720: INFO: kube-proxy-vtgsz started at 2020-07-13 12:34:38 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-proxy ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kubernetes-metrics-scraper-747b4fd5cd-8rxzm started at 2020-07-13 12:35:10 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kubernetes-metrics-scraper ready: true, restart count 1
Jul 13 13:35:45.720: INFO: quobyte-csi-node-pc7gf started at 2020-07-13 12:43:00 +0000 UTC (0+2 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container csi-node-driver-registrar ready: true, restart count 1
Jul 13 13:35:45.720: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-apiserver-k8s-2 started at 2020-07-13 12:33:45 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-apiserver ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-controller-manager-k8s-2 started at 2020-07-13 13:09:59 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-controller-manager ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-scheduler-k8s-2 started at 2020-07-13 13:09:59 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-scheduler ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-flannel-mkglw started at 2020-07-13 12:34:50 +0000 UTC (1+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Init container install-cni ready: true, restart count 1
Jul 13 13:35:45.720: INFO: Container kube-flannel ready: true, restart count 2
Jul 13 13:35:45.720: INFO: coredns-76798d84dd-mw5gk started at 2020-07-13 12:35:07 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container coredns ready: true, restart count 1
Jul 13 13:35:45.720: INFO: nodelocaldns-8wbl5 started at 2020-07-13 12:35:09 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container node-cache ready: true, restart count 1
W0713 13:35:45.723495 16162 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
Jul 13 13:35:46.152: INFO:
Latency metrics for node k8s-2
Jul 13 13:35:46.152: INFO:
Logging node info for node k8s-3
Jul 13 13:35:46.158: INFO: Node Info: &Node{ObjectMeta:{k8s-3 /api/v1/nodes/k8s-3 39056001-a933-4135-9790-c583244410e9 14877 0 2020-07-13 12:34:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-3 kubernetes.io/os:linux] map[csi.volume.kubernetes.io/nodeid:{"csi.quobyte.com":"k8s-3"} flannel.alpha.coreos.com/backend-data:{"VtepMAC":"be:a4:2d:98:e4:63"} flannel.alpha.coreos.com/backend-type:vxlan flannel.alpha.coreos.com/kube-subnet-manager:true flannel.alpha.coreos.com/public-ip:172.18.8.103 kubeadm.alpha.kubernetes.io/cri-socket:/var/run/dockershim.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.233.66.0/24,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.233.66.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{42927656960 0} {<nil>} 41921540Ki BinarySI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8099856384 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{7900 -3} {<nil>} 7900m DecimalSI},ephemeral-storage: {{38634891201 0} {<nil>} 38634891201 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7738998784 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2020-07-13 13:10:03 +0000 UTC,LastTransitionTime:2020-07-13 13:10:03 +0000 UTC,Reason:FlannelIsUp,Message:Flannel is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 12:34:36 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 12:34:36 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message: External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
Verify if offline PVC expansion works
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:159
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:101
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:90
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:151
STEP: Creating a kubernetes client
Jul 13 13:30:13.574: INFO: >>> kubeConfig: /home/vagrant/.kube/config
STEP: Building a namespace api object, basename volume-expand
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in volume-expand-1006
STEP: Waiting for a default service account to be provisioned in namespace
[It] Verify if offline PVC expansion works
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:159
Jul 13 13:30:14.209: INFO: Test running for native CSI Driver, not checking metrics
Jul 13 13:30:14.209: INFO: Creating resource for dynamic PV
Jul 13 13:30:14.209: INFO: Using claimSize:1Mi, test suite supported size:{ 1Mi}, driver(csi.quobyte.com) supported size:{ 1Mi}
STEP: creating a StorageClass quobyte-csi-volume-expand-1006-bncpl
STEP: creating a claim
Jul 13 13:30:14.218: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Jul 13 13:30:14.228: INFO: Waiting up to 5m0s for PersistentVolumeClaims [csi.quobyte.com44vwf] to have phase Bound
Jul 13 13:30:14.235: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found but phase is Pending instead of Bound.
Jul 13 13:30:16.238: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found but phase is Pending instead of Bound.
Jul 13 13:30:18.254: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found but phase is Pending instead of Bound.
Jul 13 13:30:20.268: INFO: PersistentVolumeClaim csi.quobyte.com44vwf found and phase=Bound (6.040557515s)
STEP: Creating a pod with dynamically provisioned volume
STEP: Deleting the previously created pod
Jul 13 13:30:22.338: INFO: Deleting pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" in namespace "volume-expand-1006"
Jul 13 13:30:22.360: INFO: Wait up to 5m0s for pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" to be fully deleted
STEP: Expanding current pvc
Jul 13 13:30:36.372: INFO: currentPvcSize {{1048576 0} {<nil>} 1Mi BinarySI}, newSize {{1074790400 0} {<nil>} BinarySI}
STEP: Waiting for cloudprovider resize to finish
STEP: Checking for conditions on pvc
STEP: Creating a new pod with same volume
STEP: Waiting for file system resize to finish
Jul 13 13:30:40.450: INFO: Deleting pod "security-context-c1add684-c2d6-43e1-85f1-c98a9090250c" in namespace "volume-expand-1006"
Jul 13 13:30:40.457: INFO: Wait up to 5m0s for pod "security-context-c1add684-c2d6-43e1-85f1-c98a9090250c" to be fully deleted
Jul 13 13:30:44.466: INFO: Deleting pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" in namespace "volume-expand-1006"
STEP: Deleting pod
Jul 13 13:30:44.469: INFO: Deleting pod "security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d" in namespace "volume-expand-1006"
STEP: Deleting pod2
Jul 13 13:30:44.472: INFO: Deleting pod "security-context-c1add684-c2d6-43e1-85f1-c98a9090250c" in namespace "volume-expand-1006"
STEP: Deleting pvc
** Jul 13 13:30:44.474: INFO: Deleting PersistentVolumeClaim "csi.quobyte.com44vwf"
Jul 13 13:30:44.479: INFO: Waiting up to 5m0s for PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 to get deleted
Jul 13 13:30:44.482: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Bound (2.311771ms)
Jul 13 13:30:49.494: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (5.014293413s)**
Jul 13 13:30:54.508: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (10.027941582s)
Jul 13 13:30:59.522: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (15.041949873s)
Jul 13 13:31:04.535: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (20.054948498s)
Jul 13 13:31:09.546: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (25.066617486s)
Jul 13 13:31:14.560: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (30.080377493s)
Jul 13 13:31:19.567: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (35.087021627s)
Jul 13 13:31:24.575: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (40.095202744s)
Jul 13 13:31:29.593: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (45.113054681s)
Jul 13 13:31:34.610: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (50.130130446s)
Jul 13 13:31:39.621: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (55.14098436s)
Jul 13 13:31:44.624: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m0.144800009s)
Jul 13 13:31:49.643: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m5.163372482s)
Jul 13 13:31:54.646: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m10.166611749s)
Jul 13 13:32:00.163: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m15.68367416s)
Jul 13 13:32:05.175: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m20.695581007s)
Jul 13 13:32:10.185: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m25.705022276s)
Jul 13 13:32:15.195: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m30.715756976s)
Jul 13 13:32:20.208: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m35.727970127s)
Jul 13 13:32:25.212: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m40.732298336s)
Jul 13 13:32:30.215: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m45.735349723s)
Jul 13 13:32:35.225: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m50.745596133s)
Jul 13 13:32:40.236: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (1m55.755821762s)
Jul 13 13:32:45.245: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m0.765257272s)
Jul 13 13:32:50.256: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m5.776014162s)
Jul 13 13:32:55.259: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m10.779461019s)
Jul 13 13:33:00.263: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m15.783433251s)
Jul 13 13:33:05.265: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m20.785766731s)
Jul 13 13:33:10.277: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m25.797562069s)
Jul 13 13:33:15.284: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m30.80459502s)
Jul 13 13:33:20.298: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m35.818629586s)
Jul 13 13:33:25.308: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m40.82829748s)
Jul 13 13:33:30.321: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m45.841160505s)
Jul 13 13:33:35.326: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m50.845980634s)
Jul 13 13:33:40.337: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (2m55.856988685s)
Jul 13 13:33:45.350: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m0.870328902s)
Jul 13 13:33:50.366: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m5.886271883s)
Jul 13 13:33:55.378: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m10.898017478s)
Jul 13 13:34:00.389: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m15.909448667s)
Jul 13 13:34:05.401: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m20.921007385s)
Jul 13 13:34:10.412: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m25.9322858s)
Jul 13 13:34:15.421: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m30.941222938s)
Jul 13 13:34:20.429: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m35.949774337s)
Jul 13 13:34:25.442: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m40.962296852s)
Jul 13 13:34:30.445: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m45.965424441s)
Jul 13 13:34:35.465: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m50.985649916s)
Jul 13 13:34:40.479: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (3m55.999020108s)
Jul 13 13:34:45.491: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m1.011078711s)
Jul 13 13:34:50.503: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m6.023299061s)
Jul 13 13:34:55.515: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m11.035180659s)
Jul 13 13:35:00.523: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m16.043175186s)
Jul 13 13:35:05.528: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m21.048423038s)
Jul 13 13:35:10.537: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m26.057075107s)
Jul 13 13:35:15.551: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m31.071056575s)
Jul 13 13:35:20.562: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m36.082605232s)
Jul 13 13:35:25.576: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m41.096550091s)
Jul 13 13:35:30.588: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m46.108133959s)
Jul 13 13:35:35.598: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m51.118676807s)
Jul 13 13:35:40.604: INFO: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 found and phase=Released (4m56.124041072s)
STEP: Deleting sc
Jul 13 13:35:45.636: FAIL: while cleaning up resource
Unexpected error:
<errors.aggregate | len:1, cap:1>: [
[
{
error: {
cause: {
s: "PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s",
},
msg: "Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner",
},
stack: [0x378a2d9, 0x37bff49, 0x37c1b28, 0x7d1e98, 0x7d1aef, 0x7d0f94, 0x7d7ef5, 0x7d7751, 0x7dd5cf, 0x7dd0f0, 0x7dc937, 0x7def1b, 0x7e1a77, 0x7e17bd, 0x38b1baa, 0x38b5ecb, 0x5165d9, 0x462e41],
},
],
]
Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s
occurred
[AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:152
STEP: Collecting events from namespace "volume-expand-1006".
STEP: Found 16 events.
Jul 13 13:35:45.646: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {default-scheduler } Scheduled: Successfully assigned volume-expand-1006/security-context-c1add684-c2d6-43e1-85f1-c98a9090250c to k8s-1
Jul 13 13:35:45.646: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {default-scheduler } Scheduled: Successfully assigned volume-expand-1006/security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d to k8s-1
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:14 +0000 UTC - event for csi.quobyte.com44vwf: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "csi.quobyte.com" or manually created by system administrator
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:14 +0000 UTC - event for csi.quobyte.com44vwf: {csi.quobyte.com_quobyte-csi-controller-0_53c8d682-983a-4335-980a-17f8a9293278 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-1006/csi.quobyte.com44vwf"
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:19 +0000 UTC - event for csi.quobyte.com44vwf: {csi.quobyte.com_quobyte-csi-controller-0_53c8d682-983a-4335-980a-17f8a9293278 } ProvisioningSucceeded: Successfully provisioned volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:21 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Started: Started container write-pod
Jul 13 13:35:45.646: INFO: At 2020-07-13 13:30:21 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Created: Created container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:21 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Pulled: Container image "docker.io/library/busybox:1.29" already present on machine
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:23 +0000 UTC - event for security-context-ea8c6acf-65cb-4fde-9764-1ebaa259ab6d: {kubelet k8s-1} Killing: Stopping container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:36 +0000 UTC - event for csi.quobyte.com44vwf: {external-resizer csi.quobyte.com } VolumeResizeSuccessful: Resize volume succeeded
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:36 +0000 UTC - event for csi.quobyte.com44vwf: {external-resizer csi.quobyte.com } Resizing: External resizer is resizing volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:36 +0000 UTC - event for csi.quobyte.com44vwf: {volume_expand } ExternalExpanding: Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:39 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Started: Started container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:39 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Pulled: Container image "docker.io/library/busybox:1.29" already present on machine
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:39 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Created: Created container write-pod
Jul 13 13:35:45.647: INFO: At 2020-07-13 13:30:41 +0000 UTC - event for security-context-c1add684-c2d6-43e1-85f1-c98a9090250c: {kubelet k8s-1} Killing: Stopping container write-pod
Jul 13 13:35:45.652: INFO: POD NODE PHASE GRACE CONDITIONS
Jul 13 13:35:45.652: INFO:
Jul 13 13:35:45.657: INFO:
Logging node info for node k8s-1
Jul 13 13:35:45.661: INFO: Node Info: &Node{ObjectMeta:{k8s-1 /api/v1/nodes/k8s-1 2558a9b7-5e65-4a8f-8fa9-26dc3bbe1400 14852 0 2020-07-13 12:32:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-1 kubernetes.io/os:linux node-role.kubernetes.io/master:] map[csi.volume.kubernetes.io/nodeid:{"csi.quobyte.com":"k8s-1"} flannel.alpha.coreos.com/backend-data:{"VtepMAC":"0a:e1:3e:8d:14:61"} flannel.alpha.coreos.com/backend-type:vxlan flannel.alpha.coreos.com/kube-subnet-manager:true flannel.alpha.coreos.com/public-ip:172.18.8.101 kubeadm.alpha.kubernetes.io/cri-socket:/var/run/dockershim.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.233.64.0/24,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.233.64.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{42927656960 0} {<nil>} 41921540Ki BinarySI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8099856384 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{7800 -3} {<nil>} 7800m DecimalSI},ephemeral-storage: {{38634891201 0} {<nil>} 38634891201 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7482998784 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2020-07-13 13:10:01 +0000 UTC,LastTransitionTime:2020-07-13 13:10:01 +0000 UTC,Reason:FlannelIsUp,Message:Flannel is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:32:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:32:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:32:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2020-07-13 13:35:36 +0000 UTC,LastTransitionTime:2020-07-13 12:34:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.8.101,},NodeAddress{Type:Hostname,Address:k8s-1,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:876b5f19db854cfb89eca227a0d35fdf,SystemUUID:876B5F19-DB85-4CFB-89EC-A227A0D35FDF,BootID:cf6406a5-f0ec-4dd5-9ec2-94072923bec3,KernelVersion:3.10.0-957.12.2.el7.x86_64,OSImage:CentOS Linux 7 (Core),ContainerRuntimeVersion:docker://18.9.9,KubeletVersion:v1.17.7,KubeProxyVersion:v1.17.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[nfvpe/multus:v3.4.1],SizeBytes:275759905,},ContainerImage{Names:[kubernetesui/dashboard-amd64:v2.0.0],SizeBytes:221895031,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.17.7],SizeBytes:171088667,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.17.7],SizeBytes:160979739,},ContainerImage{Names:[nginx:1.17],SizeBytes:126698063,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.17.7],SizeBytes:116521281,},ContainerImage{Names:[quay.io/quobyte/csi@sha256:e00d3ab3965671ca30fb729da885f0575ad59fd41b8d7640b35db22254d0dad5 quay.io/quobyte/csi:v1.0.5],SizeBytes:107251602,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-node-cache:1.15.12],SizeBytes:106950843,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.17.7],SizeBytes:94436123,},ContainerImage{Names:[calico/kube-controllers:v3.13.2],SizeBytes:56577289,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:daf5332100521b1256d0e3c56d697a238eaec3af48897ed9167cbadd426773b5 gcr.io/kubernetes-e2e-test-images/agnhost:2.8],SizeBytes:52800335,},ContainerImage{Names:[quay.io/coreos/flannel:v0.12.0],SizeBytes:52767393,},ContainerImage{Names:[coredns/coredns:1.6.5],SizeBytes:41578211,},ContainerImage{Names:[quay.io/coreos/etcd:v3.3.12],SizeBytes:40584227,},ContainerImage{Names:[k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1],SizeBytes:40067731,},ContainerImage{Names:[kubernetesui/metrics-scraper:v1.0.4],SizeBytes:36937728,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:9622c6a6dac7499a055a382930f4de82905a3c5735c0753f7094115c9c871309 quay.io/k8scsi/csi-node-driver-registrar:v1.3.0],SizeBytes:16322467,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jul 13 13:35:45.661: INFO:
Logging kubelet events for node k8s-1
Jul 13 13:35:45.670: INFO:
Logging pods the kubelet thinks is on node k8s-1
Jul 13 13:35:45.681: INFO: kube-flannel-7g2gj started at 2020-07-13 12:34:50 +0000 UTC (1+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Init container install-cni ready: true, restart count 1
Jul 13 13:35:45.681: INFO: Container kube-flannel ready: true, restart count 1
Jul 13 13:35:45.681: INFO: nodelocaldns-cfgvx started at 2020-07-13 12:35:09 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container node-cache ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kubernetes-dashboard-77475cf576-54lds started at 2020-07-13 12:35:10 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kubernetes-dashboard ready: true, restart count 1
Jul 13 13:35:45.681: INFO: coredns-76798d84dd-8qtv7 started at 2020-07-13 12:35:10 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container coredns ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-proxy-zzrz5 started at 2020-07-13 12:34:38 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-proxy ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-controller-manager-k8s-1 started at 2020-07-13 13:09:55 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-controller-manager ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-scheduler-k8s-1 started at 2020-07-13 13:09:55 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-scheduler ready: true, restart count 1
Jul 13 13:35:45.681: INFO: dns-autoscaler-85f898cd5c-7zbrx started at 2020-07-13 12:35:08 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container autoscaler ready: true, restart count 1
Jul 13 13:35:45.681: INFO: quobyte-csi-node-pf65l started at 2020-07-13 12:43:01 +0000 UTC (0+2 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container csi-node-driver-registrar ready: true, restart count 1
Jul 13 13:35:45.681: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:45.681: INFO: kube-apiserver-k8s-1 started at 2020-07-13 13:09:55 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.681: INFO: Container kube-apiserver ready: true, restart count 1
W0713 13:35:45.684739 16162 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
Jul 13 13:35:45.700: INFO:
Latency metrics for node k8s-1
Jul 13 13:35:45.700: INFO:
Logging node info for node k8s-2
Jul 13 13:35:45.703: INFO: Node Info: &Node{ObjectMeta:{k8s-2 /api/v1/nodes/k8s-2 61b9b3fa-43c3-4c66-a608-38a72da236e9 14868 0 2020-07-13 12:33:01 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-2 kubernetes.io/os:linux node-role.kubernetes.io/master:] map[csi.volume.kubernetes.io/nodeid:{"csi.quobyte.com":"k8s-2"} flannel.alpha.coreos.com/backend-data:{"VtepMAC":"1e:62:0e:23:84:97"} flannel.alpha.coreos.com/backend-type:vxlan flannel.alpha.coreos.com/kube-subnet-manager:true flannel.alpha.coreos.com/public-ip:172.18.8.102 kubeadm.alpha.kubernetes.io/cri-socket:/var/run/dockershim.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.233.65.0/24,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.233.65.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{42927656960 0} {<nil>} 41921540Ki BinarySI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8099856384 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{7800 -3} {<nil>} 7800m DecimalSI},ephemeral-storage: {{38634891201 0} {<nil>} 38634891201 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7482998784 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2020-07-13 13:10:23 +0000 UTC,LastTransitionTime:2020-07-13 13:10:23 +0000 UTC,Reason:FlannelIsUp,Message:Flannel is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:33:00 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:33:00 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:33:00 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2020-07-13 13:35:41 +0000 UTC,LastTransitionTime:2020-07-13 12:35:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.8.102,},NodeAddress{Type:Hostname,Address:k8s-2,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6b6c1bdd969646f3b9ddb1ff366d8ee5,SystemUUID:6B6C1BDD-9696-46F3-B9DD-B1FF366D8EE5,BootID:824a72c9-f28c-457d-861b-bfcf5ed83337,KernelVersion:3.10.0-957.12.2.el7.x86_64,OSImage:CentOS Linux 7 (Core),ContainerRuntimeVersion:docker://18.9.9,KubeletVersion:v1.17.7,KubeProxyVersion:v1.17.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[nfvpe/multus:v3.4.1],SizeBytes:275759905,},ContainerImage{Names:[kubernetesui/dashboard-amd64:v2.0.0],SizeBytes:221895031,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.17.7],SizeBytes:171088667,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.17.7],SizeBytes:160979739,},ContainerImage{Names:[nginx:1.17],SizeBytes:126698063,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.17.7],SizeBytes:116521281,},ContainerImage{Names:[quay.io/quobyte/csi@sha256:e00d3ab3965671ca30fb729da885f0575ad59fd41b8d7640b35db22254d0dad5 quay.io/quobyte/csi:v1.0.5],SizeBytes:107251602,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-node-cache:1.15.12],SizeBytes:106950843,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.17.7],SizeBytes:94436123,},ContainerImage{Names:[calico/kube-controllers:v3.13.2],SizeBytes:56577289,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/agnhost@sha256:daf5332100521b1256d0e3c56d697a238eaec3af48897ed9167cbadd426773b5 gcr.io/kubernetes-e2e-test-images/agnhost:2.8],SizeBytes:52800335,},ContainerImage{Names:[quay.io/coreos/flannel:v0.12.0],SizeBytes:52767393,},ContainerImage{Names:[coredns/coredns:1.6.5],SizeBytes:41578211,},ContainerImage{Names:[quay.io/coreos/etcd:v3.3.12],SizeBytes:40584227,},ContainerImage{Names:[k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1],SizeBytes:40067731,},ContainerImage{Names:[kubernetesui/metrics-scraper:v1.0.4],SizeBytes:36937728,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:9622c6a6dac7499a055a382930f4de82905a3c5735c0753f7094115c9c871309 quay.io/k8scsi/csi-node-driver-registrar:v1.3.0],SizeBytes:16322467,},ContainerImage{Names:[nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 nginx:1.14-alpine],SizeBytes:16032814,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jul 13 13:35:45.703: INFO:
Logging kubelet events for node k8s-2
Jul 13 13:35:45.709: INFO:
Logging pods the kubelet thinks is on node k8s-2
Jul 13 13:35:45.720: INFO: kube-proxy-vtgsz started at 2020-07-13 12:34:38 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-proxy ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kubernetes-metrics-scraper-747b4fd5cd-8rxzm started at 2020-07-13 12:35:10 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kubernetes-metrics-scraper ready: true, restart count 1
Jul 13 13:35:45.720: INFO: quobyte-csi-node-pc7gf started at 2020-07-13 12:43:00 +0000 UTC (0+2 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container csi-node-driver-registrar ready: true, restart count 1
Jul 13 13:35:45.720: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-apiserver-k8s-2 started at 2020-07-13 12:33:45 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-apiserver ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-controller-manager-k8s-2 started at 2020-07-13 13:09:59 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-controller-manager ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-scheduler-k8s-2 started at 2020-07-13 13:09:59 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container kube-scheduler ready: true, restart count 1
Jul 13 13:35:45.720: INFO: kube-flannel-mkglw started at 2020-07-13 12:34:50 +0000 UTC (1+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Init container install-cni ready: true, restart count 1
Jul 13 13:35:45.720: INFO: Container kube-flannel ready: true, restart count 2
Jul 13 13:35:45.720: INFO: coredns-76798d84dd-mw5gk started at 2020-07-13 12:35:07 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container coredns ready: true, restart count 1
Jul 13 13:35:45.720: INFO: nodelocaldns-8wbl5 started at 2020-07-13 12:35:09 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:45.720: INFO: Container node-cache ready: true, restart count 1
W0713 13:35:45.723495 16162 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
Jul 13 13:35:46.152: INFO:
Latency metrics for node k8s-2
Jul 13 13:35:46.152: INFO:
Logging node info for node k8s-3
Jul 13 13:35:46.158: INFO: Node Info: &Node{ObjectMeta:{k8s-3 /api/v1/nodes/k8s-3 39056001-a933-4135-9790-c583244410e9 14877 0 2020-07-13 12:34:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-3 kubernetes.io/os:linux] map[csi.volume.kubernetes.io/nodeid:{"csi.quobyte.com":"k8s-3"} flannel.alpha.coreos.com/backend-data:{"VtepMAC":"be:a4:2d:98:e4:63"} flannel.alpha.coreos.com/backend-type:vxlan flannel.alpha.coreos.com/kube-subnet-manager:true flannel.alpha.coreos.com/public-ip:172.18.8.103 kubeadm.alpha.kubernetes.io/cri-socket:/var/run/dockershim.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:10.233.66.0/24,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.233.66.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{42927656960 0} {<nil>} 41921540Ki BinarySI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8099856384 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{7900 -3} {<nil>} 7900m DecimalSI},ephemeral-storage: {{38634891201 0} {<nil>} 38634891201 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7738998784 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2020-07-13 13:10:03 +0000 UTC,LastTransitionTime:2020-07-13 13:10:03 +0000 UTC,Reason:FlannelIsUp,Message:Flannel is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 12:34:36 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 12:34:36 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 12:34:36 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 13:10:08 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.8.103,},NodeAddress{Type:Hostname,Address:k8s-3,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fa5133594c8d469ea231d898c98b3cd4,SystemUUID:FA513359-4C8D-469E-A231-D898C98B3CD4,BootID:b43c5ece-b586-4b3e-b3a7-99800720b9c4,KernelVersion:3.10.0-957.12.2.el7.x86_64,OSImage:CentOS Linux 7 (Core),ContainerRuntimeVersion:docker://18.9.9,KubeletVersion:v1.17.7,KubeProxyVersion:v1.17.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[nfvpe/multus:v3.4.1],SizeBytes:275759905,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.17.7],SizeBytes:171088667,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.17.7],SizeBytes:160979739,},ContainerImage{Names:[nginx:1.17],SizeBytes:126698063,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.17.7],SizeBytes:116521281,},ContainerImage{Names:[quay.io/quobyte/csi@sha256:e00d3ab3965671ca30fb729da885f0575ad59fd41b8d7640b35db22254d0dad5 quay.io/quobyte/csi:v1.0.5],SizeBytes:107251602,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-node-cache:1.15.12],SizeBytes:106950843,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.17.7],SizeBytes:94436123,},ContainerImage{Names:[calico/kube-controllers:v3.13.2],SizeBytes:56577289,},ContainerImage{Names:[quay.io/coreos/flannel:v0.12.0],SizeBytes:52767393,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:78e3393f5fd5ff6c1e5dada2478cfa456fb7164929e573cf9a87bf6532730679 quay.io/k8scsi/csi-provisioner:v1.6.0],SizeBytes:48281550,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:2ffa647e8107cfd39e5f464e738dce014c9f5e51b108da36c3ab621048d0bbab quay.io/k8scsi/csi-attacher:v2.2.0],SizeBytes:46131354,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:6c6a0332693a7c456378f6abd2bb40611826c1e1a733cadbdae2daab3125b71c quay.io/k8scsi/csi-resizer:v0.5.0],SizeBytes:46041582,},ContainerImage{Names:[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1],SizeBytes:45874691,},ContainerImage{Names:[quay.io/coreos/etcd:v3.3.12],SizeBytes:40584227,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:9622c6a6dac7499a055a382930f4de82905a3c5735c0753f7094115c9c871309 quay.io/k8scsi/csi-node-driver-registrar:v1.3.0],SizeBytes:16322467,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jul 13 13:35:46.158: INFO:
Logging kubelet events for node k8s-3
Jul 13 13:35:46.166: INFO:
Logging pods the kubelet thinks is on node k8s-3
Jul 13 13:35:46.179: INFO: kube-flannel-gsgqq started at 2020-07-13 12:35:08 +0000 UTC (1+1 container statuses recorded)
Jul 13 13:35:46.179: INFO: Init container install-cni ready: true, restart count 1
Jul 13 13:35:46.179: INFO: Container kube-flannel ready: true, restart count 1
Jul 13 13:35:46.179: INFO: nodelocaldns-ttjbs started at 2020-07-13 12:35:09 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:46.179: INFO: Container node-cache ready: true, restart count 1
Jul 13 13:35:46.179: INFO: quobyte-csi-node-5bdbs started at 2020-07-13 12:43:00 +0000 UTC (0+2 container statuses recorded)
Jul 13 13:35:46.179: INFO: Container csi-node-driver-registrar ready: true, restart count 1
Jul 13 13:35:46.179: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:46.179: INFO: quobyte-csi-controller-0 started at 2020-07-13 12:43:00 +0000 UTC (0+5 container statuses recorded)
Jul 13 13:35:46.179: INFO: Container csi-attacher ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container csi-cluster-driver-registrar ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container csi-provisioner ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container csi-resizer ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:46.180: INFO: nginx-proxy-k8s-3 started at 2020-07-13 13:09:57 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:46.180: INFO: Container nginx-proxy ready: true, restart count 1
Jul 13 13:35:46.180: INFO: kube-proxy-mmjzf started at 2020-07-13 12:34:38 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:46.180: INFO: Container kube-proxy ready: true, restart count 1
W0713 13:35:46.182944 16162 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
Jul 13 13:35:46.201: INFO:
Latency metrics for node k8s-3
Jul 13 13:35:46.201: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volume-expand-1006" for this suite.
• Failure [332.635 seconds]
External Storage [Driver: csi.quobyte.com]
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:94
[Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:100
Verify if offline PVC expansion works [It]
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:159
Jul 13 13:35:45.636: while cleaning up resource
Unexpected error:
<errors.aggregate | len:1, cap:1>: [
[
{
error: {
cause: {
s: "PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s",
},
msg: "Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner",
},
stack: [0x378a2d9, 0x37bff49, 0x37c1b28, 0x7d1e98, 0x7d1aef, 0x7d0f94, 0x7d7ef5, 0x7d7751, 0x7dd5cf, 0x7dd0f0, 0x7dc937, 0x7def1b, 0x7e1a77, 0x7e17bd, 0x38b1baa, 0x38b5ecb, 0x5165d9, 0x462e41],
},
],
]
Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s
occurred
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:139
------------------------------
{"msg":"FAILED External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works","total":148,"completed":3,"skipped":1041,"failed":2,"failures":["External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works"]}kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 12:34:36 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2020-07-13 13:35:44 +0000 UTC,LastTransitionTime:2020-07-13 13:10:08 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.8.103,},NodeAddress{Type:Hostname,Address:k8s-3,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:fa5133594c8d469ea231d898c98b3cd4,SystemUUID:FA513359-4C8D-469E-A231-D898C98B3CD4,BootID:b43c5ece-b586-4b3e-b3a7-99800720b9c4,KernelVersion:3.10.0-957.12.2.el7.x86_64,OSImage:CentOS Linux 7 (Core),ContainerRuntimeVersion:docker://18.9.9,KubeletVersion:v1.17.7,KubeProxyVersion:v1.17.7,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[nfvpe/multus:v3.4.1],SizeBytes:275759905,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.17.7],SizeBytes:171088667,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.17.7],SizeBytes:160979739,},ContainerImage{Names:[nginx:1.17],SizeBytes:126698063,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.17.7],SizeBytes:116521281,},ContainerImage{Names:[quay.io/quobyte/csi@sha256:e00d3ab3965671ca30fb729da885f0575ad59fd41b8d7640b35db22254d0dad5 quay.io/quobyte/csi:v1.0.5],SizeBytes:107251602,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-node-cache:1.15.12],SizeBytes:106950843,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.17.7],SizeBytes:94436123,},ContainerImage{Names:[calico/kube-controllers:v3.13.2],SizeBytes:56577289,},ContainerImage{Names:[quay.io/coreos/flannel:v0.12.0],SizeBytes:52767393,},ContainerImage{Names:[quay.io/k8scsi/csi-provisioner@sha256:78e3393f5fd5ff6c1e5dada2478cfa456fb7164929e573cf9a87bf6532730679 quay.io/k8scsi/csi-provisioner:v1.6.0],SizeBytes:48281550,},ContainerImage{Names:[quay.io/k8scsi/csi-attacher@sha256:2ffa647e8107cfd39e5f464e738dce014c9f5e51b108da36c3ab621048d0bbab quay.io/k8scsi/csi-attacher:v2.2.0],SizeBytes:46131354,},ContainerImage{Names:[quay.io/k8scsi/csi-resizer@sha256:6c6a0332693a7c456378f6abd2bb40611826c1e1a733cadbdae2daab3125b71c quay.io/k8scsi/csi-resizer:v0.5.0],SizeBytes:46041582,},ContainerImage{Names:[quay.io/k8scsi/csi-cluster-driver-registrar@sha256:fafd75ae5442f192cfa8c2e792903aee30d5884b62e802e4464b0a895d21e3ef quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1],SizeBytes:45874691,},ContainerImage{Names:[quay.io/coreos/etcd:v3.3.12],SizeBytes:40584227,},ContainerImage{Names:[quay.io/k8scsi/csi-node-driver-registrar@sha256:9622c6a6dac7499a055a382930f4de82905a3c5735c0753f7094115c9c871309 quay.io/k8scsi/csi-node-driver-registrar:v1.3.0],SizeBytes:16322467,},ContainerImage{Names:[gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0],SizeBytes:1563521,},ContainerImage{Names:[busybox@sha256:8ccbac733d19c0dd4d70b4f0c1e12245b5fa3ad24758a11035ee505c629c0796 busybox:1.29],SizeBytes:1154361,},ContainerImage{Names:[k8s.gcr.io/pause:3.1],SizeBytes:742472,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jul 13 13:35:46.158: INFO:
Logging kubelet events for node k8s-3
Jul 13 13:35:46.166: INFO:
Logging pods the kubelet thinks is on node k8s-3
Jul 13 13:35:46.179: INFO: kube-flannel-gsgqq started at 2020-07-13 12:35:08 +0000 UTC (1+1 container statuses recorded)
Jul 13 13:35:46.179: INFO: Init container install-cni ready: true, restart count 1
Jul 13 13:35:46.179: INFO: Container kube-flannel ready: true, restart count 1
Jul 13 13:35:46.179: INFO: nodelocaldns-ttjbs started at 2020-07-13 12:35:09 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:46.179: INFO: Container node-cache ready: true, restart count 1
Jul 13 13:35:46.179: INFO: quobyte-csi-node-5bdbs started at 2020-07-13 12:43:00 +0000 UTC (0+2 container statuses recorded)
Jul 13 13:35:46.179: INFO: Container csi-node-driver-registrar ready: true, restart count 1
Jul 13 13:35:46.179: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:46.179: INFO: quobyte-csi-controller-0 started at 2020-07-13 12:43:00 +0000 UTC (0+5 container statuses recorded)
Jul 13 13:35:46.179: INFO: Container csi-attacher ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container csi-cluster-driver-registrar ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container csi-provisioner ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container csi-resizer ready: true, restart count 1
Jul 13 13:35:46.180: INFO: Container quobyte-csi-plugin ready: true, restart count 1
Jul 13 13:35:46.180: INFO: nginx-proxy-k8s-3 started at 2020-07-13 13:09:57 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:46.180: INFO: Container nginx-proxy ready: true, restart count 1
Jul 13 13:35:46.180: INFO: kube-proxy-mmjzf started at 2020-07-13 12:34:38 +0000 UTC (0+1 container statuses recorded)
Jul 13 13:35:46.180: INFO: Container kube-proxy ready: true, restart count 1
W0713 13:35:46.182944 16162 metrics_grabber.go:79] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
Jul 13 13:35:46.201: INFO:
Latency metrics for node k8s-3
Jul 13 13:35:46.201: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volume-expand-1006" for this suite.
• Failure [332.635 seconds]
External Storage [Driver: csi.quobyte.com]
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:94
[Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:100
Verify if offline PVC expansion works [It]
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:159
Jul 13 13:35:45.636: while cleaning up resource
Unexpected error:
<errors.aggregate | len:1, cap:1>: [
[
{
error: {
cause: {
s: "PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s",
},
msg: "Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner",
},
stack: [0x378a2d9, 0x37bff49, 0x37c1b28, 0x7d1e98, 0x7d1aef, 0x7d0f94, 0x7d7ef5, 0x7d7751, 0x7dd5cf, 0x7dd0f0, 0x7dc937, 0x7def1b, 0x7e1a77, 0x7e17bd, 0x38b1baa, 0x38b5ecb, 0x5165d9, 0x462e41],
},
],
]
Persistent Volume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 not deleted by dynamic provisioner: PersistentVolume pvc-1da9a359-5220-4515-bf7e-b6d9b0313ea8 still exists within 5m0s
occurred
/workspace/anago-v1.17.7-rc.0.50+e4efcc9eb3807d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:139
------------------------------
{"msg":"FAILED External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works","total":148,"completed":3,"skipped":1041,"failed":2,"failures":["External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","External Storage [Driver: csi.quobyte.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works"]}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment