Skip to content

Instantly share code, notes, and snippets.

@Billy99

Billy99/worker-advnetlab52.log

Created March 3, 2021 21:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Billy99/4590bcfc8939edee29503b224d4972d8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Billy99/4590bcfc8939edee29503b224d4972d8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
worker-advnetlab52.log
journalctl --since "1 hour ago"
-- Logs begin at Mon 2021-03-01 05:44:38 UTC, end at Wed 2021-03-03 21:46:40 UTC. --
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: I0303 20:46:42.174321 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.174690940Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.186367685Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f NetNS:/var/run/netns/c47775b3-293e-4734-b6d9->
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.186399601Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.235857881Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.235898349Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.235926133Z" level=info msg="NetworkStart: stopping network for sandbox 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.236558053Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f NetNS:/var/run/netns/c47775b3-293e-4734-b6d9->
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.236592920Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: 2021-03-03T20:46:42Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.273601240Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/ru>
Mar 03 20:46:42 worker-advnetlab52 systemd[1]: run-utsns-c47775b3\x2d293e\x2d4734\x2db6d9\x2d2e548af7d2ca.mount: Succeeded.
Mar 03 20:46:42 worker-advnetlab52 systemd[1]: run-ipcns-c47775b3\x2d293e\x2d4734\x2db6d9\x2d2e548af7d2ca.mount: Succeeded.
Mar 03 20:46:42 worker-advnetlab52 systemd[1]: run-netns-c47775b3\x2d293e\x2d4734\x2db6d9\x2d2e548af7d2ca.mount: Succeeded.
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.327556256Z" level=info msg="RunSandbox: deleting pod ID 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f from idIndex" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.327585414Z" level=info msg="RunSandbox: removing pod sandbox 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.327605237Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.327619936Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:42 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f-userdata-shm.mount: Succeeded.
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.356703622Z" level=info msg="RunSandbox: removing pod sandbox from storage: 5963e8dee48ec49f6cf983785c9861b77df0fe85eaa1eb366a5fc49ed97f6f4f" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2.Ru>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.359606477Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alph>
Mar 03 20:46:42 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:42.359627496Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=d3c5e0a3-7baf-4767-a293-d09de144db5c name=/runtime.v1alpha2>
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: E0303 20:46:42.359785 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: E0303 20:46:42.359839 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: E0303 20:46:42.359853 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:42 worker-advnetlab52 hyperkube[4431]: E0303 20:46:42.359899 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: I0303 20:46:48.174281 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.174593227Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.188426575Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2 NetNS:/var/run/netns/1f127bac-3>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.188452251Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.237810721Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.237860014Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.237887366Z" level=info msg="NetworkStart: stopping network for sandbox 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.238516860Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2 NetNS:/var/run/netns/1f127bac-3>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.238554216Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: 2021-03-03T20:46:48Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.275063523Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/ru>
Mar 03 20:46:48 worker-advnetlab52 systemd[1]: run-utsns-1f127bac\x2d34e3\x2d4c02\x2db4c9\x2d30dd2ad8a06d.mount: Succeeded.
Mar 03 20:46:48 worker-advnetlab52 systemd[1]: run-ipcns-1f127bac\x2d34e3\x2d4c02\x2db4c9\x2d30dd2ad8a06d.mount: Succeeded.
Mar 03 20:46:48 worker-advnetlab52 systemd[1]: run-netns-1f127bac\x2d34e3\x2d4c02\x2db4c9\x2d30dd2ad8a06d.mount: Succeeded.
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.313584597Z" level=info msg="RunSandbox: deleting pod ID 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2 from idIndex" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.313615336Z" level=info msg="RunSandbox: removing pod sandbox 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.313637647Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.313658314Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:48 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2-userdata-shm.mount: Succeeded.
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.325833032Z" level=info msg="RunSandbox: removing pod sandbox from storage: 4ad76a394d1f1c344d4be94588771be152564548b596a55d33dcd3568f48efb2" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/runtime.v1alpha2.Ru>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.328797563Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/>
Mar 03 20:46:48 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:48.328818847Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=3bcd7264-9cf4-4df2-991b-e69cd4ea7a53 name=/ru>
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: E0303 20:46:48.329113 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: E0303 20:46:48.329167 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: E0303 20:46:48.329180 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:48 worker-advnetlab52 hyperkube[4431]: E0303 20:46:48.329229 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: I0303 20:46:49.174239 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.174578671Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.185399560Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7 NetNS:/var/run/netns/214db0f9-8>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.185417616Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.235008230Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.235045439Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.235073259Z" level=info msg="NetworkStart: stopping network for sandbox 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.235747826Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7 NetNS:/var/run/netns/214db0f9-8>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.235782562Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: 2021-03-03T20:46:49Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.288402717Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/ru>
Mar 03 20:46:49 worker-advnetlab52 systemd[1]: run-utsns-214db0f9\x2d8526\x2d4d44\x2d8d95\x2decd3111830c9.mount: Succeeded.
Mar 03 20:46:49 worker-advnetlab52 systemd[1]: run-ipcns-214db0f9\x2d8526\x2d4d44\x2d8d95\x2decd3111830c9.mount: Succeeded.
Mar 03 20:46:49 worker-advnetlab52 systemd[1]: run-netns-214db0f9\x2d8526\x2d4d44\x2d8d95\x2decd3111830c9.mount: Succeeded.
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.327616247Z" level=info msg="RunSandbox: deleting pod ID 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7 from idIndex" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.327670734Z" level=info msg="RunSandbox: removing pod sandbox 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.327733697Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.327751066Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:49 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7-userdata-shm.mount: Succeeded.
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.343673419Z" level=info msg="RunSandbox: removing pod sandbox from storage: 7b8daca7e8c2e8e8d513cb318bb19a9a5ad0e7a736e9d95d02ed9f77d0b806f7" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/runtime.v1alpha2.Ru>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.346523169Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/>
Mar 03 20:46:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:49.346544927Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=32b0eabb-eeb9-4298-9d80-94b398ff98ee name=/ru>
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: E0303 20:46:49.346703 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: E0303 20:46:49.346737 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: E0303 20:46:49.346752 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: E0303 20:46:49.346789 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:46:49 worker-advnetlab52 hyperkube[4431]: I0303 20:46:49.828910 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: I0303 20:46:51.174172 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: I0303 20:46:51.174298 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.174435045Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.174669081Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.196083386Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d NetNS:/var/run/netns>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.196110116Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.196512591Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae NetNS:/var/run/netns/e7db894c-3b48-49d9>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.196535593Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: I0303 20:46:51.235262 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.247067431Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.247114364Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.247154226Z" level=info msg="NetworkStart: stopping network for sandbox 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.247855132Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d NetNS:/var/run/netns>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.247892756Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.266424950Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.266506017Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.266607218Z" level=info msg="NetworkStart: stopping network for sandbox 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.267292902Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae NetNS:/var/run/netns/e7db894c-3b48-49d9>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.267328040Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: 2021-03-03T20:46:51Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.285003581Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/ru>
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-utsns-e00fe370\x2dcb4b\x2d4068\x2db950\x2df7502efa1725.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: 2021-03-03T20:46:51Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.305042595Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/ru>
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-ipcns-e00fe370\x2dcb4b\x2d4068\x2db950\x2df7502efa1725.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-utsns-e7db894c\x2d3b48\x2d49d9\x2d815b\x2d6a8a65d5ea2a.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-ipcns-e7db894c\x2d3b48\x2d49d9\x2d815b\x2d6a8a65d5ea2a.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-netns-e00fe370\x2dcb4b\x2d4068\x2db950\x2df7502efa1725.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.335553715Z" level=info msg="RunSandbox: deleting pod ID 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d from idIndex" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.335580235Z" level=info msg="RunSandbox: removing pod sandbox 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.335597737Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.335615137Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-netns-e7db894c\x2d3b48\x2d49d9\x2d815b\x2d6a8a65d5ea2a.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d-userdata-shm.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.346613128Z" level=info msg="RunSandbox: deleting pod ID 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae from idIndex" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.346668204Z" level=info msg="RunSandbox: removing pod sandbox 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.346707629Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.346739676Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:51 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae-userdata-shm.mount: Succeeded.
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.354808028Z" level=info msg="RunSandbox: removing pod sandbox from storage: 1540b8af58f80d1ea8a50cddb749fc3ebc7f4af0146e88f9324794c7dc7fb15d" id=bd7e5373-8252-440c-a625-ec3a1c4a75d5 name=/runtime.v1alpha2.Ru>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.357862142Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=bd7e5373-8252-440c-a625-ec3a1c4a>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.357887021Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=bd7e5373-8252-440c-a625-ec3a1c4a75>
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.358128 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.358179 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.358192 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.358240 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.371680185Z" level=info msg="RunSandbox: removing pod sandbox from storage: 9ce9575bc1a9a1c41d618fe52b6850d33a6a2e80a425ae5fda4c1586845ad3ae" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1alpha2.Ru>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.374560810Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.>
Mar 03 20:46:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:51.374589442Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=e32b8578-6f86-4f3d-8895-522d28a344c6 name=/runtime.v1>
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.374798 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.374832 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.374844 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:51 worker-advnetlab52 hyperkube[4431]: E0303 20:46:51.374883 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: I0303 20:46:55.174485 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.174819612Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.186468812Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c NetNS:/var/run/netns/d12a494b-be47-4a98-ab51->
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.186489372Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.238548152Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.238620511Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.238673430Z" level=info msg="NetworkStart: stopping network for sandbox c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.239314523Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c NetNS:/var/run/netns/d12a494b-be47-4a98-ab51->
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.239348066Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: 2021-03-03T20:46:55Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.278397703Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/ru>
Mar 03 20:46:55 worker-advnetlab52 systemd[1]: run-utsns-d12a494b\x2dbe47\x2d4a98\x2dab51\x2dd10831e4aedb.mount: Succeeded.
Mar 03 20:46:55 worker-advnetlab52 systemd[1]: run-ipcns-d12a494b\x2dbe47\x2d4a98\x2dab51\x2dd10831e4aedb.mount: Succeeded.
Mar 03 20:46:55 worker-advnetlab52 systemd[1]: run-netns-d12a494b\x2dbe47\x2d4a98\x2dab51\x2dd10831e4aedb.mount: Succeeded.
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.319630905Z" level=info msg="RunSandbox: deleting pod ID c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c from idIndex" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.319661282Z" level=info msg="RunSandbox: removing pod sandbox c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.319677911Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.319698351Z" level=info msg="RunSandbox: unmounting shmPath for sandbox c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2.Runtim>
Mar 03 20:46:55 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c-userdata-shm.mount: Succeeded.
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.335727117Z" level=info msg="RunSandbox: removing pod sandbox from storage: c514b983d42f7568a819d3606cfb0bb0e679e81f80c9109df94142331b8e1c1c" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2.Ru>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.340629226Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alph>
Mar 03 20:46:55 worker-advnetlab52 crio[4383]: time="2021-03-03 20:46:55.340657895Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=4f1c6f4d-b90c-494e-a890-b62801ce5615 name=/runtime.v1alpha2>
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: E0303 20:46:55.340906 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: E0303 20:46:55.340941 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: E0303 20:46:55.340954 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:46:55 worker-advnetlab52 hyperkube[4431]: E0303 20:46:55.340994 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:46:57 worker-advnetlab52 hyperkube[4431]: I0303 20:46:57.086034 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/coredns-worker-advnetlab52" status=Running
Mar 03 20:46:57 worker-advnetlab52 hyperkube[4431]: I0303 20:46:57.086059 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/keepalived-worker-advnetlab52" status=Running
Mar 03 20:46:57 worker-advnetlab52 hyperkube[4431]: I0303 20:46:57.086067 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/mdns-publisher-worker-advnetlab52" status=Running
Mar 03 20:46:57 worker-advnetlab52 hyperkube[4431]: I0303 20:46:57.303416 4431 container_manager_linux.go:986] Found 791 PIDs in root, 791 of them are not to be moved
Mar 03 20:46:59 worker-advnetlab52 hyperkube[4431]: I0303 20:46:59.836375 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: I0303 20:47:00.174433 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.174870854Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.186851298Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e NetNS:/var/run/netns/49f9f90f-6>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.186881318Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.241517662Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.241572132Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.241599555Z" level=info msg="NetworkStart: stopping network for sandbox 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.242269536Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e NetNS:/var/run/netns/49f9f90f-6>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.242313303Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:00Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.280602427Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/ru>
Mar 03 20:47:00 worker-advnetlab52 systemd[1]: run-utsns-49f9f90f\x2d623c\x2d4048\x2da538\x2d3d36417f3a56.mount: Succeeded.
Mar 03 20:47:00 worker-advnetlab52 systemd[1]: run-ipcns-49f9f90f\x2d623c\x2d4048\x2da538\x2d3d36417f3a56.mount: Succeeded.
Mar 03 20:47:00 worker-advnetlab52 systemd[1]: run-netns-49f9f90f\x2d623c\x2d4048\x2da538\x2d3d36417f3a56.mount: Succeeded.
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.317614103Z" level=info msg="RunSandbox: deleting pod ID 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e from idIndex" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.317670241Z" level=info msg="RunSandbox: removing pod sandbox 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.317714514Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.317760234Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:00 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e-userdata-shm.mount: Succeeded.
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.333695002Z" level=info msg="RunSandbox: removing pod sandbox from storage: 171b8daf7ba2c8dd061a5a94f6fdfd9f88789605d6bdfc197566bed6f85d341e" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/runtime.v1alpha2.Ru>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.336424383Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/>
Mar 03 20:47:00 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:00.336447127Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=3f30a7d2-6e23-489b-8fe5-d9b7ff3ba4fb name=/ru>
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: E0303 20:47:00.336611 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: E0303 20:47:00.336671 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: E0303 20:47:00.336688 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:00 worker-advnetlab52 hyperkube[4431]: E0303 20:47:00.336730 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: I0303 20:47:03.174304 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: I0303 20:47:03.174382 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.174740708Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.174740765Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.193307792Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8 NetNS:/var/run/netns>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.193338598Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.193956050Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80 NetNS:/var/run/netns/f33dd9ad-c76f-4615>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.193979631Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: W0303 20:47:03.214568 4431 http2_server.go:482] transport: http2Server.HandleStreams failed to read frame: read unix /var/lib/kubelet/pod-resources/769277831->@: read: connection reset by peer
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: I0303 20:47:03.214621 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.242761266Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.242801467Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.242834100Z" level=info msg="NetworkStart: stopping network for sandbox 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.243576191Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8 NetNS:/var/run/netns>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.243627631Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.244452342Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.244487160Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.244518317Z" level=info msg="NetworkStart: stopping network for sandbox 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.245049535Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80 NetNS:/var/run/netns/f33dd9ad-c76f-4615>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.245083675Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:03Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.280436726Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/ru>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:03Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.281605337Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/ru>
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-utsns-4e852dbe\x2d83e0\x2d4b46\x2d9b6e\x2dd626613a0c1c.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-utsns-f33dd9ad\x2dc76f\x2d4615\x2d824a\x2d36795230808f.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-ipcns-4e852dbe\x2d83e0\x2d4b46\x2d9b6e\x2dd626613a0c1c.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-ipcns-f33dd9ad\x2dc76f\x2d4615\x2d824a\x2d36795230808f.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-netns-4e852dbe\x2d83e0\x2d4b46\x2d9b6e\x2dd626613a0c1c.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-netns-f33dd9ad\x2dc76f\x2d4615\x2d824a\x2d36795230808f.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.336627904Z" level=info msg="RunSandbox: deleting pod ID 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8 from idIndex" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.336686639Z" level=info msg="RunSandbox: removing pod sandbox 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.336743718Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.336777277Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8-userdata-shm.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.344597857Z" level=info msg="RunSandbox: deleting pod ID 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80 from idIndex" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.344622356Z" level=info msg="RunSandbox: removing pod sandbox 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.344640959Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.344657840Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:03 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80-userdata-shm.mount: Succeeded.
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.352744904Z" level=info msg="RunSandbox: removing pod sandbox from storage: 18eed134c149242861bdcd06e5c323be38734e525e4271918d96445da1224cf8" id=f21e7985-0521-47c3-945f-11c735c7c432 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.355933130Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=f21e7985-0521-47c3-945f-11c735c7>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.355956706Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=f21e7985-0521-47c3-945f-11c735c7c4>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.356207 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.356258 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.356271 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.356323 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.360785005Z" level=info msg="RunSandbox: removing pod sandbox from storage: 97a6a1f533d5eefa9104272945e81bc889775c3c004792119c17343b4ea16d80" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.363589531Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.>
Mar 03 20:47:03 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:03.363611653Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=927e902a-d978-4727-8ff3-2eedcab5c9a9 name=/runtime.v1>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.363767 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.363809 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.363824 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:03 worker-advnetlab52 hyperkube[4431]: E0303 20:47:03.363878 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: I0303 20:47:04.174217 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.174585210Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.185651589Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb NetNS:/var/run/netns/8f3efdc7-6>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.185669204Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.236729621Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.236764683Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.236798743Z" level=info msg="NetworkStart: stopping network for sandbox 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.237401650Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb NetNS:/var/run/netns/8f3efdc7-6>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.237435596Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:04Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.273746475Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/ru>
Mar 03 20:47:04 worker-advnetlab52 systemd[1]: run-utsns-8f3efdc7\x2d6405\x2d4d44\x2d945d\x2dd27f1174acdf.mount: Succeeded.
Mar 03 20:47:04 worker-advnetlab52 systemd[1]: run-ipcns-8f3efdc7\x2d6405\x2d4d44\x2d945d\x2dd27f1174acdf.mount: Succeeded.
Mar 03 20:47:04 worker-advnetlab52 systemd[1]: run-netns-8f3efdc7\x2d6405\x2d4d44\x2d945d\x2dd27f1174acdf.mount: Succeeded.
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.313646943Z" level=info msg="RunSandbox: deleting pod ID 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb from idIndex" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.313739530Z" level=info msg="RunSandbox: removing pod sandbox 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.313784438Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.313817744Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:04 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb-userdata-shm.mount: Succeeded.
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.322663177Z" level=info msg="RunSandbox: removing pod sandbox from storage: 8984d43ca9b66bc533667b466055c9fde77b67073a24e17d4515bcd7893910bb" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.325515810Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/>
Mar 03 20:47:04 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:04.325535545Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=dd71e992-a4bc-47cf-8931-8848dc2ab712 name=/ru>
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: E0303 20:47:04.325711 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: E0303 20:47:04.325757 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: E0303 20:47:04.325770 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:04 worker-advnetlab52 hyperkube[4431]: E0303 20:47:04.325818 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:09 worker-advnetlab52 hyperkube[4431]: I0303 20:47:09.843865 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: I0303 20:47:11.174276 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.174544925Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.185987341Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1 NetNS:/var/run/netns/e67c8f74-d3c3-43b4-8007->
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.186012761Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.241567030Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.241613199Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.241642258Z" level=info msg="NetworkStart: stopping network for sandbox 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.242303427Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1 NetNS:/var/run/netns/e67c8f74-d3c3-43b4-8007->
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.242337550Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:11Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.279818440Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/ru>
Mar 03 20:47:11 worker-advnetlab52 systemd[1]: run-utsns-e67c8f74\x2dd3c3\x2d43b4\x2d8007\x2de598b27811a4.mount: Succeeded.
Mar 03 20:47:11 worker-advnetlab52 systemd[1]: run-ipcns-e67c8f74\x2dd3c3\x2d43b4\x2d8007\x2de598b27811a4.mount: Succeeded.
Mar 03 20:47:11 worker-advnetlab52 systemd[1]: run-netns-e67c8f74\x2dd3c3\x2d43b4\x2d8007\x2de598b27811a4.mount: Succeeded.
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.317619186Z" level=info msg="RunSandbox: deleting pod ID 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1 from idIndex" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.317677891Z" level=info msg="RunSandbox: removing pod sandbox 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.317715468Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.317751882Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:11 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1-userdata-shm.mount: Succeeded.
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.329778048Z" level=info msg="RunSandbox: removing pod sandbox from storage: 6bf7a9541b4be3d7fb299c858022652007d84a364d79738254ed974d969f0ec1" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.332771147Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alph>
Mar 03 20:47:11 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:11.332792098Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=51edf387-8730-4fe7-b295-a531d9f53981 name=/runtime.v1alpha2>
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: E0303 20:47:11.333095 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: E0303 20:47:11.333183 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: E0303 20:47:11.333196 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:11 worker-advnetlab52 hyperkube[4431]: E0303 20:47:11.333234 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: I0303 20:47:14.174011 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.174298401Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.186229135Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4 NetNS:/var/run/netns/e2b639ec-35e6-418e>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.186255276Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: I0303 20:47:14.215623 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.245870657Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.245932004Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.245958182Z" level=info msg="NetworkStart: stopping network for sandbox ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.246903686Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4 NetNS:/var/run/netns/e2b639ec-35e6-418e>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.246944574Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:14Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.283106848Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/ru>
Mar 03 20:47:14 worker-advnetlab52 systemd[1]: run-utsns-e2b639ec\x2d35e6\x2d418e\x2dafe9\x2d396c2152a766.mount: Succeeded.
Mar 03 20:47:14 worker-advnetlab52 systemd[1]: run-ipcns-e2b639ec\x2d35e6\x2d418e\x2dafe9\x2d396c2152a766.mount: Succeeded.
Mar 03 20:47:14 worker-advnetlab52 systemd[1]: run-netns-e2b639ec\x2d35e6\x2d418e\x2dafe9\x2d396c2152a766.mount: Succeeded.
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.331616639Z" level=info msg="RunSandbox: deleting pod ID ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4 from idIndex" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.331675370Z" level=info msg="RunSandbox: removing pod sandbox ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.331714496Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.331746186Z" level=info msg="RunSandbox: unmounting shmPath for sandbox ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:14 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4-userdata-shm.mount: Succeeded.
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.346680874Z" level=info msg="RunSandbox: removing pod sandbox from storage: ace86e3b004be8856aa58d2a5157525959587dc2bd30bb8a71c9f6350351cea4" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.349437651Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.>
Mar 03 20:47:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:14.349458283Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=954c34ff-46c8-4b82-af28-c35227a9a980 name=/runtime.v1>
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: E0303 20:47:14.349684 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: E0303 20:47:14.349716 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: E0303 20:47:14.349734 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:14 worker-advnetlab52 hyperkube[4431]: E0303 20:47:14.349771 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: I0303 20:47:15.174539 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.174831566Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.188888091Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2 NetNS:/var/run/netns/b22571f6-d>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.188912439Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.236105949Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.236146891Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.236177886Z" level=info msg="NetworkStart: stopping network for sandbox 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.236816515Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2 NetNS:/var/run/netns/b22571f6-d>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.236850352Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:15Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.272033270Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/ru>
Mar 03 20:47:15 worker-advnetlab52 systemd[1]: run-utsns-b22571f6\x2dd2a2\x2d4b68\x2d9097\x2d2b092ad43fe5.mount: Succeeded.
Mar 03 20:47:15 worker-advnetlab52 systemd[1]: run-ipcns-b22571f6\x2dd2a2\x2d4b68\x2d9097\x2d2b092ad43fe5.mount: Succeeded.
Mar 03 20:47:15 worker-advnetlab52 systemd[1]: run-netns-b22571f6\x2dd2a2\x2d4b68\x2d9097\x2d2b092ad43fe5.mount: Succeeded.
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.324629126Z" level=info msg="RunSandbox: deleting pod ID 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2 from idIndex" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.324695746Z" level=info msg="RunSandbox: removing pod sandbox 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.324734951Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.324772447Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:15 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2-userdata-shm.mount: Succeeded.
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.340724463Z" level=info msg="RunSandbox: removing pod sandbox from storage: 7d28fe8705f95d700c5f8a563624abf55ecb87afab8122c3762387751ad097a2" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.343598402Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/>
Mar 03 20:47:15 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:15.343618050Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=5648d356-35f4-4f8b-8ecb-75141f91beb3 name=/ru>
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: E0303 20:47:15.343786 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: E0303 20:47:15.343820 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: E0303 20:47:15.343834 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:15 worker-advnetlab52 hyperkube[4431]: E0303 20:47:15.343870 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: I0303 20:47:18.174246 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: I0303 20:47:18.174340 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.174695556Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.174711419Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.188842497Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc NetNS:/var/run/netns/04a83e77-8>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.188883924Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.192336988Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20 NetNS:/var/run/netns>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.192360071Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.238479038Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.238529427Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.238560881Z" level=info msg="NetworkStart: stopping network for sandbox 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.239204901Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc NetNS:/var/run/netns/04a83e77-8>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.239239454Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.245593118Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.245632083Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.245661222Z" level=info msg="NetworkStart: stopping network for sandbox 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.246311547Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20 NetNS:/var/run/netns>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.246345296Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:18Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.277514049Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/ru>
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-utsns-04a83e77\x2d85a4\x2d4744\x2db283\x2d54772984f43a.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:18Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.282432538Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/ru>
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-utsns-3967f6f6\x2d83cc\x2d46ce\x2db991\x2de47e11aacdaf.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-ipcns-04a83e77\x2d85a4\x2d4744\x2db283\x2d54772984f43a.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-ipcns-3967f6f6\x2d83cc\x2d46ce\x2db991\x2de47e11aacdaf.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-netns-04a83e77\x2d85a4\x2d4744\x2db283\x2d54772984f43a.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-netns-3967f6f6\x2d83cc\x2d46ce\x2db991\x2de47e11aacdaf.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.332631930Z" level=info msg="RunSandbox: deleting pod ID 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc from idIndex" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.332704651Z" level=info msg="RunSandbox: removing pod sandbox 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.332720681Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.332734743Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc-userdata-shm.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.335551520Z" level=info msg="RunSandbox: deleting pod ID 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20 from idIndex" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.335574475Z" level=info msg="RunSandbox: removing pod sandbox 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.335589115Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.335603461Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:18 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20-userdata-shm.mount: Succeeded.
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.347818643Z" level=info msg="RunSandbox: removing pod sandbox from storage: 69bb9f883444e46831852c515742ebdef2f39d5de022b81bd2cc3067cbc51dfc" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/runtime.v1alpha2.Ru>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.352807793Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.352839458Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=7957464f-478e-4ce1-b238-06d41e23e5dd name=/ru>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.353026 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.353069 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.353084 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.353136 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.356823320Z" level=info msg="RunSandbox: removing pod sandbox from storage: 89e3e0476905f4bae2d650906cb5e8a04398d8a4c0c8dad03b120ad34077ca20" id=9d0577ab-0fa1-48fa-9ae2-852607e5590f name=/runtime.v1alpha2.Ru>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.359606517Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=9d0577ab-0fa1-48fa-9ae2-852607e5>
Mar 03 20:47:18 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:18.359627835Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=9d0577ab-0fa1-48fa-9ae2-852607e559>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.359800 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.359835 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.359846 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:18 worker-advnetlab52 hyperkube[4431]: E0303 20:47:18.359881 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:47:19 worker-advnetlab52 hyperkube[4431]: I0303 20:47:19.851436 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: I0303 20:47:26.174188 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.174457913Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.185721369Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989 NetNS:/var/run/netns/8c3bc7df-eb45-4026-9381->
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.185742880Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.237790609Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.237835241Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.237861337Z" level=info msg="NetworkStart: stopping network for sandbox af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.238496604Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989 NetNS:/var/run/netns/8c3bc7df-eb45-4026-9381->
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.238539819Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:26Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.275766696Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/ru>
Mar 03 20:47:26 worker-advnetlab52 systemd[1]: run-utsns-8c3bc7df\x2deb45\x2d4026\x2d9381\x2dbc10e936d322.mount: Succeeded.
Mar 03 20:47:26 worker-advnetlab52 systemd[1]: run-ipcns-8c3bc7df\x2deb45\x2d4026\x2d9381\x2dbc10e936d322.mount: Succeeded.
Mar 03 20:47:26 worker-advnetlab52 systemd[1]: run-netns-8c3bc7df\x2deb45\x2d4026\x2d9381\x2dbc10e936d322.mount: Succeeded.
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.324551599Z" level=info msg="RunSandbox: deleting pod ID af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989 from idIndex" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.324576469Z" level=info msg="RunSandbox: removing pod sandbox af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.324609496Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.324637144Z" level=info msg="RunSandbox: unmounting shmPath for sandbox af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:26 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989-userdata-shm.mount: Succeeded.
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.335650533Z" level=info msg="RunSandbox: removing pod sandbox from storage: af1fe7064fc22e9d93d5612463b7778bd60c95cf0694b03d2a4c58196a2df989" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2.Ru>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.338581703Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alph>
Mar 03 20:47:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:26.338602150Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=6f617226-66a7-4d22-af8c-8af1043fdbfa name=/runtime.v1alpha2>
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: E0303 20:47:26.338945 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: E0303 20:47:26.339010 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: E0303 20:47:26.339031 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:26 worker-advnetlab52 hyperkube[4431]: E0303 20:47:26.339071 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: I0303 20:47:29.174309 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.174649948Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.186927433Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418 NetNS:/var/run/netns/d364bf18-b9d7-42ab>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.186960529Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: I0303 20:47:29.207366 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.238598566Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.238679575Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.238748842Z" level=info msg="NetworkStart: stopping network for sandbox 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.239305234Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418 NetNS:/var/run/netns/d364bf18-b9d7-42ab>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.239337153Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:29Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.276783316Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418" id=43e8212d-850a-4934-b5c7-40362526577c name=/ru>
Mar 03 20:47:29 worker-advnetlab52 systemd[1]: run-utsns-d364bf18\x2db9d7\x2d42ab\x2da7b9\x2d089df1aa7542.mount: Succeeded.
Mar 03 20:47:29 worker-advnetlab52 systemd[1]: run-ipcns-d364bf18\x2db9d7\x2d42ab\x2da7b9\x2d089df1aa7542.mount: Succeeded.
Mar 03 20:47:29 worker-advnetlab52 systemd[1]: run-netns-d364bf18\x2db9d7\x2d42ab\x2da7b9\x2d089df1aa7542.mount: Succeeded.
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.321582615Z" level=info msg="RunSandbox: deleting pod ID 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418 from idIndex" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.321605512Z" level=info msg="RunSandbox: removing pod sandbox 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.321621689Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.321635637Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:29 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418-userdata-shm.mount: Succeeded.
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.333659388Z" level=info msg="RunSandbox: removing pod sandbox from storage: 00bd1b219aaadbf51f64b11e825cfa18f5c466c93ff330dff5f1542da7016418" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1alpha2.Ru>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.336550823Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.>
Mar 03 20:47:29 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:29.336571643Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=43e8212d-850a-4934-b5c7-40362526577c name=/runtime.v1>
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: E0303 20:47:29.336823 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: E0303 20:47:29.336883 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: E0303 20:47:29.336897 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: E0303 20:47:29.336949 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:47:29 worker-advnetlab52 hyperkube[4431]: I0303 20:47:29.858664 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: I0303 20:47:30.174121 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.174372010Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.187001790Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf NetNS:/var/run/netns/17ea455f-9>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.187022319Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.238328111Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.238367338Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.238393203Z" level=info msg="NetworkStart: stopping network for sandbox 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.239163405Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf NetNS:/var/run/netns/17ea455f-9>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.239202963Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:30Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.274460679Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf" id=f562e488-7e78-4b86-936d-047750568c3f name=/ru>
Mar 03 20:47:30 worker-advnetlab52 systemd[1]: run-utsns-17ea455f\x2d9066\x2d49dd\x2d860b\x2d36b89bab651f.mount: Succeeded.
Mar 03 20:47:30 worker-advnetlab52 systemd[1]: run-ipcns-17ea455f\x2d9066\x2d49dd\x2d860b\x2d36b89bab651f.mount: Succeeded.
Mar 03 20:47:30 worker-advnetlab52 systemd[1]: run-netns-17ea455f\x2d9066\x2d49dd\x2d860b\x2d36b89bab651f.mount: Succeeded.
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.308612306Z" level=info msg="RunSandbox: deleting pod ID 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf from idIndex" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.308663363Z" level=info msg="RunSandbox: removing pod sandbox 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.308715307Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.308751794Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:30 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf-userdata-shm.mount: Succeeded.
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.320769733Z" level=info msg="RunSandbox: removing pod sandbox from storage: 30d576b4a5805f2935f0c6ba1ea8281c69b7b96cfcc47b16345df619a03205cf" id=f562e488-7e78-4b86-936d-047750568c3f name=/runtime.v1alpha2.Ru>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.323450469Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=f562e488-7e78-4b86-936d-047750568c3f name=/>
Mar 03 20:47:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:30.323471244Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=f562e488-7e78-4b86-936d-047750568c3f name=/ru>
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: E0303 20:47:30.323654 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: E0303 20:47:30.323685 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: E0303 20:47:30.323699 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:30 worker-advnetlab52 hyperkube[4431]: E0303 20:47:30.323733 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: I0303 20:47:32.174302 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: I0303 20:47:32.174351 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.174550724Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.174656375Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.193064358Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1 NetNS:/var/run/netns/309e0914-d>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.193094483Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.196143181Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977 NetNS:/var/run/netns>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.196171383Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.243184959Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.243220194Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.243246628Z" level=info msg="NetworkStart: stopping network for sandbox 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.243968638Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1 NetNS:/var/run/netns/309e0914-d>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.244011177Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.247052082Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.247096898Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.247123762Z" level=info msg="NetworkStart: stopping network for sandbox aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.247906155Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977 NetNS:/var/run/netns>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.247951292Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:32Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.282579059Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/ru>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:32Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.284689203Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/ru>
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-utsns-309e0914\x2dd0ff\x2d4d6d\x2d9212\x2d18f2f9d3be12.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-utsns-fc125af5\x2dd897\x2d4c7d\x2d861f\x2d13c846a40bc7.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-ipcns-309e0914\x2dd0ff\x2d4d6d\x2d9212\x2d18f2f9d3be12.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-ipcns-fc125af5\x2dd897\x2d4c7d\x2d861f\x2d13c846a40bc7.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-netns-309e0914\x2dd0ff\x2d4d6d\x2d9212\x2d18f2f9d3be12.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-netns-fc125af5\x2dd897\x2d4c7d\x2d861f\x2d13c846a40bc7.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.337634003Z" level=info msg="RunSandbox: deleting pod ID 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1 from idIndex" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.337696462Z" level=info msg="RunSandbox: removing pod sandbox 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.337735831Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.337770062Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1-userdata-shm.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.341625425Z" level=info msg="RunSandbox: deleting pod ID aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977 from idIndex" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.341705220Z" level=info msg="RunSandbox: removing pod sandbox aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.341747966Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.341780259Z" level=info msg="RunSandbox: unmounting shmPath for sandbox aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:32 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977-userdata-shm.mount: Succeeded.
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.350682585Z" level=info msg="RunSandbox: removing pod sandbox from storage: 6a209eafb3eddcf8fd60d245aa9da686c76215a1623bd9989bda3b6419eef7b1" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.354094280Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.354129272Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=38b3187d-e185-4e54-b7bf-b206d1e2b629 name=/ru>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.354350 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.354407 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.354422 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.354482 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.357839393Z" level=info msg="RunSandbox: removing pod sandbox from storage: aa17da2f86406ee018f38c8dc6de6b2650645d6a75724c670700d196056d2977" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda8b name=/runtime.v1alpha2.Ru>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.360641954Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=c21ac3e3-f12c-46ff-8c99-c88c5fdd>
Mar 03 20:47:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:32.360663846Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=c21ac3e3-f12c-46ff-8c99-c88c5fddda>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.360935 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.361010 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.361023 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:32 worker-advnetlab52 hyperkube[4431]: E0303 20:47:32.361065 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:47:39 worker-advnetlab52 hyperkube[4431]: I0303 20:47:39.865744 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: I0303 20:47:40.174106 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.174374234Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.185788768Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6 NetNS:/var/run/netns/6b309897-e552-45f3-8b9b->
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.185810717Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.237678797Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.237721016Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.237747085Z" level=info msg="NetworkStart: stopping network for sandbox f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.238512889Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6 NetNS:/var/run/netns/6b309897-e552-45f3-8b9b->
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.238556726Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:40Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.276270137Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/ru>
Mar 03 20:47:40 worker-advnetlab52 systemd[1]: run-utsns-6b309897\x2de552\x2d45f3\x2d8b9b\x2d9c799c36c6b7.mount: Succeeded.
Mar 03 20:47:40 worker-advnetlab52 systemd[1]: run-ipcns-6b309897\x2de552\x2d45f3\x2d8b9b\x2d9c799c36c6b7.mount: Succeeded.
Mar 03 20:47:40 worker-advnetlab52 systemd[1]: run-netns-6b309897\x2de552\x2d45f3\x2d8b9b\x2d9c799c36c6b7.mount: Succeeded.
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.323634319Z" level=info msg="RunSandbox: deleting pod ID f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6 from idIndex" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.323693915Z" level=info msg="RunSandbox: removing pod sandbox f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.323731838Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.323764577Z" level=info msg="RunSandbox: unmounting shmPath for sandbox f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:40 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6-userdata-shm.mount: Succeeded.
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.336780522Z" level=info msg="RunSandbox: removing pod sandbox from storage: f3d6384fa3e4ff5e6b7432c4c70a3f0a1f8b4e7de96c52e7d1be670d210931f6" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.339865234Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alph>
Mar 03 20:47:40 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:40.339891813Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=37498ff9-4e68-49ac-904c-1eacd1ee0d74 name=/runtime.v1alpha2>
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: E0303 20:47:40.340138 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: E0303 20:47:40.340192 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: E0303 20:47:40.340205 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:40 worker-advnetlab52 hyperkube[4431]: E0303 20:47:40.340246 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: I0303 20:47:41.174186 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.174678699Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.187677399Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06 NetNS:/var/run/netns/1a944b63-8990-4081>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.187699889Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: I0303 20:47:41.225079 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.257386389Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.257429525Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.257452174Z" level=info msg="NetworkStart: stopping network for sandbox fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.258034998Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06 NetNS:/var/run/netns/1a944b63-8990-4081>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.258064068Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:41Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.294352461Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/ru>
Mar 03 20:47:41 worker-advnetlab52 systemd[1]: run-utsns-1a944b63\x2d8990\x2d4081\x2da91f\x2dfb0a08a4a7e1.mount: Succeeded.
Mar 03 20:47:41 worker-advnetlab52 systemd[1]: run-ipcns-1a944b63\x2d8990\x2d4081\x2da91f\x2dfb0a08a4a7e1.mount: Succeeded.
Mar 03 20:47:41 worker-advnetlab52 systemd[1]: run-netns-1a944b63\x2d8990\x2d4081\x2da91f\x2dfb0a08a4a7e1.mount: Succeeded.
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.328583372Z" level=info msg="RunSandbox: deleting pod ID fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06 from idIndex" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.328644999Z" level=info msg="RunSandbox: removing pod sandbox fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.328676081Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.328692842Z" level=info msg="RunSandbox: unmounting shmPath for sandbox fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:41 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06-userdata-shm.mount: Succeeded.
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.343807283Z" level=info msg="RunSandbox: removing pod sandbox from storage: fbd68535467fdb7ed14a524845c38ac65470f519f518c3439f3e36d3f88dad06" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1alpha2.Ru>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.346630495Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.>
Mar 03 20:47:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:41.346650610Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=ff0b6edf-463d-4ae6-8260-368bbd390bcd name=/runtime.v1>
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: E0303 20:47:41.346877 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: E0303 20:47:41.346926 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: E0303 20:47:41.346939 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:41 worker-advnetlab52 hyperkube[4431]: E0303 20:47:41.346984 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: I0303 20:47:43.174324 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.174743458Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.186166937Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e NetNS:/var/run/netns/c4c620e1-a>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.186190561Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.236513751Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.236546078Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.236571150Z" level=info msg="NetworkStart: stopping network for sandbox 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.237380784Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e NetNS:/var/run/netns/c4c620e1-a>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.237420928Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:43Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.276601705Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/ru>
Mar 03 20:47:43 worker-advnetlab52 systemd[1]: run-utsns-c4c620e1\x2da866\x2d4716\x2d93de\x2d50216d4be21a.mount: Succeeded.
Mar 03 20:47:43 worker-advnetlab52 systemd[1]: run-ipcns-c4c620e1\x2da866\x2d4716\x2d93de\x2d50216d4be21a.mount: Succeeded.
Mar 03 20:47:43 worker-advnetlab52 systemd[1]: run-netns-c4c620e1\x2da866\x2d4716\x2d93de\x2d50216d4be21a.mount: Succeeded.
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.327614566Z" level=info msg="RunSandbox: deleting pod ID 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e from idIndex" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.327682447Z" level=info msg="RunSandbox: removing pod sandbox 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.327726005Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.327761570Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:43 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e-userdata-shm.mount: Succeeded.
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.343701272Z" level=info msg="RunSandbox: removing pod sandbox from storage: 9dcd96607da41b9a8bf9bd6f7c3429cdab6ea215c0e21f83dc5aa6d5cb1b256e" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/runtime.v1alpha2.Ru>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.346637805Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/>
Mar 03 20:47:43 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:43.346658080Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=a2bceda0-ac60-4dfd-9c5a-3a10e563779b name=/ru>
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: E0303 20:47:43.346886 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: E0303 20:47:43.346936 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: E0303 20:47:43.346952 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:43 worker-advnetlab52 hyperkube[4431]: E0303 20:47:43.346991 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: I0303 20:47:44.174429 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.174809980Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.187944785Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0 NetNS:/var/run/netns/9ad4a4ab-e>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.187972371Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.237894227Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.237938469Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.237966222Z" level=info msg="NetworkStart: stopping network for sandbox f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.238631201Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0 NetNS:/var/run/netns/9ad4a4ab-e>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.238670038Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:44Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.279989259Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/ru>
Mar 03 20:47:44 worker-advnetlab52 systemd[1]: run-utsns-9ad4a4ab\x2de291\x2d42f6\x2da20f\x2d78edeaa970b5.mount: Succeeded.
Mar 03 20:47:44 worker-advnetlab52 systemd[1]: run-ipcns-9ad4a4ab\x2de291\x2d42f6\x2da20f\x2d78edeaa970b5.mount: Succeeded.
Mar 03 20:47:44 worker-advnetlab52 systemd[1]: run-netns-9ad4a4ab\x2de291\x2d42f6\x2da20f\x2d78edeaa970b5.mount: Succeeded.
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.317623386Z" level=info msg="RunSandbox: deleting pod ID f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0 from idIndex" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.317690214Z" level=info msg="RunSandbox: removing pod sandbox f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.317710529Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.317725684Z" level=info msg="RunSandbox: unmounting shmPath for sandbox f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:44 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0-userdata-shm.mount: Succeeded.
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.332700893Z" level=info msg="RunSandbox: removing pod sandbox from storage: f965ede12e2d8f5eeecb1f96e169f83838076f4b311cd49894e99b137cfdabb0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/runtime.v1alpha2.Ru>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.335658608Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/>
Mar 03 20:47:44 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:44.335680399Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=ecea61fd-8232-47d9-932c-321ab1b3b9bf name=/ru>
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: E0303 20:47:44.335946 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: E0303 20:47:44.336000 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: E0303 20:47:44.336017 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:44 worker-advnetlab52 hyperkube[4431]: E0303 20:47:44.336064 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: I0303 20:47:46.173983 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.174305788Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.185406456Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f NetNS:/var/run/netns>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.185428499Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.237911010Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.237951801Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.237978574Z" level=info msg="NetworkStart: stopping network for sandbox 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.238803524Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f NetNS:/var/run/netns>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.238843858Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:46Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.275843312Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/ru>
Mar 03 20:47:46 worker-advnetlab52 systemd[1]: run-utsns-58c87313\x2d199e\x2d43cb\x2da2c5\x2dbb8e976887ed.mount: Succeeded.
Mar 03 20:47:46 worker-advnetlab52 systemd[1]: run-ipcns-58c87313\x2d199e\x2d43cb\x2da2c5\x2dbb8e976887ed.mount: Succeeded.
Mar 03 20:47:46 worker-advnetlab52 systemd[1]: run-netns-58c87313\x2d199e\x2d43cb\x2da2c5\x2dbb8e976887ed.mount: Succeeded.
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.313615670Z" level=info msg="RunSandbox: deleting pod ID 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f from idIndex" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.313672529Z" level=info msg="RunSandbox: removing pod sandbox 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.313724625Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.313739961Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:46 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f-userdata-shm.mount: Succeeded.
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.329784804Z" level=info msg="RunSandbox: removing pod sandbox from storage: 9a168355c882f7d51023ea4518ab20ed33e22e80d3d521b30f2aa8489b8e3d1f" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a09 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.332760352Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea>
Mar 03 20:47:46 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:46.332780767Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=b96bc066-f1ac-4f7b-83c9-2bfdb2ea5a>
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: E0303 20:47:46.332964 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: E0303 20:47:46.333009 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: E0303 20:47:46.333023 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:46 worker-advnetlab52 hyperkube[4431]: E0303 20:47:46.333064 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:47:49 worker-advnetlab52 hyperkube[4431]: I0303 20:47:49.877866 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: I0303 20:47:52.174411 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.174736592Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.189100125Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0 NetNS:/var/run/netns/281280be-c3e2-40a8-9340->
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.189124415Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.240784729Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.240836339Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.240865374Z" level=info msg="NetworkStart: stopping network for sandbox 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.241500350Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0 NetNS:/var/run/netns/281280be-c3e2-40a8-9340->
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.241540258Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:52Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.279875079Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/ru>
Mar 03 20:47:52 worker-advnetlab52 systemd[1]: run-utsns-281280be\x2dc3e2\x2d40a8\x2d9340\x2d900dd4981300.mount: Succeeded.
Mar 03 20:47:52 worker-advnetlab52 systemd[1]: run-ipcns-281280be\x2dc3e2\x2d40a8\x2d9340\x2d900dd4981300.mount: Succeeded.
Mar 03 20:47:52 worker-advnetlab52 systemd[1]: run-netns-281280be\x2dc3e2\x2d40a8\x2d9340\x2d900dd4981300.mount: Succeeded.
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.340621500Z" level=info msg="RunSandbox: deleting pod ID 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0 from idIndex" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.340691154Z" level=info msg="RunSandbox: removing pod sandbox 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.340710882Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.340726254Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:52 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0-userdata-shm.mount: Succeeded.
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.355729850Z" level=info msg="RunSandbox: removing pod sandbox from storage: 7623c701754ee6d284e5de5aa0b715078f11e66eb6e36220b74ea950c9a8b0d0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2.Ru>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.358749501Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alph>
Mar 03 20:47:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:52.358775764Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=aa226d8d-999d-4be8-bf27-17f62e16493c name=/runtime.v1alpha2>
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: E0303 20:47:52.359049 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: E0303 20:47:52.359148 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: E0303 20:47:52.359179 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:52 worker-advnetlab52 hyperkube[4431]: E0303 20:47:52.359287 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: I0303 20:47:54.174339 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.174640409Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.185728731Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63 NetNS:/var/run/netns/98a6274a-15d6-46cb>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.185749274Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: I0303 20:47:54.208916 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.241263081Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.241350019Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.241414256Z" level=info msg="NetworkStart: stopping network for sandbox 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.242116624Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63 NetNS:/var/run/netns/98a6274a-15d6-46cb>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.242148664Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:54Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.282062727Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/ru>
Mar 03 20:47:54 worker-advnetlab52 systemd[1]: run-utsns-98a6274a\x2d15d6\x2d46cb\x2d812b\x2d6d08f08e72f3.mount: Succeeded.
Mar 03 20:47:54 worker-advnetlab52 systemd[1]: run-ipcns-98a6274a\x2d15d6\x2d46cb\x2d812b\x2d6d08f08e72f3.mount: Succeeded.
Mar 03 20:47:54 worker-advnetlab52 systemd[1]: run-netns-98a6274a\x2d15d6\x2d46cb\x2d812b\x2d6d08f08e72f3.mount: Succeeded.
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.320549580Z" level=info msg="RunSandbox: deleting pod ID 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63 from idIndex" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.320571037Z" level=info msg="RunSandbox: removing pod sandbox 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.320586495Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.320600138Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:54 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63-userdata-shm.mount: Succeeded.
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.331796816Z" level=info msg="RunSandbox: removing pod sandbox from storage: 0b19bad03705c008369e833543746021c53986d96cba6e5d6cb7193322555f63" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.334519411Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.>
Mar 03 20:47:54 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:54.334540425Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=3e7362e0-6370-4636-85cb-9affd7b87b51 name=/runtime.v1>
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: E0303 20:47:54.334856 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: E0303 20:47:54.334910 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: E0303 20:47:54.334923 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:54 worker-advnetlab52 hyperkube[4431]: E0303 20:47:54.334965 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: I0303 20:47:57.086368 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/keepalived-worker-advnetlab52" status=Running
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: I0303 20:47:57.086391 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/mdns-publisher-worker-advnetlab52" status=Running
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: I0303 20:47:57.086399 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/coredns-worker-advnetlab52" status=Running
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: I0303 20:47:57.174705 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: I0303 20:47:57.174816 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.175012779Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.175058512Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.194342107Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a NetNS:/var/run/netns/37a1f44c-5>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.194392250Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.195569174Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d NetNS:/var/run/netns/10fbcc35-8>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.195597641Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.246381465Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.246432540Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.246461644Z" level=info msg="NetworkStart: stopping network for sandbox 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.247187112Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d NetNS:/var/run/netns/10fbcc35-8>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.247223093Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.247929856Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.247969420Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.247992826Z" level=info msg="NetworkStart: stopping network for sandbox 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.248593272Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a NetNS:/var/run/netns/37a1f44c-5>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.248627997Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:57Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.285523247Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/ru>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:57Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.286831726Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/ru>
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-utsns-10fbcc35\x2d86c7\x2d49da\x2db281\x2d2550f0615a65.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-utsns-37a1f44c\x2d51c2\x2d4a43\x2d9e66\x2dfb3beb5335ad.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-ipcns-10fbcc35\x2d86c7\x2d49da\x2db281\x2d2550f0615a65.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-ipcns-37a1f44c\x2d51c2\x2d4a43\x2d9e66\x2dfb3beb5335ad.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: I0303 20:47:57.311929 4431 container_manager_linux.go:986] Found 790 PIDs in root, 790 of them are not to be moved
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-netns-10fbcc35\x2d86c7\x2d49da\x2db281\x2d2550f0615a65.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-netns-37a1f44c\x2d51c2\x2d4a43\x2d9e66\x2dfb3beb5335ad.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.340616072Z" level=info msg="RunSandbox: deleting pod ID 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a from idIndex" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.340647150Z" level=info msg="RunSandbox: removing pod sandbox 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.340664621Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.340683165Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.341561269Z" level=info msg="RunSandbox: deleting pod ID 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d from idIndex" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.341594284Z" level=info msg="RunSandbox: removing pod sandbox 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.341610970Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.341625380Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a-userdata-shm.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d-userdata-shm.mount: Succeeded.
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.356672206Z" level=info msg="RunSandbox: removing pod sandbox from storage: 1cd421312c7bf9538028c230882dfb25960fb51956bd0e12da6a0b5ca8318e9a" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/runtime.v1alpha2.Ru>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.356697081Z" level=info msg="RunSandbox: removing pod sandbox from storage: 170ab27e333fb33aa522e28ab29da921697fe4e242a00c756c0120cb1e78bc5d" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/runtime.v1alpha2.Ru>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.359613502Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.359634890Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=a783c9f7-c80a-45b0-90aa-78a28e092d8a name=/ru>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.359926 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.360002 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.360015 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.360062 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.362783754Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/>
Mar 03 20:47:57 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:57.362811206Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=e0f84621-dca3-4084-8d78-81c2747a551d name=/ru>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.363013 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.363043 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.363056 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:57 worker-advnetlab52 hyperkube[4431]: E0303 20:47:57.363089 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: I0303 20:47:59.174183 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.174478272Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.185690851Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96 NetNS:/var/run/netns>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.185715133Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.235586846Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.235624664Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.235652801Z" level=info msg="NetworkStart: stopping network for sandbox 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.236218618Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96 NetNS:/var/run/netns>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.236253190Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: 2021-03-03T20:47:59Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.276569441Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/ru>
Mar 03 20:47:59 worker-advnetlab52 systemd[1]: run-utsns-83c3a8f2\x2d02ea\x2d4e64\x2da3dc\x2dc5c6d701afc8.mount: Succeeded.
Mar 03 20:47:59 worker-advnetlab52 systemd[1]: run-ipcns-83c3a8f2\x2d02ea\x2d4e64\x2da3dc\x2dc5c6d701afc8.mount: Succeeded.
Mar 03 20:47:59 worker-advnetlab52 systemd[1]: run-netns-83c3a8f2\x2d02ea\x2d4e64\x2da3dc\x2dc5c6d701afc8.mount: Succeeded.
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.319623989Z" level=info msg="RunSandbox: deleting pod ID 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96 from idIndex" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.319698848Z" level=info msg="RunSandbox: removing pod sandbox 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.319758589Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.319794365Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime.v1alpha2.Runtim>
Mar 03 20:47:59 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96-userdata-shm.mount: Succeeded.
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.335860007Z" level=info msg="RunSandbox: removing pod sandbox from storage: 167f7e0427abb1f3e937dbf1894b3e028b39659dc294ff22404e3f1092558b96" id=dca08587-639c-4176-a6f9-aaf9c0626368 name=/runtime.v1alpha2.Ru>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.340869340Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=dca08587-639c-4176-a6f9-aaf9c062>
Mar 03 20:47:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:47:59.340896167Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=dca08587-639c-4176-a6f9-aaf9c06263>
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: E0303 20:47:59.341069 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: E0303 20:47:59.341108 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: E0303 20:47:59.341120 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: E0303 20:47:59.341164 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:47:59 worker-advnetlab52 hyperkube[4431]: I0303 20:47:59.890328 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: I0303 20:48:05.174320 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.174763011Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.186834182Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56 NetNS:/var/run/netns/7619c4f7-36f0-4cde>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.186860757Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: I0303 20:48:05.211200 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.243937740Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.243976386Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.244003878Z" level=info msg="NetworkStart: stopping network for sandbox ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.244687709Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56 NetNS:/var/run/netns/7619c4f7-36f0-4cde>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.244722163Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:05Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.281908552Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/ru>
Mar 03 20:48:05 worker-advnetlab52 systemd[1]: run-utsns-7619c4f7\x2d36f0\x2d4cde\x2d9f85\x2d300dd397f0ef.mount: Succeeded.
Mar 03 20:48:05 worker-advnetlab52 systemd[1]: run-ipcns-7619c4f7\x2d36f0\x2d4cde\x2d9f85\x2d300dd397f0ef.mount: Succeeded.
Mar 03 20:48:05 worker-advnetlab52 systemd[1]: run-netns-7619c4f7\x2d36f0\x2d4cde\x2d9f85\x2d300dd397f0ef.mount: Succeeded.
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.330584078Z" level=info msg="RunSandbox: deleting pod ID ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56 from idIndex" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.330626040Z" level=info msg="RunSandbox: removing pod sandbox ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.330641282Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.330665898Z" level=info msg="RunSandbox: unmounting shmPath for sandbox ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:05 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56-userdata-shm.mount: Succeeded.
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.342685371Z" level=info msg="RunSandbox: removing pod sandbox from storage: ab66e929d202755e8778ace210829ca28d7720d019240362b893edca65500d56" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.345392662Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.>
Mar 03 20:48:05 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:05.345423081Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=c7bdb3ed-3ccb-4d55-b6a6-dc6d44a52833 name=/runtime.v1>
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: E0303 20:48:05.345727 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: E0303 20:48:05.345810 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: E0303 20:48:05.345842 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:05 worker-advnetlab52 hyperkube[4431]: E0303 20:48:05.345939 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:48:05 worker-advnetlab52 systemd[1]: run-runc-06878f1eb9b31ec278289d6d9c0480bd4a03afe47ed3fe93d5991ea131492cb9-runc.fRIAoR.mount: Succeeded.
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: I0303 20:48:06.174188 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.174487923Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.186134536Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b NetNS:/var/run/netns/dc14d9b6-c32f-4f15-b01c->
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.186161019Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.239048396Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.239120846Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.239174162Z" level=info msg="NetworkStart: stopping network for sandbox d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.239869195Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b NetNS:/var/run/netns/dc14d9b6-c32f-4f15-b01c->
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.239901122Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:06Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.276144452Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/ru>
Mar 03 20:48:06 worker-advnetlab52 systemd[1]: run-utsns-dc14d9b6\x2dc32f\x2d4f15\x2db01c\x2db3275358d5d8.mount: Succeeded.
Mar 03 20:48:06 worker-advnetlab52 systemd[1]: run-ipcns-dc14d9b6\x2dc32f\x2d4f15\x2db01c\x2db3275358d5d8.mount: Succeeded.
Mar 03 20:48:06 worker-advnetlab52 systemd[1]: run-netns-dc14d9b6\x2dc32f\x2d4f15\x2db01c\x2db3275358d5d8.mount: Succeeded.
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.319626915Z" level=info msg="RunSandbox: deleting pod ID d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b from idIndex" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.319686748Z" level=info msg="RunSandbox: removing pod sandbox d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.319740727Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.319777114Z" level=info msg="RunSandbox: unmounting shmPath for sandbox d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:06 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b-userdata-shm.mount: Succeeded.
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.331799850Z" level=info msg="RunSandbox: removing pod sandbox from storage: d2422a290e7a2c5545f8cdb8bc019e7fa9bf79adc203db317b64efb6af651d7b" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.334739156Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alph>
Mar 03 20:48:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:06.334762193Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=d8f2da8d-3717-4aef-86b8-3de2f6c726e0 name=/runtime.v1alpha2>
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: E0303 20:48:06.334929 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: E0303 20:48:06.334961 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: E0303 20:48:06.334974 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:06 worker-advnetlab52 hyperkube[4431]: E0303 20:48:06.335013 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:48:09 worker-advnetlab52 hyperkube[4431]: I0303 20:48:09.900402 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: I0303 20:48:10.174243 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.174566283Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.185544474Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7 NetNS:/var/run/netns>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.185578382Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.236498205Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.236548342Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.236578307Z" level=info msg="NetworkStart: stopping network for sandbox 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.237239318Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7 NetNS:/var/run/netns>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.237272709Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:10Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.278407721Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/ru>
Mar 03 20:48:10 worker-advnetlab52 systemd[1]: run-utsns-1a888c6a\x2dc7c9\x2d457c\x2dbcbf\x2d4d4fd2c851ac.mount: Succeeded.
Mar 03 20:48:10 worker-advnetlab52 systemd[1]: run-ipcns-1a888c6a\x2dc7c9\x2d457c\x2dbcbf\x2d4d4fd2c851ac.mount: Succeeded.
Mar 03 20:48:10 worker-advnetlab52 systemd[1]: run-netns-1a888c6a\x2dc7c9\x2d457c\x2dbcbf\x2d4d4fd2c851ac.mount: Succeeded.
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.320614459Z" level=info msg="RunSandbox: deleting pod ID 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7 from idIndex" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.320675291Z" level=info msg="RunSandbox: removing pod sandbox 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.320714290Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.320732342Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:10 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7-userdata-shm.mount: Succeeded.
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.334832163Z" level=info msg="RunSandbox: removing pod sandbox from storage: 46b52622f2240a4f38ce445ad5f73be5a7c4089ee8e0b80e2c2b970926729fe7" id=8f1bbc6c-d468-4409-a792-fc4a01001687 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.337577821Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=8f1bbc6c-d468-4409-a792-fc4a0100>
Mar 03 20:48:10 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:10.337598052Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=8f1bbc6c-d468-4409-a792-fc4a010016>
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: E0303 20:48:10.337828 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: E0303 20:48:10.337878 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: E0303 20:48:10.337891 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:10 worker-advnetlab52 hyperkube[4431]: E0303 20:48:10.337937 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: I0303 20:48:12.174380 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: I0303 20:48:12.174443 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.174749763Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.174782141Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.190885851Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e NetNS:/var/run/netns/1b23ae9b-5>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.190910125Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.191715724Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039 NetNS:/var/run/netns/63f5e47f-e>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.191739379Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.243691181Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.243731456Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.243691494Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.243804565Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.243826870Z" level=info msg="NetworkStart: stopping network for sandbox b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.243772207Z" level=info msg="NetworkStart: stopping network for sandbox 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.244468447Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039 NetNS:/var/run/netns/63f5e47f-e>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.244502261Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.244541845Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e NetNS:/var/run/netns/1b23ae9b-5>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.244575750Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:12Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.282822916Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/ru>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:12Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.284275621Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/ru>
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-utsns-63f5e47f\x2de1f2\x2d4511\x2d85ba\x2d5456036bafcd.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-utsns-1b23ae9b\x2d5b3a\x2d4901\x2d8827\x2dbacf922ef3d6.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-ipcns-63f5e47f\x2de1f2\x2d4511\x2d85ba\x2d5456036bafcd.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-ipcns-1b23ae9b\x2d5b3a\x2d4901\x2d8827\x2dbacf922ef3d6.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-netns-63f5e47f\x2de1f2\x2d4511\x2d85ba\x2d5456036bafcd.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-netns-1b23ae9b\x2d5b3a\x2d4901\x2d8827\x2dbacf922ef3d6.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.339561560Z" level=info msg="RunSandbox: deleting pod ID b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039 from idIndex" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.339591202Z" level=info msg="RunSandbox: removing pod sandbox b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.339608484Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.339631476Z" level=info msg="RunSandbox: unmounting shmPath for sandbox b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039-userdata-shm.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.348577025Z" level=info msg="RunSandbox: deleting pod ID 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e from idIndex" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.348616377Z" level=info msg="RunSandbox: removing pod sandbox 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.348636245Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.348650958Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:12 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e-userdata-shm.mount: Succeeded.
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.359647567Z" level=info msg="RunSandbox: removing pod sandbox from storage: b29a784f2ec89444e40a9e5f26e7558932c0a6feca59395f6081ac9cbea47039" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.364571344Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.364601679Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=a52f374a-fee1-47b3-bb2a-1c82a746d017 name=/ru>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.364763 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.364809 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.364822 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.364866 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.367706617Z" level=info msg="RunSandbox: removing pod sandbox from storage: 0f50cb1ca5e451f988ab2c418993b1f0f9840ab0f59e909c0a422f25045a8f4e" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.370560661Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/>
Mar 03 20:48:12 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:12.370582313Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=f4540261-d9c9-45bc-9367-5cfad6a834a9 name=/ru>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.370706 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.370740 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.370750 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:12 worker-advnetlab52 hyperkube[4431]: E0303 20:48:12.370788 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4074] policy: auto-activating connection 'Wired connection 1' (c6e9aa15-94ad-3967-9283-3a0c402f495a)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4078] policy: auto-activating connection 'Wired connection 5' (c5ac7bab-9767-3a07-96e6-e5fdf6b54c90)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4080] policy: auto-activating connection 'Wired connection 6' (eda6fa6e-7dcf-34d4-a9cf-d2e490bf56f9)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4083] device (eno1): Activation: starting connection 'Wired connection 1' (c6e9aa15-94ad-3967-9283-3a0c402f495a)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4085] device (ens1f1): Activation: starting connection 'Wired connection 5' (c5ac7bab-9767-3a07-96e6-e5fdf6b54c90)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4086] device (ens3f0): Activation: starting connection 'Wired connection 6' (eda6fa6e-7dcf-34d4-a9cf-d2e490bf56f9)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4087] device (eno1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4089] device (eno1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4097] device (ens1f1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4099] device (ens1f1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4102] device (ens3f0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4104] device (ens3f0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4107] device (eno1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4109] dhcp4 (eno1): activation: beginning transaction (timeout in 45 seconds)
Mar 03 20:48:17 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00115|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4129] device (ens1f1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4131] dhcp4 (ens1f1): activation: beginning transaction (timeout in 45 seconds)
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4140] device (ens3f0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:48:17 worker-advnetlab52 NetworkManager[2521]: <info> [1614804497.4142] dhcp4 (ens3f0): activation: beginning transaction (timeout in 45 seconds)
Mar 03 20:48:17 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00116|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:48:17 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00117|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:48:19 worker-advnetlab52 hyperkube[4431]: I0303 20:48:19.909357 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: I0303 20:48:20.174281 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.174675282Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.186770677Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5 NetNS:/var/run/netns/dd6e450f-31fd-421b>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.186791524Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: I0303 20:48:20.211556 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.246074567Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.246115962Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.246145373Z" level=info msg="NetworkStart: stopping network for sandbox 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.246792554Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5 NetNS:/var/run/netns/dd6e450f-31fd-421b>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.246826998Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:20Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.283556084Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/ru>
Mar 03 20:48:20 worker-advnetlab52 systemd[1]: run-utsns-dd6e450f\x2d31fd\x2d421b\x2dae90\x2d5f9af6db01fc.mount: Succeeded.
Mar 03 20:48:20 worker-advnetlab52 systemd[1]: run-ipcns-dd6e450f\x2d31fd\x2d421b\x2dae90\x2d5f9af6db01fc.mount: Succeeded.
Mar 03 20:48:20 worker-advnetlab52 systemd[1]: run-netns-dd6e450f\x2d31fd\x2d421b\x2dae90\x2d5f9af6db01fc.mount: Succeeded.
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.324609636Z" level=info msg="RunSandbox: deleting pod ID 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5 from idIndex" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.324669171Z" level=info msg="RunSandbox: removing pod sandbox 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.324723449Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.324764603Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:20 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5-userdata-shm.mount: Succeeded.
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.337739354Z" level=info msg="RunSandbox: removing pod sandbox from storage: 0ff4a2a2e803e52688d2bd340b3b92503279c872b4aff1d3eb4a857cffe30fc5" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.340599798Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.>
Mar 03 20:48:20 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:20.340621119Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=f31e301c-77d7-4ff1-ad1c-05bbb9786da8 name=/runtime.v1>
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: E0303 20:48:20.340934 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: E0303 20:48:20.341026 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: E0303 20:48:20.341038 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:20 worker-advnetlab52 hyperkube[4431]: E0303 20:48:20.341083 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: I0303 20:48:21.174225 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.174558934Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.187560789Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a NetNS:/var/run/netns/c9766203-5537-437a-819e->
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.187585616Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.242596059Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.242666561Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.242723939Z" level=info msg="NetworkStart: stopping network for sandbox 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.243640794Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a NetNS:/var/run/netns/c9766203-5537-437a-819e->
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.243683491Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:21Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.280780528Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/ru>
Mar 03 20:48:21 worker-advnetlab52 systemd[1]: run-utsns-c9766203\x2d5537\x2d437a\x2d819e\x2d012c57817a0e.mount: Succeeded.
Mar 03 20:48:21 worker-advnetlab52 systemd[1]: run-ipcns-c9766203\x2d5537\x2d437a\x2d819e\x2d012c57817a0e.mount: Succeeded.
Mar 03 20:48:21 worker-advnetlab52 systemd[1]: run-netns-c9766203\x2d5537\x2d437a\x2d819e\x2d012c57817a0e.mount: Succeeded.
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.321618867Z" level=info msg="RunSandbox: deleting pod ID 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a from idIndex" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.321675760Z" level=info msg="RunSandbox: removing pod sandbox 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.321732248Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.321767087Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:21 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a-userdata-shm.mount: Succeeded.
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.333770635Z" level=info msg="RunSandbox: removing pod sandbox from storage: 7b283c5e94a470747ed68db34ee49516ee0904dc591347d20de437c101e6492a" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.336576880Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alph>
Mar 03 20:48:21 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:21.336597947Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=f2887c2f-dfd4-4c8c-9022-5a49bb186112 name=/runtime.v1alpha2>
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: E0303 20:48:21.336855 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: E0303 20:48:21.336901 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: E0303 20:48:21.336920 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:21 worker-advnetlab52 hyperkube[4431]: E0303 20:48:21.336970 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: I0303 20:48:25.174454 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.175002872Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.186648158Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282 NetNS:/var/run/netns/b2d51acd-8>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.186674585Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.238517621Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.238562612Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.238590531Z" level=info msg="NetworkStart: stopping network for sandbox 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.239247636Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282 NetNS:/var/run/netns/b2d51acd-8>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.239282181Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:25Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.277066826Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/ru>
Mar 03 20:48:25 worker-advnetlab52 systemd[1]: run-utsns-b2d51acd\x2d801a\x2d4a8f\x2d8e46\x2d45a21019c7c0.mount: Succeeded.
Mar 03 20:48:25 worker-advnetlab52 systemd[1]: run-ipcns-b2d51acd\x2d801a\x2d4a8f\x2d8e46\x2d45a21019c7c0.mount: Succeeded.
Mar 03 20:48:25 worker-advnetlab52 systemd[1]: run-netns-b2d51acd\x2d801a\x2d4a8f\x2d8e46\x2d45a21019c7c0.mount: Succeeded.
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.328642415Z" level=info msg="RunSandbox: deleting pod ID 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282 from idIndex" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.328708983Z" level=info msg="RunSandbox: removing pod sandbox 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.328750695Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.328785236Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:25 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282-userdata-shm.mount: Succeeded.
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.344810866Z" level=info msg="RunSandbox: removing pod sandbox from storage: 7fd7a57a2af5009def83ff976b6b5bf96719c21e611a5b1446684abfc57e6282" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.348295909Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/>
Mar 03 20:48:25 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:25.348315707Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=6c0dba9f-a59e-408e-a034-3d915ed78726 name=/ru>
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: E0303 20:48:25.348484 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: E0303 20:48:25.348530 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: E0303 20:48:25.348542 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:25 worker-advnetlab52 hyperkube[4431]: E0303 20:48:25.348584 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: I0303 20:48:26.174026 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.174461289Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.185949308Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8 NetNS:/var/run/netns>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.185975044Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.235889177Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.235924542Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.235952830Z" level=info msg="NetworkStart: stopping network for sandbox 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.236601554Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8 NetNS:/var/run/netns>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.236636617Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:26Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.275788442Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/ru>
Mar 03 20:48:26 worker-advnetlab52 systemd[1]: run-utsns-6bac7c34\x2d042b\x2d4cff\x2db878\x2de3f2b82447bd.mount: Succeeded.
Mar 03 20:48:26 worker-advnetlab52 systemd[1]: run-ipcns-6bac7c34\x2d042b\x2d4cff\x2db878\x2de3f2b82447bd.mount: Succeeded.
Mar 03 20:48:26 worker-advnetlab52 systemd[1]: run-netns-6bac7c34\x2d042b\x2d4cff\x2db878\x2de3f2b82447bd.mount: Succeeded.
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.323626152Z" level=info msg="RunSandbox: deleting pod ID 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8 from idIndex" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.323693842Z" level=info msg="RunSandbox: removing pod sandbox 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.323740973Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.323784418Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:26 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8-userdata-shm.mount: Succeeded.
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.332776980Z" level=info msg="RunSandbox: removing pod sandbox from storage: 6c126d5dcffd94f8951663a332286c995399151eb3609cca2227eab762caf5e8" id=3378d7f4-1ea5-449c-8d7a-ea9245ab0062 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.335700440Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=3378d7f4-1ea5-449c-8d7a-ea9245ab>
Mar 03 20:48:26 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:26.335723117Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=3378d7f4-1ea5-449c-8d7a-ea9245ab00>
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: E0303 20:48:26.335899 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: E0303 20:48:26.335970 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: E0303 20:48:26.335986 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:26 worker-advnetlab52 hyperkube[4431]: E0303 20:48:26.336031 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: I0303 20:48:27.174704 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.175004137Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.188344564Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e NetNS:/var/run/netns/6d099526-6>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.188371418Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.238264169Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.238349377Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.238406832Z" level=info msg="NetworkStart: stopping network for sandbox b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.239073610Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e NetNS:/var/run/netns/6d099526-6>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.239103429Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:27Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.281723862Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/ru>
Mar 03 20:48:27 worker-advnetlab52 systemd[1]: run-utsns-6d099526\x2d6b53\x2d4fa8\x2d95c4\x2dcb8f6763980f.mount: Succeeded.
Mar 03 20:48:27 worker-advnetlab52 systemd[1]: run-ipcns-6d099526\x2d6b53\x2d4fa8\x2d95c4\x2dcb8f6763980f.mount: Succeeded.
Mar 03 20:48:27 worker-advnetlab52 systemd[1]: run-netns-6d099526\x2d6b53\x2d4fa8\x2d95c4\x2dcb8f6763980f.mount: Succeeded.
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.335580926Z" level=info msg="RunSandbox: deleting pod ID b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e from idIndex" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.335607435Z" level=info msg="RunSandbox: removing pod sandbox b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.335626266Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.335641196Z" level=info msg="RunSandbox: unmounting shmPath for sandbox b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:27 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e-userdata-shm.mount: Succeeded.
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.354639395Z" level=info msg="RunSandbox: removing pod sandbox from storage: b308c44b4869774b257c54d5ffe048aca703e77b12b3c2d72f633ae07330d35e" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.357540802Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/>
Mar 03 20:48:27 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:27.357561070Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=8d9557c7-4f64-42cd-8e4c-8504f07b12f7 name=/ru>
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: E0303 20:48:27.357753 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: E0303 20:48:27.357795 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: E0303 20:48:27.357809 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:27 worker-advnetlab52 hyperkube[4431]: E0303 20:48:27.357851 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:29 worker-advnetlab52 hyperkube[4431]: I0303 20:48:29.917862 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: I0303 20:48:32.174353 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.174682839Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.186553048Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5 NetNS:/var/run/netns/99300130-8bfa-4add-b77d->
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.186579540Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.239593836Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.239668873Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.239698899Z" level=info msg="NetworkStart: stopping network for sandbox f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.240305114Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5 NetNS:/var/run/netns/99300130-8bfa-4add-b77d->
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.240338396Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:32Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.278669038Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/ru>
Mar 03 20:48:32 worker-advnetlab52 systemd[1]: run-utsns-99300130\x2d8bfa\x2d4add\x2db77d\x2d436c091dc4f6.mount: Succeeded.
Mar 03 20:48:32 worker-advnetlab52 systemd[1]: run-ipcns-99300130\x2d8bfa\x2d4add\x2db77d\x2d436c091dc4f6.mount: Succeeded.
Mar 03 20:48:32 worker-advnetlab52 systemd[1]: run-netns-99300130\x2d8bfa\x2d4add\x2db77d\x2d436c091dc4f6.mount: Succeeded.
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.321623482Z" level=info msg="RunSandbox: deleting pod ID f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5 from idIndex" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.321702204Z" level=info msg="RunSandbox: removing pod sandbox f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.321744856Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.321784108Z" level=info msg="RunSandbox: unmounting shmPath for sandbox f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:32 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5-userdata-shm.mount: Succeeded.
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.333790263Z" level=info msg="RunSandbox: removing pod sandbox from storage: f3a8a8348ccb94ae8ce8614a37a6fc79c341fcafe0420ac9cbb05f9fce4630c5" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.336593028Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alph>
Mar 03 20:48:32 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:32.336613442Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=f1d44669-14b2-4022-b6f1-d38ad594df71 name=/runtime.v1alpha2>
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: E0303 20:48:32.336900 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: E0303 20:48:32.336983 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: E0303 20:48:32.337015 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:32 worker-advnetlab52 hyperkube[4431]: E0303 20:48:32.337073 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: I0303 20:48:34.174105 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.174407330Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.186843597Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98 NetNS:/var/run/netns/fa93fc49-6ddc-49dc>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.186867682Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: I0303 20:48:34.211983 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.246130537Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.246206595Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.246272121Z" level=info msg="NetworkStart: stopping network for sandbox 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.247137077Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98 NetNS:/var/run/netns/fa93fc49-6ddc-49dc>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.247184930Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:34Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.283279339Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/ru>
Mar 03 20:48:34 worker-advnetlab52 systemd[1]: run-utsns-fa93fc49\x2d6ddc\x2d49dc\x2dad55\x2d38c34aedefc2.mount: Succeeded.
Mar 03 20:48:34 worker-advnetlab52 systemd[1]: run-ipcns-fa93fc49\x2d6ddc\x2d49dc\x2dad55\x2d38c34aedefc2.mount: Succeeded.
Mar 03 20:48:34 worker-advnetlab52 systemd[1]: run-netns-fa93fc49\x2d6ddc\x2d49dc\x2dad55\x2d38c34aedefc2.mount: Succeeded.
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.327617222Z" level=info msg="RunSandbox: deleting pod ID 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98 from idIndex" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.327682574Z" level=info msg="RunSandbox: removing pod sandbox 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.327721273Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.327753917Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98-userdata-shm.mount: Succeeded.
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.339796356Z" level=info msg="RunSandbox: removing pod sandbox from storage: 60ed800de81557bc6267fb0275d4bf2e9b516cd00639dc3c4bf0a4e6b6fc6a98" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1alpha2.Ru>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.342707093Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.>
Mar 03 20:48:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:34.342728003Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=222a8c15-7418-4dc8-b8c5-dee7d7b607bb name=/runtime.v1>
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: E0303 20:48:34.342970 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: E0303 20:48:34.343014 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: E0303 20:48:34.343028 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:34 worker-advnetlab52 hyperkube[4431]: E0303 20:48:34.343075 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: I0303 20:48:38.174246 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: I0303 20:48:38.174564 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.174604864Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.174787866Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.191185330Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0 NetNS:/var/run/netns/660fc159-e>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.191211656Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.191875356Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad NetNS:/var/run/netns/936f2dfc-d>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.191899138Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.242140598Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.242184540Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.242216857Z" level=info msg="NetworkStart: stopping network for sandbox b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.243153983Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad NetNS:/var/run/netns/936f2dfc-d>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.243188946Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.244340202Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.244374837Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.244400797Z" level=info msg="NetworkStart: stopping network for sandbox 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.245061422Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0 NetNS:/var/run/netns/660fc159-e>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.245096512Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:38Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.281549235Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/ru>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:38Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.281937312Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/ru>
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-utsns-936f2dfc\x2ddfe4\x2d44ea\x2dadb0\x2d26dd62a049eb.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-utsns-660fc159\x2de8cb\x2d470f\x2dbc61\x2d054ac034349c.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-ipcns-936f2dfc\x2ddfe4\x2d44ea\x2dadb0\x2d26dd62a049eb.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-ipcns-660fc159\x2de8cb\x2d470f\x2dbc61\x2d054ac034349c.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-netns-936f2dfc\x2ddfe4\x2d44ea\x2dadb0\x2d26dd62a049eb.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-netns-660fc159\x2de8cb\x2d470f\x2dbc61\x2d054ac034349c.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329586330Z" level=info msg="RunSandbox: deleting pod ID b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad from idIndex" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329622467Z" level=info msg="RunSandbox: removing pod sandbox b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329639685Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329647056Z" level=info msg="RunSandbox: deleting pod ID 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0 from idIndex" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329710385Z" level=info msg="RunSandbox: removing pod sandbox 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329770426Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329807146Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.329654341Z" level=info msg="RunSandbox: unmounting shmPath for sandbox b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad-userdata-shm.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0-userdata-shm.mount: Succeeded.
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.341832293Z" level=info msg="RunSandbox: removing pod sandbox from storage: b68ae9d0fad88ae2bcace2e3a1a41e81476a89db8d1f1b9ec59cd38e05a5a8ad" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/runtime.v1alpha2.Ru>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.346687215Z" level=info msg="RunSandbox: removing pod sandbox from storage: 870a481b44e0f8ecb78e18d937981a217e3158248d285701beab670c8a5963a0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.346975258Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.346991591Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=00bdeeea-5c97-4278-8ec5-06ffb31fd14c name=/ru>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.347189 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.347234 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.347248 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.347291 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.350044032Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/>
Mar 03 20:48:38 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:38.350067489Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=1ece2d60-0423-45fa-8158-d202367684a1 name=/ru>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.350206 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.350229 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.350257 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:38 worker-advnetlab52 hyperkube[4431]: E0303 20:48:38.350290 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:39 worker-advnetlab52 hyperkube[4431]: I0303 20:48:39.927258 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: I0303 20:48:41.174437 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.174712429Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.186387324Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e NetNS:/var/run/netns>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.186415806Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.247565066Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.247639553Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.247707423Z" level=info msg="NetworkStart: stopping network for sandbox d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.248519182Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e NetNS:/var/run/netns>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.248561871Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:41Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.284409360Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/ru>
Mar 03 20:48:41 worker-advnetlab52 systemd[1]: run-utsns-031dd90c\x2d75a2\x2d4f01\x2db1d7\x2df7b43555a769.mount: Succeeded.
Mar 03 20:48:41 worker-advnetlab52 systemd[1]: run-ipcns-031dd90c\x2d75a2\x2d4f01\x2db1d7\x2df7b43555a769.mount: Succeeded.
Mar 03 20:48:41 worker-advnetlab52 systemd[1]: run-netns-031dd90c\x2d75a2\x2d4f01\x2db1d7\x2df7b43555a769.mount: Succeeded.
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.330663075Z" level=info msg="RunSandbox: deleting pod ID d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e from idIndex" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.330704125Z" level=info msg="RunSandbox: removing pod sandbox d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.330721092Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.330736700Z" level=info msg="RunSandbox: unmounting shmPath for sandbox d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:41 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e-userdata-shm.mount: Succeeded.
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.339641051Z" level=info msg="RunSandbox: removing pod sandbox from storage: d535c1e7f5969d5f4b1552a6e5abcf58e4e2e697f1924a0173950eadc6feeb3e" id=1c19a5ad-8704-4b00-a987-d4de8050eb2c name=/runtime.v1alpha2.Ru>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.342643214Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=1c19a5ad-8704-4b00-a987-d4de8050>
Mar 03 20:48:41 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:41.342671962Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=1c19a5ad-8704-4b00-a987-d4de8050eb>
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: E0303 20:48:41.342869 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: E0303 20:48:41.342913 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: E0303 20:48:41.342928 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:41 worker-advnetlab52 hyperkube[4431]: E0303 20:48:41.342977 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: I0303 20:48:45.174420 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: I0303 20:48:45.174520 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.174813510Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.174900585Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.193016465Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d NetNS:/var/run/netns/58649250-b1ba-43a7>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.193048811Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.194400345Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5 NetNS:/var/run/netns/06820157-407e-46ed-91fe->
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.194425016Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: W0303 20:48:45.214483 4431 http2_server.go:482] transport: http2Server.HandleStreams failed to read frame: read unix /var/lib/kubelet/pod-resources/769277831->@: read: connection reset by peer
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: I0303 20:48:45.214537 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.244295563Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.244344182Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.244373298Z" level=info msg="NetworkStart: stopping network for sandbox 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.245089052Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5 NetNS:/var/run/netns/06820157-407e-46ed-91fe->
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.245124805Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.246753634Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.246827021Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.246892076Z" level=info msg="NetworkStart: stopping network for sandbox 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.247551226Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d NetNS:/var/run/netns/58649250-b1ba-43a7>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.247585622Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:45Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.282564123Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/ru>
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-utsns-58649250\x2db1ba\x2d43a7\x2d9d5f\x2d2a07c9484664.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:45Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.288378199Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/ru>
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-utsns-06820157\x2d407e\x2d46ed\x2d91fe\x2daf753231dddb.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-ipcns-58649250\x2db1ba\x2d43a7\x2d9d5f\x2d2a07c9484664.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-ipcns-06820157\x2d407e\x2d46ed\x2d91fe\x2daf753231dddb.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-netns-06820157\x2d407e\x2d46ed\x2d91fe\x2daf753231dddb.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.335592743Z" level=info msg="RunSandbox: deleting pod ID 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5 from idIndex" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.335626863Z" level=info msg="RunSandbox: removing pod sandbox 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.335643312Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.335658643Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-netns-58649250\x2db1ba\x2d43a7\x2d9d5f\x2d2a07c9484664.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5-userdata-shm.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.348626067Z" level=info msg="RunSandbox: deleting pod ID 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d from idIndex" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.348710285Z" level=info msg="RunSandbox: removing pod sandbox 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.348724779Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.348737916Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:45 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d-userdata-shm.mount: Succeeded.
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.351642499Z" level=info msg="RunSandbox: removing pod sandbox from storage: 69d10cc6ec4a06eb1cce9b7bbbdd78581a9f7c41d2712cb04be302f57260a6b5" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.354414321Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alph>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.354436940Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=b19367a5-3de4-4bfc-ace0-6dd1fd4fb230 name=/runtime.v1alpha2>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.354614 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.354653 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.354667 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.354714 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.360810918Z" level=info msg="RunSandbox: removing pod sandbox from storage: 1641fd8542c3cc08fbceb4298d469d025abc8578dc643765ae414161ba5ab86d" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1alpha2.Ru>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.363556564Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.>
Mar 03 20:48:45 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:45.363587166Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=ae7c8e61-8177-476f-8b35-8623e4e1276e name=/runtime.v1>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.363778 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.363823 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.363833 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:45 worker-advnetlab52 hyperkube[4431]: E0303 20:48:45.363871 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: I0303 20:48:49.174116 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.174571879Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.185787772Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43 NetNS:/var/run/netns/61dd3043-f>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.185817687Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.236364949Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.236403019Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.236436220Z" level=info msg="NetworkStart: stopping network for sandbox 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.237204255Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43 NetNS:/var/run/netns/61dd3043-f>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.237251683Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:49Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.276491003Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/ru>
Mar 03 20:48:49 worker-advnetlab52 systemd[1]: run-utsns-61dd3043\x2dff6c\x2d4e16\x2dbb40\x2d2bfaacf8a84f.mount: Succeeded.
Mar 03 20:48:49 worker-advnetlab52 systemd[1]: run-ipcns-61dd3043\x2dff6c\x2d4e16\x2dbb40\x2d2bfaacf8a84f.mount: Succeeded.
Mar 03 20:48:49 worker-advnetlab52 systemd[1]: run-netns-61dd3043\x2dff6c\x2d4e16\x2dbb40\x2d2bfaacf8a84f.mount: Succeeded.
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.311620450Z" level=info msg="RunSandbox: deleting pod ID 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43 from idIndex" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.311689218Z" level=info msg="RunSandbox: removing pod sandbox 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.311730285Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.311744283Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:49 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43-userdata-shm.mount: Succeeded.
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.322836041Z" level=info msg="RunSandbox: removing pod sandbox from storage: 56129981fa19a707bef188038c8f658a2b8f8206e778d837615b3da91c82ed43" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.327872370Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/>
Mar 03 20:48:49 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:49.327906048Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=6d51ac0b-363f-4477-a7b9-f698f1a4f282 name=/ru>
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: E0303 20:48:49.328133 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: E0303 20:48:49.328189 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: E0303 20:48:49.328203 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: E0303 20:48:49.328248 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:49 worker-advnetlab52 hyperkube[4431]: I0303 20:48:49.939878 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: I0303 20:48:51.174450 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.174863388Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.187292818Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300 NetNS:/var/run/netns/7eb2f764-5>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.187313522Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.238242426Z" level=error msg="Error adding network: [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.238297446Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-multus/network-metrics-daemon-q22tv:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.238318782Z" level=info msg="NetworkStart: stopping network for sandbox 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.238848744Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300 NetNS:/var/run/netns/7eb2f764-5>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.238878903Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:51Z [verbose] Del: openshift-multus:network-metrics-daemon-q22tv:c9b280e5-3bac-411b-8df7-ff7abc1ce2e7:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.275933266Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/ru>
Mar 03 20:48:51 worker-advnetlab52 systemd[1]: run-utsns-7eb2f764\x2d5b18\x2d487c\x2db03c\x2d46bfd05d61ec.mount: Succeeded.
Mar 03 20:48:51 worker-advnetlab52 systemd[1]: run-ipcns-7eb2f764\x2d5b18\x2d487c\x2db03c\x2d46bfd05d61ec.mount: Succeeded.
Mar 03 20:48:51 worker-advnetlab52 systemd[1]: run-netns-7eb2f764\x2d5b18\x2d487c\x2db03c\x2d46bfd05d61ec.mount: Succeeded.
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.327573521Z" level=info msg="RunSandbox: deleting pod ID 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300 from idIndex" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.327602590Z" level=info msg="RunSandbox: removing pod sandbox 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.327627340Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.327647576Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:51 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300-userdata-shm.mount: Succeeded.
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.342775513Z" level=info msg="RunSandbox: removing pod sandbox from storage: 401b27cb2b4a5d3aafecc30bdbf0e30b29de101f789e7e202c7e43b3f902f300" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.345604123Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/>
Mar 03 20:48:51 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:51.345624477Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3bac-411b-8df7-ff7abc1ce2e7_0" id=c1da210f-6efa-4ae8-94df-7f182edea1b8 name=/ru>
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: E0303 20:48:51.345796 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-q22tv_openshift-multus_c9b280e5-3b>
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: E0303 20:48:51.345844 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: E0303 20:48:51.345857 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:51 worker-advnetlab52 hyperkube[4431]: E0303 20:48:51.345896 4431 pod_workers.go:191] Error syncing pod c9b280e5-3bac-411b-8df7-ff7abc1ce2e7 ("network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)"), skipping: failed to "CreatePodSan>
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: I0303 20:48:52.174029 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.174450896Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.185847929Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1 NetNS:/var/run/netns>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.185870129Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.234507258Z" level=error msg="Error adding network: [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with st>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.234563580Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-network-diagnostics/network-check-target-xlg7p:ovn-kubernetes]: error adding container to network \"ovn->
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.234592056Z" level=info msg="NetworkStart: stopping network for sandbox 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.235359860Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1 NetNS:/var/run/netns>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.235395597Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:52Z [verbose] Del: openshift-network-diagnostics:network-check-target-xlg7p:0cb4f70f-89c5-49d6-b2fa-9fc75e724935:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.272697579Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/ru>
Mar 03 20:48:52 worker-advnetlab52 systemd[1]: run-utsns-9383536b\x2dca2f\x2d4aaa\x2daf9d\x2d5360b18f8e59.mount: Succeeded.
Mar 03 20:48:52 worker-advnetlab52 systemd[1]: run-ipcns-9383536b\x2dca2f\x2d4aaa\x2daf9d\x2d5360b18f8e59.mount: Succeeded.
Mar 03 20:48:52 worker-advnetlab52 systemd[1]: run-netns-9383536b\x2dca2f\x2d4aaa\x2daf9d\x2d5360b18f8e59.mount: Succeeded.
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.322609244Z" level=info msg="RunSandbox: deleting pod ID 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1 from idIndex" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.322669863Z" level=info msg="RunSandbox: removing pod sandbox 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.322706629Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.322740256Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:52 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1-userdata-shm.mount: Succeeded.
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.333707989Z" level=info msg="RunSandbox: removing pod sandbox from storage: 04d2f3ff1e0b8e5849133fc2eeeba8d17e896fe024e9c13776850297703b78e1" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a0c name=/runtime.v1alpha2.Ru>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.336756014Z" level=info msg="RunSandbox: releasing container name: k8s_POD_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa586>
Mar 03 20:48:52 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:52.336780714Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_network-check-target-xlg7p_openshift-network-diagnostics_0cb4f70f-89c5-49d6-b2fa-9fc75e724935_0" id=ca6e2b8e-4aef-4c4a-bbd7-43ffa5861a>
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: E0303 20:48:52.337074 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-check-target-xlg7p_openshift-network-diagnostics_>
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: E0303 20:48:52.337156 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed >
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: E0303 20:48:52.337187 4431 kuberuntime_manager.go:755] createPodSandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" failed: rpc error: code = Unknown desc = failed>
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:52 worker-advnetlab52 hyperkube[4431]: E0303 20:48:52.337289 4431 pod_workers.go:191] Error syncing pod 0cb4f70f-89c5-49d6-b2fa-9fc75e724935 ("network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)"), skipping: failed to "C>
Mar 03 20:48:57 worker-advnetlab52 hyperkube[4431]: I0303 20:48:57.086578 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/keepalived-worker-advnetlab52" status=Running
Mar 03 20:48:57 worker-advnetlab52 hyperkube[4431]: I0303 20:48:57.086621 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/mdns-publisher-worker-advnetlab52" status=Running
Mar 03 20:48:57 worker-advnetlab52 hyperkube[4431]: I0303 20:48:57.086632 4431 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/coredns-worker-advnetlab52" status=Running
Mar 03 20:48:57 worker-advnetlab52 hyperkube[4431]: I0303 20:48:57.315319 4431 container_manager_linux.go:986] Found 790 PIDs in root, 790 of them are not to be moved
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: I0303 20:48:58.174256 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.174578772Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.186237227Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f NetNS:/var/run/netns/d84a78af-c341-432e-b7d3->
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.186263460Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.241261590Z" level=error msg="Error adding network: [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-dns>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.241309996Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-dns/dns-default-zz4cg:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.241338271Z" level=info msg="NetworkStart: stopping network for sandbox 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.242013892Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f NetNS:/var/run/netns/d84a78af-c341-432e-b7d3->
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.242084554Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:58Z [verbose] Del: openshift-dns:dns-default-zz4cg:2894274c-9163-477e-b5ae-f39051b38f6b:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLe>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.285063863Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/ru>
Mar 03 20:48:58 worker-advnetlab52 systemd[1]: run-utsns-d84a78af\x2dc341\x2d432e\x2db7d3\x2d9cb7e557dab2.mount: Succeeded.
Mar 03 20:48:58 worker-advnetlab52 systemd[1]: run-ipcns-d84a78af\x2dc341\x2d432e\x2db7d3\x2d9cb7e557dab2.mount: Succeeded.
Mar 03 20:48:58 worker-advnetlab52 systemd[1]: run-netns-d84a78af\x2dc341\x2d432e\x2db7d3\x2d9cb7e557dab2.mount: Succeeded.
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.322611245Z" level=info msg="RunSandbox: deleting pod ID 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f from idIndex" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.322670577Z" level=info msg="RunSandbox: removing pod sandbox 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.322713625Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.322746364Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:58 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f-userdata-shm.mount: Succeeded.
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.332823273Z" level=info msg="RunSandbox: removing pod sandbox from storage: 46ba529e7482181c8f4fabdd9f8263d80847c7418e0258a5088a2727c308488f" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2.Ru>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.336027139Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alph>
Mar 03 20:48:58 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:58.336047908Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=a200b08f-d578-42ce-9722-3317d1168c78 name=/runtime.v1alpha2>
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: E0303 20:48:58.336360 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: E0303 20:48:58.336460 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: E0303 20:48:58.336493 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:58 worker-advnetlab52 hyperkube[4431]: E0303 20:48:58.336575 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: I0303 20:48:59.174242 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.174599900Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.186338849Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396 NetNS:/var/run/netns/3ace263f-a24a-4ae0>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.186521507Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: I0303 20:48:59.207291 4431 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.237952882Z" level=error msg="Error adding network: [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[default>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.237993169Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [default/web-client-pod-smartnic-f4nwn:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI r>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.238020835Z" level=info msg="NetworkStart: stopping network for sandbox 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.238904986Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396 NetNS:/var/run/netns/3ace263f-a24a-4ae0>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.238945993Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: 2021-03-03T20:48:59Z [verbose] Del: default:web-client-pod-smartnic-f4nwn:cc954f31-a832-444d-b95a-e6849bfa4ed5:ovn-kubernetes:eth0 {"cniVersion":"0.3.1","deviceID":"0000:3b:01.5","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.276769101Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/ru>
Mar 03 20:48:59 worker-advnetlab52 systemd[1]: run-utsns-3ace263f\x2da24a\x2d4ae0\x2d8e07\x2d0bc3ea1af7cf.mount: Succeeded.
Mar 03 20:48:59 worker-advnetlab52 systemd[1]: run-ipcns-3ace263f\x2da24a\x2d4ae0\x2d8e07\x2d0bc3ea1af7cf.mount: Succeeded.
Mar 03 20:48:59 worker-advnetlab52 systemd[1]: run-netns-3ace263f\x2da24a\x2d4ae0\x2d8e07\x2d0bc3ea1af7cf.mount: Succeeded.
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.319577022Z" level=info msg="RunSandbox: deleting pod ID 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396 from idIndex" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.319615613Z" level=info msg="RunSandbox: removing pod sandbox 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.319641403Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.319657809Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1alpha2.Runtim>
Mar 03 20:48:59 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396-userdata-shm.mount: Succeeded.
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.335823452Z" level=info msg="RunSandbox: removing pod sandbox from storage: 77ab5247de14627d3705caa95c59a08f58a8e1e02e8ab0af0aaf483092292396" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1alpha2.Ru>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.338764440Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.>
Mar 03 20:48:59 worker-advnetlab52 crio[4383]: time="2021-03-03 20:48:59.338785128Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=8cfecaa9-e58c-4c54-8bf3-e8e2d1281eaf name=/runtime.v1>
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: E0303 20:48:59.338997 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: E0303 20:48:59.339044 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: E0303 20:48:59.339057 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: E0303 20:48:59.339111 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:48:59 worker-advnetlab52 hyperkube[4431]: I0303 20:48:59.952077 4431 setters.go:86] Using node IP: "192.168.111.29"
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: I0303 20:49:01.174371 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.174651406Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.188007274Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d NetNS:/var/run/netns/f6744017-6>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.188034227Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.238686049Z" level=error msg="Error adding network: [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.238736049Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": [openshift-ingress-canary/ingress-canary-s6rz5:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.238795397Z" level=info msg="NetworkStart: stopping network for sandbox e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.239478529Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d NetNS:/var/run/netns/f6744017-6>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.239521895Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: 2021-03-03T20:49:01Z [verbose] Del: openshift-ingress-canary:ingress-canary-s6rz5:bffd580f-35a2-43e9-bdbb-779bd5394c5a:ovn-kubernetes:eth0 {"cniVersion":"0.4.0","dns":{},"ipam":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overl>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.277723750Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/ru>
Mar 03 20:49:01 worker-advnetlab52 systemd[1]: run-utsns-f6744017\x2d6591\x2d4779\x2daa2a\x2dfc319f01d30a.mount: Succeeded.
Mar 03 20:49:01 worker-advnetlab52 systemd[1]: run-ipcns-f6744017\x2d6591\x2d4779\x2daa2a\x2dfc319f01d30a.mount: Succeeded.
Mar 03 20:49:01 worker-advnetlab52 systemd[1]: run-netns-f6744017\x2d6591\x2d4779\x2daa2a\x2dfc319f01d30a.mount: Succeeded.
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.311614897Z" level=info msg="RunSandbox: deleting pod ID e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d from idIndex" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.311671968Z" level=info msg="RunSandbox: removing pod sandbox e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.311719991Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.311757074Z" level=info msg="RunSandbox: unmounting shmPath for sandbox e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:01 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d-userdata-shm.mount: Succeeded.
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.327820016Z" level=info msg="RunSandbox: removing pod sandbox from storage: e5913b1d41bcb9430ed45724b435cb7488a3039571a409a1bd4a2b146e9f0a6d" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/runtime.v1alpha2.Ru>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.330628446Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/>
Mar 03 20:49:01 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:01.330650298Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=18efcaf1-eb54-4ed1-acfc-7cc914c08a1a name=/ru>
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: E0303 20:49:01.330904 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: E0303 20:49:01.330958 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: E0303 20:49:01.330972 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: '
Mar 03 20:49:01 worker-advnetlab52 hyperkube[4431]: E0303 20:49:01.331015 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <warn> [1614804542.3197] dhcp4 (eno1): request timed out
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3199] dhcp4 (eno1): state changed unknown -> timeout
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <warn> [1614804542.3199] dhcp4 (ens1f1): request timed out
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3199] dhcp4 (ens1f1): state changed unknown -> timeout
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <warn> [1614804542.3199] dhcp4 (ens3f0): request timed out
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3199] dhcp4 (ens3f0): state changed unknown -> timeout
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3200] device (eno1): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <warn> [1614804542.3207] device (eno1): Activation: failed for connection 'Wired connection 1'
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3207] device (ens1f1): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <warn> [1614804542.3213] device (ens1f1): Activation: failed for connection 'Wired connection 5'
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3213] device (ens3f0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <warn> [1614804542.3219] device (ens3f0): Activation: failed for connection 'Wired connection 6'
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3221] device (eno1): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00118|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3236] dhcp4 (eno1): canceled DHCP transaction
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3237] dhcp4 (eno1): state changed timeout -> done
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3245] device (ens1f1): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00119|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3451] dhcp4 (ens1f1): canceled DHCP transaction
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3452] dhcp4 (ens1f1): state changed timeout -> done
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3462] device (ens3f0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00120|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3976] dhcp4 (ens3f0): canceled DHCP transaction
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3976] dhcp4 (ens3f0): state changed timeout -> done
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3989] policy: auto-activating connection 'Wired connection 1' (c6e9aa15-94ad-3967-9283-3a0c402f495a)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3993] policy: auto-activating connection 'Wired connection 5' (c5ac7bab-9767-3a07-96e6-e5fdf6b54c90)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.3994] policy: auto-activating connection 'Wired connection 6' (eda6fa6e-7dcf-34d4-a9cf-d2e490bf56f9)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4000] device (eno1): Activation: starting connection 'Wired connection 1' (c6e9aa15-94ad-3967-9283-3a0c402f495a)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4002] device (ens1f1): Activation: starting connection 'Wired connection 5' (c5ac7bab-9767-3a07-96e6-e5fdf6b54c90)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4004] device (ens3f0): Activation: starting connection 'Wired connection 6' (eda6fa6e-7dcf-34d4-a9cf-d2e490bf56f9)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4005] device (eno1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4009] device (eno1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4016] device (ens1f1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4020] device (ens1f1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4023] device (ens3f0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4026] device (ens3f0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4030] device (eno1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4032] dhcp4 (eno1): activation: beginning transaction (timeout in 45 seconds)
Mar 03 20:49:02 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00121|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4053] device (ens1f1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4056] dhcp4 (ens1f1): activation: beginning transaction (timeout in 45 seconds)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4065] device (ens3f0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Mar 03 20:49:02 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00122|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:02 worker-advnetlab52 NetworkManager[2521]: <info> [1614804542.4068] dhcp4 (ens3f0): activation: beginning transaction (timeout in 45 seconds)
Mar 03 20:49:02 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00123|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:02 worker-advnetlab52 kernel: mlx5_core 0000:3b:00.0: Port module event: module 0, Cable unplugged
Mar 03 20:49:02 worker-advnetlab52 kernel: mlx5_core 0000:3b:00.1: mlx5_pcie_event:296:(pid 44359): PCIe slot advertised sufficient power (75W).
Mar 03 20:49:02 worker-advnetlab52 kernel: mlx5_core 0000:3b:00.0: mlx5_pcie_event:296:(pid 8): PCIe slot advertised sufficient power (75W).
Mar 03 20:49:06 worker-advnetlab52 hyperkube[4431]: I0303 20:49:06.174519 4431 kuberuntime_manager.go:439] No sandbox for pod "network-check-target-xlg7p_openshift-network-diagnostics(0cb4f70f-89c5-49d6-b2fa-9fc75e724935)" can be found. Need to start a new one
Mar 03 20:49:06 worker-advnetlab52 hyperkube[4431]: I0303 20:49:06.174569 4431 kuberuntime_manager.go:439] No sandbox for pod "network-metrics-daemon-q22tv_openshift-multus(c9b280e5-3bac-411b-8df7-ff7abc1ce2e7)" can be found. Need to start a new one
Mar 03 20:49:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:06.174942465Z" level=info msg="Running pod sandbox: openshift-multus/network-metrics-daemon-q22tv/POD" id=6f46c315-abd4-42fe-9e59-5b500b55fba2 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:49:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:06.174987429Z" level=info msg="Running pod sandbox: openshift-network-diagnostics/network-check-target-xlg7p/POD" id=2f7c50b2-41e5-4136-959a-2c54044c86bb name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:49:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:06.192671021Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:0d3611892691f17622e0402d12cd98973696470ad444a85d5149b6a45deb23a6 NetNS:/var/run/netns>
Mar 03 20:49:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:06.192696049Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:49:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:06.193804101Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:ee699f21dc87a8cde8c0c8e87238684e44d3dbefe481f4e3740f6b29d2b0957a NetNS:/var/run/netns/4a8cbad1-8>
Mar 03 20:49:06 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:06.193830001Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:49:13 worker-advnetlab52 hyperkube[4431]: I0303 20:49:13.175791 4431 kuberuntime_manager.go:439] No sandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" can be found. Need to start a new one
Mar 03 20:49:13 worker-advnetlab52 hyperkube[4431]: I0303 20:49:13.175915 4431 kuberuntime_manager.go:439] No sandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" can be found. Need to start a new one
Mar 03 20:49:13 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:13.176691125Z" level=info msg="Running pod sandbox: openshift-dns/dns-default-zz4cg/POD" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:49:13 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:13.176750273Z" level=info msg="Running pod sandbox: default/web-client-pod-smartnic-f4nwn/POD" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:49:13 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:13.193721569Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5 NetNS:/var/run/netns/50aed01b-4080-41ed-96aa->
Mar 03 20:49:13 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:13.193747759Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:49:13 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:13.194407889Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c NetNS:/var/run/netns/5741cb58-8f9e-4607>
Mar 03 20:49:13 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:13.194441054Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:49:14 worker-advnetlab52 hyperkube[4431]: I0303 20:49:14.174441 4431 kuberuntime_manager.go:439] No sandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" can be found. Need to start a new one
Mar 03 20:49:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:14.174783460Z" level=info msg="Running pod sandbox: openshift-ingress-canary/ingress-canary-s6rz5/POD" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Mar 03 20:49:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:14.185539884Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880 NetNS:/var/run/netns/76277732-5>
Mar 03 20:49:14 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:14.185564622Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Mar 03 20:49:18 worker-advnetlab52 kernel: ixgbe 0000:19:00.0 eno1: NIC Link is Down
Mar 03 20:49:18 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00124|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:19 worker-advnetlab52 hyperkube[4431]: E0303 20:49:19.957236 4431 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "worker-advnetlab52": Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes/worker-advnetlab52?resour>
Mar 03 20:49:21 worker-advnetlab52 hyperkube[4431]: E0303 20:49:21.958198 4431 controller.go:187] failed to update lease, error: Put "https://api-int.sriov.dev.metalkube.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-advnetlab52?timeout=10s": ne>
Mar 03 20:49:23 worker-advnetlab52 kernel: mlx5_core 0000:d8:00.0: Port module event: module 0, Cable unplugged
Mar 03 20:49:23 worker-advnetlab52 kernel: mlx5_core 0000:d8:00.1: mlx5_pcie_event:296:(pid 8): PCIe slot advertised sufficient power (75W).
Mar 03 20:49:23 worker-advnetlab52 kernel: mlx5_core 0000:d8:00.0: mlx5_pcie_event:296:(pid 28309): PCIe slot advertised sufficient power (75W).
Mar 03 20:49:23 worker-advnetlab52 kernel: mlx5_core 0000:d8:00.0 ens3f0: Link down
Mar 03 20:49:23 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00125|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:24 worker-advnetlab52 NetworkManager[2521]: <info> [1614804564.2606] device (eno1): state change: ip-config -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Mar 03 20:49:24 worker-advnetlab52 NetworkManager[2521]: <info> [1614804564.2665] dhcp4 (eno1): canceled DHCP transaction
Mar 03 20:49:24 worker-advnetlab52 NetworkManager[2521]: <info> [1614804564.2666] dhcp4 (eno1): state changed unknown -> done
Mar 03 20:49:26 worker-advnetlab52 kernel: igb 0000:01:00.0 eno3: igb: eno3 NIC Link is Down
Mar 03 20:49:26 worker-advnetlab52 ovs-vswitchd[2372]: ovs|00126|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Mar 03 20:49:29 worker-advnetlab52 NetworkManager[2521]: <info> [1614804569.4569] device (ens3f0): state change: ip-config -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Mar 03 20:49:29 worker-advnetlab52 NetworkManager[2521]: <info> [1614804569.4645] dhcp4 (ens3f0): canceled DHCP transaction
Mar 03 20:49:29 worker-advnetlab52 NetworkManager[2521]: <info> [1614804569.4645] dhcp4 (ens3f0): state changed unknown -> done
Mar 03 20:49:29 worker-advnetlab52 hyperkube[4431]: E0303 20:49:29.957582 4431 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "worker-advnetlab52": Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes/worker-advnetlab52?timeou>
Mar 03 20:49:29 worker-advnetlab52 hyperkube[4431]: E0303 20:49:29.957763 4431 controller.go:187] failed to update lease, error: Put "https://api-int.sriov.dev.metalkube.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-advnetlab52?timeout=10s": re>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.946050 4431 reflector.go:138] object-"openshift-network-diagnostics"/"default-token-5bq72": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-network-dia>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.946738 4431 reflector.go:138] object-"default"/"sriovdp-config": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/configmaps?allowWatchBookmarks=true&f>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.946814 4431 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/services?allowWatchBookmarks=true&resourceVersion=>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.946792 4431 reflector.go:138] object-"openshift-image-registry"/"node-ca-token-g9fhq": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-image-registry/s>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.946865 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-node-metrics-cert": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ovn-kubernetes>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947016 4431 reflector.go:138] object-"openshift-dns"/"dns-dockercfg-mmlx8": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-dns/secrets?allowWatchBookm>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.946820 4431 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/kubelet.go:438: Failed to watch *v1.Node: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes?allowWatchBookmarks=true&fieldSelector=metad>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947084 4431 reflector.go:138] object-"openshift-dns"/"dns-default-metrics-tls": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-dns/secrets?allowWatchB>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947158 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-ca": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ovn-kubernetes/configmaps?>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947185 4431 reflector.go:138] object-"openshift-cluster-node-tuning-operator"/"tuned-token-7tgc8": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-clus>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947177 4431 reflector.go:138] object-"openshift-machine-config-operator"/"machine-config-daemon-dockercfg-hlplg": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947156 4431 controller.go:187] failed to update lease, error: Put "https://api-int.sriov.dev.metalkube.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-advnetlab52?timeout=10s": di>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947213 4431 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: Get "https://api-int.sriov.dev.metalkube.org:6443/apis/node.k8s.io/v1/runtimeclasses?allowWatchBookmar>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947235 4431 reflector.go:138] object-"openshift-dns"/"dns-token-dn7l4": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-dns/secrets?allowWatchBookmarks>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947246 4431 reflector.go:138] object-"openshift-monitoring"/"node-exporter-tls": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-monitoring/secrets?all>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947255 4431 reflector.go:138] object-"openshift-cluster-node-tuning-operator"/"tuned-dockercfg-2jcpr": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift->
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947264 4431 reflector.go:138] object-"openshift-ingress-canary"/"default-dockercfg-mvmnw": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ingress-cana>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947257 4431 reflector.go:138] object-"openshift-image-registry"/"image-registry-certificates": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-image>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947293 4431 reflector.go:138] object-"openshift-multus"/"metrics-daemon-sa-token-vsgzh": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-multus/secrets>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947300 4431 reflector.go:138] object-"openshift-multus"/"multus-dockercfg-hbglk": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-multus/secrets?allowW>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947291 4431 reflector.go:138] object-"openshift-multus"/"metrics-daemon-sa-dockercfg-nd4n6": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-multus/sec>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947319 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovnkube-config": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ovn-kubernetes/con>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947339 4431 reflector.go:138] object-"openshift-multus"/"metrics-daemon-secret": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-multus/secrets?allowWa>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947193 4431 reflector.go:138] object-"openshift-machine-config-operator"/"machine-config-daemon-token-6rj4m": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/ope>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947340 4431 reflector.go:138] object-"openshift-cluster-node-tuning-operator"/"tuned-profiles": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-clus>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947304 4431 reflector.go:138] object-"default"/"default-token-frx6z": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/secrets?allowWatchBookmarks=true&fi>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947362 4431 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "worker-advnetlab52": Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes/worker-advnetlab52?timeou>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947368 4431 reflector.go:138] object-"openshift-machine-config-operator"/"cookie-secret": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-machine-confi>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947385 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"env-overrides": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ovn-kubernetes/conf>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947361 4431 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to watch *v1.Pod: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/pods?allowWatchBookmarks=true&fieldSelector>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947384 4431 reflector.go:138] object-"openshift-machine-config-operator"/"proxy-tls": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-machine-config-op>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947335 4431 reflector.go:138] object-"default"/"sriov-device-plugin-token-bk58k": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/secrets?allowWatchBookm>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947412 4431 reflector.go:138] object-"openshift-dns"/"dns-default": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-dns/configmaps?allowWatchBookmar>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947171 4431 reflector.go:138] object-"default"/"default-dockercfg-dd52l": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/secrets?allowWatchBookmarks=tru>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947459 4431 reflector.go:138] object-"openshift-multus"/"multus-token-2zlrk": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-multus/secrets?allowWatch>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947493 4431 reflector.go:138] object-"openshift-image-registry"/"node-ca-dockercfg-q8dhp": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-image-regist>
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947586 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-kubernetes-node-token-z6ts7": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ovn->
Mar 03 20:49:30 worker-advnetlab52 hyperkube[4431]: E0303 20:49:30.947633 4431 reflector.go:138] object-"openshift-monitoring"/"node-exporter-token-vbbz8": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-monitoring/sec>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.947836143Z" level=error msg="Error adding network: Multus: [default/web-client-pod-smartnic-f4nwn]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/default/pods/web>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.947904233Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": Multus: [default/web-client-pod-smartnic-f4nwn]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.947942582Z" level=info msg="NetworkStart: stopping network for sandbox f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.948003462Z" level=error msg="Error adding network: Multus: [openshift-dns/dns-default-zz4cg]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/openshift-dns/pods/dns>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.948038811Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": Multus: [openshift-dns/dns-default-zz4cg]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.948070325Z" level=info msg="NetworkStart: stopping network for sandbox 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.948082186Z" level=error msg="Error adding network: Multus: [openshift-ingress-canary/ingress-canary-s6rz5]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/openshif>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.948121471Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": Multus: [openshift-ingress-canary/ingress-canary-s6rz5]: error getting pod: Get \"https://[api-int.sriov.dev.metalk>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.948162647Z" level=info msg="NetworkStart: stopping network for sandbox 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.949183153Z" level=info msg="Got pod network &{Name:web-client-pod-smartnic-f4nwn Namespace:default ID:f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c NetNS:/var/run/netns/5741cb58-8f9e-4607>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.949221292Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.949427239Z" level=info msg="Got pod network &{Name:dns-default-zz4cg Namespace:openshift-dns ID:30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5 NetNS:/var/run/netns/50aed01b-4080-41ed-96aa->
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.949503413Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.949581951Z" level=info msg="Got pod network &{Name:ingress-canary-s6rz5 Namespace:openshift-ingress-canary ID:6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880 NetNS:/var/run/netns/76277732-5>
Mar 03 20:49:30 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:30.949619367Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:49:32 worker-advnetlab52 NetworkManager[2521]: <info> [1614804572.5932] device (eno3): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Mar 03 20:49:32 worker-advnetlab52 NetworkManager[2521]: <info> [1614804572.6025] dhcp4 (eno3): canceled DHCP transaction
Mar 03 20:49:32 worker-advnetlab52 NetworkManager[2521]: <info> [1614804572.6025] dhcp4 (eno3): state changed bound -> done
Mar 03 20:49:32 worker-advnetlab52 dbus-daemon[2200]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.6' (uid=0 pid=2521 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:>
Mar 03 20:49:32 worker-advnetlab52 systemd[1]: Starting Network Manager Script Dispatcher Service...
Mar 03 20:49:32 worker-advnetlab52 dbus-daemon[2200]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Mar 03 20:49:32 worker-advnetlab52 systemd[1]: Started Network Manager Script Dispatcher Service.
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + [[ OVNKubernetes == \O\V\N\K\u\b\e\r\n\e\t\e\s ]]
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + [[ Wired connection 3 == \W\i\r\e\d\ \C\o\n\n\e\c\t\i\o\n ]]
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + '[' -z ']'
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + echo 'Not a DHCP4 address. Ignoring.'
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: Not a DHCP4 address. Ignoring.
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + exit 0
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + '[' -z ']'
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + echo 'Not a DHCP6 address. Ignoring.'
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: Not a DHCP6 address. Ignoring.
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: + exit 0
Mar 03 20:49:32 worker-advnetlab52 root[64196]: NM mdns-hostname triggered by down.
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: <13>Mar 3 20:49:32 root: NM mdns-hostname triggered by down.
Mar 03 20:49:32 worker-advnetlab52 root[64200]: Hostname changed: worker-advnetlab52
Mar 03 20:49:32 worker-advnetlab52 nm-dispatcher[64155]: <13>Mar 3 20:49:32 root: Hostname changed: worker-advnetlab52
Mar 03 20:49:32 worker-advnetlab52 systemd[1]: Starting Generate console-login-helper-messages issue snippet...
Mar 03 20:49:33 worker-advnetlab52 systemd[1]: console-login-helper-messages-issuegen.service: Succeeded.
Mar 03 20:49:33 worker-advnetlab52 systemd[1]: Started Generate console-login-helper-messages issue snippet.
Mar 03 20:49:33 worker-advnetlab52 systemd[1]: console-login-helper-messages-issuegen.service: Consumed 13ms CPU time
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018271 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-kubernetes-node-dockercfg-72bkh": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift->
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018274 4431 reflector.go:138] object-"openshift-monitoring"/"node-exporter-dockercfg-fn99l": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-monitoring>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018275 4431 reflector.go:138] object-"default"/"default-dockercfg-dd52l": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/secr>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018274 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-cert": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ovn-kubernetes/secrets?allo>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018302 4431 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "worker-advnetlab52": Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes/worker-advnetlab52?timeou>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018331 4431 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: Get "https://api-int.sriov.dev.metalkube.org:6443/apis/storage.k8s.io/v1/csidrivers?allowWatchBookmarks=t>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018334 4431 reflector.go:138] object-"openshift-multus"/"cni-binary-copy-script": Failed to watch *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-multus/configmaps?>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018346 4431 reflector.go:138] object-"openshift-network-diagnostics"/"default-dockercfg-sjw89": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-network>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018331 4431 controller.go:187] failed to update lease, error: Put "https://api-int.sriov.dev.metalkube.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-advnetlab52?timeout=10s": di>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018361 4431 reflector.go:138] object-"openshift-ingress-canary"/"default-token-wtkmq": Failed to watch *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-ingress-canary/s>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018472 4431 reflector.go:138] object-"openshift-multus"/"metrics-daemon-sa-dockercfg-nd4n6": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/name>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018489 4431 reflector.go:138] object-"default"/"default-token-frx6z": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/secrets?>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018524 4431 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/kubelet.go:438: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes?fieldSelector=metad>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018531 4431 reflector.go:138] object-"openshift-image-registry"/"node-ca-dockercfg-q8dhp": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namesp>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018565 4431 reflector.go:138] object-"openshift-cluster-node-tuning-operator"/"tuned-dockercfg-2jcpr": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/a>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018573 4431 reflector.go:138] object-"openshift-dns"/"dns-token-dn7l4": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-dns/>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018588 4431 reflector.go:138] object-"default"/"sriov-device-plugin-token-bk58k": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/defa>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018587 4431 reflector.go:138] object-"openshift-dns"/"dns-dockercfg-mmlx8": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift->
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018599 4431 reflector.go:138] object-"openshift-multus"/"metrics-daemon-secret": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/opens>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018605 4431 reflector.go:138] object-"openshift-image-registry"/"node-ca-token-g9fhq": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018617 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-node-metrics-cert": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespac>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018631 4431 reflector.go:138] object-"openshift-machine-config-operator"/"cookie-secret": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespa>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018604 4431 reflector.go:138] object-"openshift-multus"/"multus-token-2zlrk": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshif>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018644 4431 reflector.go:138] object-"openshift-monitoring"/"node-exporter-tls": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/opens>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018606 4431 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/pods?fieldSelector=>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018604 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovn-kubernetes-node-token-z6ts7": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018657 4431 reflector.go:138] object-"openshift-cluster-node-tuning-operator"/"tuned-token-7tgc8": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018671 4431 reflector.go:138] object-"openshift-machine-config-operator"/"machine-config-daemon-token-6rj4m": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018605 4431 reflector.go:138] object-"openshift-network-diagnostics"/"default-token-5bq72": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/names>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018688 4431 reflector.go:138] object-"openshift-ovn-kubernetes"/"ovnkube-config": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespace>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018693 4431 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/services?resourceVersi>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018703 4431 reflector.go:138] object-"openshift-cluster-node-tuning-operator"/"tuned-profiles": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/ap>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018720 4431 reflector.go:138] object-"openshift-machine-config-operator"/"proxy-tls": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018729 4431 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://api-int.sriov.dev.metalkube.org:6443/apis/node.k8s.io/v1>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018624 4431 reflector.go:138] object-"openshift-machine-config-operator"/"machine-config-daemon-dockercfg-hlplg": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018646 4431 reflector.go:138] object-"openshift-ingress-canary"/"default-dockercfg-mvmnw": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namesp>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018643 4431 reflector.go:138] object-"openshift-multus"/"metrics-daemon-sa-token-vsgzh": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespac>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018648 4431 reflector.go:138] object-"openshift-dns"/"dns-default": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/openshift-dn>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018664 4431 reflector.go:138] object-"openshift-dns"/"dns-default-metrics-tls": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/opensh>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018690 4431 reflector.go:138] object-"openshift-multus"/"multus-dockercfg-hbglk": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/open>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018721 4431 reflector.go:138] object-"openshift-image-registry"/"image-registry-certificates": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018729 4431 reflector.go:138] object-"default"/"sriovdp-config": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespaces/default/configm>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.018810 4431 reflector.go:138] object-"openshift-monitoring"/"node-exporter-token-vbbz8": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/namespac>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020131597Z" level=error msg="Error deleting network: Multus: [openshift-dns/dns-default-zz4cg]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/openshift-dns/pods/d>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020301871Z" level=error msg="Error while removing pod from CNI network \"multus-cni-network\": Multus: [openshift-dns/dns-default-zz4cg]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020355207Z" level=error msg="Error stopping network on cleanup: failed to destroy network for pod sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0(30d77ae21f324d264c4fa67b3>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020385503Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/ru>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020228324Z" level=error msg="Error deleting network: Multus: [openshift-ingress-canary/ingress-canary-s6rz5]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/opensh>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020486885Z" level=error msg="Error while removing pod from CNI network \"multus-cni-network\": Multus: [openshift-ingress-canary/ingress-canary-s6rz5]: error getting pod: Get \"https://[api-int.sriov.dev.me>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020547102Z" level=error msg="Error stopping network on cleanup: failed to destroy network for pod sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0(6567f8d6ad7>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020573028Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/ru>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020741234Z" level=error msg="Error deleting network: Multus: [default/web-client-pod-smartnic-f4nwn]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/default/pods/w>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020766393Z" level=error msg="Error while removing pod from CNI network \"multus-cni-network\": Multus: [default/web-client-pod-smartnic-f4nwn]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020807505Z" level=error msg="Error stopping network on cleanup: failed to destroy network for pod sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0(f1ed3329424f77eb2db>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.020834652Z" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/ru>
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-utsns-76277732\x2d5ec9\x2d42b7\x2d81fa\x2d3e109953b904.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-utsns-76277732\x2d5ec9\x2d42b7\x2d81fa\x2d3e109953b904.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-utsns-5741cb58\x2d8f9e\x2d4607\x2da8e5\x2d97ff23685330.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-utsns-5741cb58\x2d8f9e\x2d4607\x2da8e5\x2d97ff23685330.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-utsns-50aed01b\x2d4080\x2d41ed\x2d96aa\x2d66a030e97704.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-utsns-50aed01b\x2d4080\x2d41ed\x2d96aa\x2d66a030e97704.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-ipcns-50aed01b\x2d4080\x2d41ed\x2d96aa\x2d66a030e97704.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-ipcns-50aed01b\x2d4080\x2d41ed\x2d96aa\x2d66a030e97704.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-ipcns-5741cb58\x2d8f9e\x2d4607\x2da8e5\x2d97ff23685330.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-ipcns-5741cb58\x2d8f9e\x2d4607\x2da8e5\x2d97ff23685330.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-ipcns-76277732\x2d5ec9\x2d42b7\x2d81fa\x2d3e109953b904.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-ipcns-76277732\x2d5ec9\x2d42b7\x2d81fa\x2d3e109953b904.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-netns-50aed01b\x2d4080\x2d41ed\x2d96aa\x2d66a030e97704.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-netns-50aed01b\x2d4080\x2d41ed\x2d96aa\x2d66a030e97704.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-netns-5741cb58\x2d8f9e\x2d4607\x2da8e5\x2d97ff23685330.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-netns-5741cb58\x2d8f9e\x2d4607\x2da8e5\x2d97ff23685330.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.069629087Z" level=info msg="RunSandbox: deleting pod ID f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c from idIndex" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.069663097Z" level=info msg="RunSandbox: removing pod sandbox f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.069678254Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.069692878Z" level=info msg="RunSandbox: unmounting shmPath for sandbox f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.070587444Z" level=info msg="RunSandbox: deleting pod ID 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5 from idIndex" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.070626059Z" level=info msg="RunSandbox: removing pod sandbox 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.070644485Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.070658390Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c-userdata-shm.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c-userdata-shm.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-netns-76277732\x2d5ec9\x2d42b7\x2d81fa\x2d3e109953b904.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-netns-76277732\x2d5ec9\x2d42b7\x2d81fa\x2d3e109953b904.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5-userdata-shm.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5-userdata-shm.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.082566459Z" level=info msg="RunSandbox: deleting pod ID 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880 from idIndex" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime.v1alpha2.RuntimeS>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.082599638Z" level=info msg="RunSandbox: removing pod sandbox 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime.v1alpha2.RuntimeService/R>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.082617736Z" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.082633401Z" level=info msg="RunSandbox: unmounting shmPath for sandbox 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880-userdata-shm.mount: Succeeded.
Mar 03 20:49:34 worker-advnetlab52 systemd[1]: run-containers-storage-overlay\x2dcontainers-6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880-userdata-shm.mount: Consumed 0 CPU time
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.085658496Z" level=info msg="RunSandbox: removing pod sandbox from storage: f1ed3329424f77eb2db035e41c7926b24fab4fe45ecae63739c160b275da911c" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1alpha2.Ru>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.085669379Z" level=info msg="RunSandbox: removing pod sandbox from storage: 30d77ae21f324d264c4fa67b33bde91ecec5af25d517f7f163509d23a108c3c5" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2.Ru>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.089160065Z" level=info msg="RunSandbox: releasing container name: k8s_POD_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.089180484Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d-b95a-e6849bfa4ed5_0" id=86e09d12-ffcc-447b-8484-3112597a2cfe name=/runtime.v1>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.089362 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_web-client-pod-smartnic-f4nwn_default_cc954f31-a832-444d->
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.089405 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netwo>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.089418 4431 kuberuntime_manager.go:755] createPodSandbox for pod "web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)" failed: rpc error: code = Unknown desc = failed to create pod netw>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.089455 4431 pod_workers.go:191] Error syncing pod cc954f31-a832-444d-b95a-e6849bfa4ed5 ("web-client-pod-smartnic-f4nwn_default(cc954f31-a832-444d-b95a-e6849bfa4ed5)"), skipping: failed to "CreatePodSandbox" fo>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.092053680Z" level=info msg="RunSandbox: releasing container name: k8s_POD_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alph>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.092079691Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f39051b38f6b_0" id=0f3e46d7-17a7-4a71-9e43-b2b8719214df name=/runtime.v1alpha2>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.092252 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dns-default-zz4cg_openshift-dns_2894274c-9163-477e-b5ae-f>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.092296 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network san>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.092309 4431 kuberuntime_manager.go:755] createPodSandbox for pod "dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)" failed: rpc error: code = Unknown desc = failed to create pod network sa>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.092354 4431 pod_workers.go:191] Error syncing pod 2894274c-9163-477e-b5ae-f39051b38f6b ("dns-default-zz4cg_openshift-dns(2894274c-9163-477e-b5ae-f39051b38f6b)"), skipping: failed to "CreatePodSandbox" for "dns>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.094689257Z" level=info msg="RunSandbox: removing pod sandbox from storage: 6567f8d6ad7f61f539f751d00cdec00effcdb5875e7cd38b7e1c7581c6ac9880" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/runtime.v1alpha2.Ru>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.097648569Z" level=info msg="RunSandbox: releasing container name: k8s_POD_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/>
Mar 03 20:49:34 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:34.097673132Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35a2-43e9-bdbb-779bd5394c5a_0" id=4c36c8a5-c777-480f-9f95-f29b577a827c name=/ru>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.097781 4431 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ingress-canary-s6rz5_openshift-ingress-canary_bffd580f-35>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.097810 4431 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create p>
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.097825 4431 kuberuntime_manager.go:755] createPodSandbox for pod "ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)" failed: rpc error: code = Unknown desc = failed to create >
Mar 03 20:49:34 worker-advnetlab52 hyperkube[4431]: E0303 20:49:34.097865 4431 pod_workers.go:191] Error syncing pod bffd580f-35a2-43e9-bdbb-779bd5394c5a ("ingress-canary-s6rz5_openshift-ingress-canary(bffd580f-35a2-43e9-bdbb-779bd5394c5a)"), skipping: failed to "CreatePodSan>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.203006526Z" level=error msg="Error adding network: Multus: [openshift-network-diagnostics/network-check-target-xlg7p]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespac>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.203223715Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": Multus: [openshift-network-diagnostics/network-check-target-xlg7p]: error getting pod: Get \"https://[api-int.sriov>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.203245951Z" level=info msg="NetworkStart: stopping network for sandbox 0d3611892691f17622e0402d12cd98973696470ad444a85d5149b6a45deb23a6" id=2f7c50b2-41e5-4136-959a-2c54044c86bb name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.203097380Z" level=error msg="Error adding network: Multus: [openshift-multus/network-metrics-daemon-q22tv]: error getting pod: Get \"https://[api-int.sriov.dev.metalkube.org]:6443/api/v1/namespaces/openshif>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.203359167Z" level=error msg="Error while adding pod to CNI network \"multus-cni-network\": Multus: [openshift-multus/network-metrics-daemon-q22tv]: error getting pod: Get \"https://[api-int.sriov.dev.metalk>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.203379745Z" level=info msg="NetworkStart: stopping network for sandbox ee699f21dc87a8cde8c0c8e87238684e44d3dbefe481f4e3740f6b29d2b0957a" id=6f46c315-abd4-42fe-9e59-5b500b55fba2 name=/runtime.v1alpha2.Runtim>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.204359264Z" level=info msg="Got pod network &{Name:network-check-target-xlg7p Namespace:openshift-network-diagnostics ID:0d3611892691f17622e0402d12cd98973696470ad444a85d5149b6a45deb23a6 NetNS:/var/run/netns>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.204402755Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.204555408Z" level=info msg="Got pod network &{Name:network-metrics-daemon-q22tv Namespace:openshift-multus ID:ee699f21dc87a8cde8c0c8e87238684e44d3dbefe481f4e3740f6b29d2b0957a NetNS:/var/run/netns/4a8cbad1-8>
Mar 03 20:49:36 worker-advnetlab52 crio[4383]: time="2021-03-03 20:49:36.204592738Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Mar 03 20:49:37 worker-advnetlab52 hyperkube[4431]: E0303 20:49:37.090048 4431 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "worker-advnetlab52": Get "https://api-int.sriov.dev.metalkube.org:6443/api/v1/nodes/worker-advnetlab52?timeou>
Mar 03 20:49:37 worker-advnetlab52 hyperkube[4431]: E0303 20:49:37.090067 4431 kubelet_node_status.go:457] Unable to update node status: update node status exceeds retry count
Mar 03 20:49:37 worker-advnetlab52 hyperkube[4431]: E0303 20:49:37.090124 4431 controller.go:187] failed to update lease, error: Put "https://api-int.sriov.dev.metalkube.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-advnetlab52?timeout=10s": di>
Mar 03 20:49:37 worker-advnetlab52 hyperkube[4431]: I0303 20:49:37.090199 4431 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
-- Reboot --
Mar 03 20:56:01 localhost kernel: microcode: microcode updated early to revision 0x5003003, date = 2020-06-18
Mar 03 20:56:01 localhost kernel: Linux version 4.18.0-280.el8.bzbftest.v1.dt2.x86_64 ([email protected]) (gcc version 8.4.1 20200928 (Red Hat 8.4.1-1) (GCC)) #1 SMP Thu Feb 4 03:31:01 EST 2021
Mar 03 20:56:01 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-53f2f04e30e2d7a86355859ce318b2d9a7b7112e61b3195a0953b25c1526ec63/vmlinuz-4.18.0-280.el8.bzbftest.v1.dt2.x86_64 random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/rhcos/5>
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
Mar 03 20:56:01 localhost kernel: x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[5]: 960, xstate_sizes[5]: 64
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]: 512
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
Mar 03 20:56:01 localhost kernel: x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]: 8
Mar 03 20:56:01 localhost kernel: x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
Mar 03 20:56:01 localhost kernel: BIOS-provided physical RAM map:
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x0000000000100000-0x000000004fd5afff] usable
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x000000004fd5b000-0x000000004ffdffff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x000000004ffe0000-0x0000000068bfefff] usable
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x0000000068bff000-0x000000006ebfefff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x000000006ebff000-0x000000006f9fefff] ACPI NVS
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x000000006f9ff000-0x000000006fffefff] ACPI data
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x00000000fd000000-0x00000000fe7fffff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x00000000fec80000-0x00000000fed00fff] reserved
Mar 03 20:56:01 localhost kernel: BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment