Generated: 2026-01-23 Last Updated: 2026-01-23
These jobs fail with both of the following error messages:
State Store: Required value: Please set the --state flag or export KOPS_STATE_STORE.Could not retrieve location for AWS bucket
Total: 47 unique CI jobs
| Category | Count |
|---|---|
| Miscellaneous jobs (kops-misc) | 7 |
| Upgrade jobs (kops-upgrades) | 20 |
| Upgrade jobs with many addons (kops-upgrades-many-addons) | 20 |
| CI Job Name | TestGrid URL | Dashboards |
|---|---|---|
| e2e-kops-aws-addon-resource-tracking | https://testgrid.k8s.io/kops-misc#kops-aws-addon-resource-tracking | kops-misc, kops-k8s-stable |
| e2e-kops-aws-aws-load-balancer-controller | https://testgrid.k8s.io/kops-misc#kops-aws-aws-load-balancer-controller | kops-misc, kops-k8s-stable |
| e2e-kops-aws-ipv6-karpenter | https://testgrid.k8s.io/kops-misc#kops-aws-ipv6-karpenter | kops-misc, kops-k8s-stable |
| e2e-kops-aws-karpenter | https://testgrid.k8s.io/kops-misc#kops-aws-karpenter | kops-misc, kops-k8s-stable |
| e2e-kops-aws-keypair-rotation-ha | https://testgrid.k8s.io/kops-misc#kops-aws-keypair-rotation-ha | kops-misc, kops-k8s-stable |
| e2e-kops-aws-metrics-server | https://testgrid.k8s.io/kops-misc#kops-aws-metrics-server | kops-misc, kops-k8s-stable |
| e2e-kops-aws-pod-identity-webhook | https://testgrid.k8s.io/kops-misc#kops-aws-pod-identity-webhook | kops-misc, kops-k8s-stable |
| CI Job Name | TestGrid URL | Dashboards |
|---|---|---|
| e2e-kops-aws-upgrade-k128-ko132-to-k129-kolatest | https://testgrid.k8s.io/kops-upgrades#k128-ko132-to-k129-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k129-ko132-to-k130-kolatest | https://testgrid.k8s.io/kops-upgrades#k129-ko132-to-k130-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k129-ko133-to-k130-kolatest | https://testgrid.k8s.io/kops-upgrades#k129-ko133-to-k130-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k129-kolatest-to-k130-kolatest | https://testgrid.k8s.io/kops-upgrades#k129-kolatest-to-k130-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-ko132-to-k131-kolatest | https://testgrid.k8s.io/kops-upgrades#k130-ko132-to-k131-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-ko133-to-k131-kolatest | https://testgrid.k8s.io/kops-upgrades#k130-ko133-to-k131-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-ko134-to-k131-kolatest | https://testgrid.k8s.io/kops-upgrades#k130-ko134-to-k131-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-kolatest-to-k131-kolatest | https://testgrid.k8s.io/kops-upgrades#k130-kolatest-to-k131-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-ko132-to-k132-kolatest | https://testgrid.k8s.io/kops-upgrades#k131-ko132-to-k132-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-ko133-to-k132-kolatest | https://testgrid.k8s.io/kops-upgrades#k131-ko133-to-k132-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-ko134-to-k132-kolatest | https://testgrid.k8s.io/kops-upgrades#k131-ko134-to-k132-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-kolatest-to-k132-kolatest | https://testgrid.k8s.io/kops-upgrades#k131-kolatest-to-k132-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k132-ko133-to-k133-kolatest | https://testgrid.k8s.io/kops-upgrades#k132-ko133-to-k133-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k132-ko134-to-k133-kolatest | https://testgrid.k8s.io/kops-upgrades#k132-ko134-to-k133-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k132-kolatest-to-k133-kolatest | https://testgrid.k8s.io/kops-upgrades#k132-kolatest-to-k133-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k133-ko134-to-k134-kolatest | https://testgrid.k8s.io/kops-upgrades#k133-ko134-to-k134-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k133-kolatest-to-k134-kolatest | https://testgrid.k8s.io/kops-upgrades#k133-kolatest-to-k134-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-k134-kolatest-to-klatest-kolatest | https://testgrid.k8s.io/kops-upgrades#k134-kolatest-to-klatest-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-kstable-kolatest-to-kci-kolatest | https://testgrid.k8s.io/kops-upgrades#kstable-kolatest-to-kci-kolatest | kops-upgrades, kops-k8s-stable |
| e2e-kops-aws-upgrade-kstable-kolatest-to-klatest-kolatest | https://testgrid.k8s.io/kops-upgrades#kstable-kolatest-to-klatest-kolatest | kops-upgrades, kops-k8s-stable |
| CI Job Name | TestGrid URL | Dashboards |
|---|---|---|
| e2e-kops-aws-upgrade-k128-ko132-to-k129-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k128-ko132-to-k129-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k129-ko132-to-k130-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k129-ko132-to-k130-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k129-ko133-to-k130-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k129-ko133-to-k130-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k129-kolatest-to-k130-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k129-kolatest-to-k130-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-ko132-to-k131-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k130-ko132-to-k131-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-ko133-to-k131-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k130-ko133-to-k131-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-ko134-to-k131-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k130-ko134-to-k131-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k130-kolatest-to-k131-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k130-kolatest-to-k131-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-ko132-to-k132-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k131-ko132-to-k132-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-ko133-to-k132-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k131-ko133-to-k132-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-ko134-to-k132-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k131-ko134-to-k132-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k131-kolatest-to-k132-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k131-kolatest-to-k132-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k132-ko133-to-k133-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k132-ko133-to-k133-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k132-ko134-to-k133-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k132-ko134-to-k133-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k132-kolatest-to-k133-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k132-kolatest-to-k133-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k133-ko134-to-k134-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k133-ko134-to-k134-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k133-kolatest-to-k134-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k133-kolatest-to-k134-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-k134-kolatest-to-klatest-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#k134-kolatest-to-klatest-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-kstable-kolatest-to-kci-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#kstable-kolatest-to-kci-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
| e2e-kops-aws-upgrade-kstable-kolatest-to-klatest-kolatest-many-addons | https://testgrid.k8s.io/kops-upgrades-many-addons#kstable-kolatest-to-klatest-kolatest-many-addons | kops-upgrades-many-addons, kops-k8s-stable |
Each job was verified by:
- Checking TestGrid for jobs with FAILING status
- Fetching the most recent build log from GCS (
kubernetes-ci-logs/logs/<job-name>/<build-number>/build-log.txt) - Confirming presence of BOTH error messages in the log
Jobs verified with actual build log analysis:
- e2e-kops-aws-addon-resource-tracking (build 2014680399849984000)
- e2e-kops-aws-keypair-rotation-ha (build 2014676121735401472)
- e2e-kops-aws-upgrade-k128-ko132-to-k129-kolatest (build 2014579735710928896)
- e2e-kops-aws-karpenter (build 2014371098774212608)
- e2e-kops-aws-ipv6-karpenter (build 2014607921677078528)
- e2e-kops-aws-aws-load-balancer-controller (build 2014693989617242112)
- e2e-kops-aws-metrics-server (build 2014649949135835136)
- e2e-kops-aws-pod-identity-webhook (build 2014591563367190528)
- e2e-kops-aws-upgrade-k134-kolatest-to-klatest-kolatest (build 2014671088637710336)
-
Many other dashboards (kops-grid, kops-gce, kops-azure, kops-distros, kops-nftables, kops-network-plugins, etc.) have failing jobs but they fail due to different root causes:
- Network connectivity issues (i/o timeout, TLS handshake timeout)
- ARM64 compatibility issues (EBS CSI, CoreDNS)
- GCE backend conflicts
- Cilium/kindnet networking issues
- Regional AWS connectivity issues
-
These other failing jobs do NOT contain both of the specific error messages listed above.
-
Jobs confirmed NOT to have both errors:
- e2e-kops-grid-gce-cilium-deb12-k32-ko32 (test execution succeeded, different failure)
- e2e-kops-grid-amazonvpc-al2023-k32 (AWS ap-south-1 connectivity issues)
- e2e-kops-aws-distro-rhel10arm64 (ARM64/EBS CSI issues)
- e2e-kops-aws-nftables-amzn2 (network connectivity issues)
- e2e-kops-azure-cni-kindnet (CoreDNS/kindnet issues)
- ci-kubernetes-e2e-kops-aws-scale-amazonvpc-using-cl2 (no state store errors)
- e2e-kops-aws-hostname-bug121018 (bucket error was transient, self-resolved)