-
Bug
-
Resolution: Not a Bug
-
None
-
None
Description of problem: Upgrade to 4.12.3 (staging) from 4.11.4 (117) is stuck
Version-Release number of selected component (if applicable):
Upgrade path 4.11.4->4.12.3
How reproducible:
2 out of 2
Steps to Reproduce:
1. Upgrade CNV from 4.11.4->4.12.3
2.
3.
Actual results: CSV is stuck in pending state:
[cnv-qe-jenkins@cnv-qe-infra-01 ~]$ oc get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
jaeger-operator.v1.42.0-5 Red Hat OpenShift distributed tracing platform 1.42.0-5 jaeger-operator.v1.34.1-5 Succeeded
kiali-operator.v1.57.7 Kiali Operator 1.57.7 kiali-operator.v1.57.6 Succeeded
kubevirt-hyperconverged-operator.v4.11.4 OpenShift Virtualization 4.11.4 kubevirt-hyperconverged-operator.v4.11.3 Installing
kubevirt-hyperconverged-operator.v4.12.3 OpenShift Virtualization 4.12.3 kubevirt-hyperconverged-operator.v4.11.4 Pending
servicemeshoperator.v2.3.3 Red Hat OpenShift Service Mesh 2.3.3-0 servicemeshoperator.v2.3.2 Succeeded
[cnv-qe-jenkins@cnv-qe-infra-01 ~]$ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[
]
[cnv-qe-jenkins@cnv-qe-infra-01 ~]$
==================
status.conditions shows:
==================
- lastTransitionTime: "2023-05-21T00:54:51Z"
lastUpdateTime: "2023-05-21T00:54:56Z"
message: 'installing: waiting for deployment hco-operator to become ready: deployment
"hco-operator" not available: Deployment does not have minimum availability.'
phase: Installing
reason: InstallWaiting - lastTransitionTime: "2023-05-21T00:59:50Z"
lastUpdateTime: "2023-05-21T00:59:50Z"
message: install timeout
phase: Failed
reason: InstallCheckFailed
===================
[cnv-qe-jenkins@cnv-qe-infra-01 ~]$ oc get pods -n openshift-cnv | grep -v Running
NAME READY STATUS RESTARTS AGE
cdi-apiserver-66c8f9b7db-gj2rj 0/1 ImagePullBackOff 0 5h21m
cdi-operator-b6dc64c6d-lcx5j 0/1 ImagePullBackOff 0 5h21m
hco-operator-66589495d6-2hnt7 0/1 ImagePullBackOff 0 5h21m
kubemacpool-cert-manager-559df4db5-s54x7 0/1 ImagePullBackOff 0 5h21m
kubemacpool-mac-controller-manager-cfcd5b5d-jskg8 1/2 ImagePullBackOff 0 5h22m
kubevirt-plugin-78765dc97b-sdm2b 0/1 ImagePullBackOff 0 5h21m
ssp-operator-656859d6-b7c5c 0/1 ImagePullBackOff 0 5h21m
virt-template-validator-57fb94c45c-2jm8g 0/1 ImagePullBackOff 0 5h18m
virt-template-validator-57fb94c45c-2lh6t 0/1 ImagePullBackOff 0 5h21m
[cnv-qe-jenkins@cnv-qe-infra-01 ~]$
=====================
Events:
Type Reason Age From Message-
-
- ------ ---- ---- -------
Warning Failed 136m (x41 over 5h21m) kubelet Failed to pull image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641": rpc error: code = Unknown desc = (Mirrors also failed: [registry.stage.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641: reading manifest sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641 in registry.stage.redhat.io/container-native-virtualization/hyperconverged-cluster-operator: manifest unknown: manifest unknown]): registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641: reading manifest sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641 in registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator: unsupported: Not Found, or unsupported. V2 schema 1 manifest digest are no longer supported for image pulls. Use the equivalent schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332
Normal Pulling 131m (x42 over 5h21m) kubelet Pulling image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641"
Normal BackOff 89s (x1398 over 5h21m) kubelet Back-off pulling image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641"
[cnv-qe-jenkins@cnv-qe-infra-01 ~]$
======================
- ------ ---- ---- -------
-
-
Expected results:
Upgrade from 4.11.4->4.12.3 would be successful
Additional info:
Cluster is available for further triage