Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-28939

[2208809] Upgrade from 4.11.4->4.12.3 is stuck

XMLWordPrintable

    • High

      Description of problem: Upgrade to 4.12.3 (staging) from 4.11.4 (117) is stuck

      Version-Release number of selected component (if applicable):
      Upgrade path 4.11.4->4.12.3

      How reproducible:
      2 out of 2

      Steps to Reproduce:
      1. Upgrade CNV from 4.11.4->4.12.3
      2.
      3.

      Actual results: CSV is stuck in pending state:
      [cnv-qe-jenkins@cnv-qe-infra-01 ~]$ oc get csv -n openshift-cnv
      NAME DISPLAY VERSION REPLACES PHASE
      jaeger-operator.v1.42.0-5 Red Hat OpenShift distributed tracing platform 1.42.0-5 jaeger-operator.v1.34.1-5 Succeeded
      kiali-operator.v1.57.7 Kiali Operator 1.57.7 kiali-operator.v1.57.6 Succeeded
      kubevirt-hyperconverged-operator.v4.11.4 OpenShift Virtualization 4.11.4 kubevirt-hyperconverged-operator.v4.11.3 Installing
      kubevirt-hyperconverged-operator.v4.12.3 OpenShift Virtualization 4.12.3 kubevirt-hyperconverged-operator.v4.11.4 Pending
      servicemeshoperator.v2.3.3 Red Hat OpenShift Service Mesh 2.3.3-0 servicemeshoperator.v2.3.2 Succeeded
      [cnv-qe-jenkins@cnv-qe-infra-01 ~]$ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
      [

      { "name": "operator", "version": "4.11.4" }

      ]
      [cnv-qe-jenkins@cnv-qe-infra-01 ~]$
      ==================
      status.conditions shows:
      ==================

      • lastTransitionTime: "2023-05-21T00:54:51Z"
        lastUpdateTime: "2023-05-21T00:54:56Z"
        message: 'installing: waiting for deployment hco-operator to become ready: deployment
        "hco-operator" not available: Deployment does not have minimum availability.'
        phase: Installing
        reason: InstallWaiting
      • lastTransitionTime: "2023-05-21T00:59:50Z"
        lastUpdateTime: "2023-05-21T00:59:50Z"
        message: install timeout
        phase: Failed
        reason: InstallCheckFailed
        ===================
        [cnv-qe-jenkins@cnv-qe-infra-01 ~]$ oc get pods -n openshift-cnv | grep -v Running
        NAME READY STATUS RESTARTS AGE
        cdi-apiserver-66c8f9b7db-gj2rj 0/1 ImagePullBackOff 0 5h21m
        cdi-operator-b6dc64c6d-lcx5j 0/1 ImagePullBackOff 0 5h21m
        hco-operator-66589495d6-2hnt7 0/1 ImagePullBackOff 0 5h21m
        kubemacpool-cert-manager-559df4db5-s54x7 0/1 ImagePullBackOff 0 5h21m
        kubemacpool-mac-controller-manager-cfcd5b5d-jskg8 1/2 ImagePullBackOff 0 5h22m
        kubevirt-plugin-78765dc97b-sdm2b 0/1 ImagePullBackOff 0 5h21m
        ssp-operator-656859d6-b7c5c 0/1 ImagePullBackOff 0 5h21m
        virt-template-validator-57fb94c45c-2jm8g 0/1 ImagePullBackOff 0 5h18m
        virt-template-validator-57fb94c45c-2lh6t 0/1 ImagePullBackOff 0 5h21m
        [cnv-qe-jenkins@cnv-qe-infra-01 ~]$
        =====================
        Events:
        Type Reason Age From Message
            • ------ ---- ---- -------
              Warning Failed 136m (x41 over 5h21m) kubelet Failed to pull image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641": rpc error: code = Unknown desc = (Mirrors also failed: [registry.stage.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641: reading manifest sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641 in registry.stage.redhat.io/container-native-virtualization/hyperconverged-cluster-operator: manifest unknown: manifest unknown]): registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641: reading manifest sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641 in registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator: unsupported: Not Found, or unsupported. V2 schema 1 manifest digest are no longer supported for image pulls. Use the equivalent schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332
              Normal Pulling 131m (x42 over 5h21m) kubelet Pulling image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641"
              Normal BackOff 89s (x1398 over 5h21m) kubelet Back-off pulling image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:213269780b017a8c024ca90364df54d67aa7e1874c17aad22ab6fcf68689e641"
              [cnv-qe-jenkins@cnv-qe-infra-01 ~]$
              ======================

      Expected results:
      Upgrade from 4.11.4->4.12.3 would be successful

      Additional info:
      Cluster is available for further triage

            stirabos Simone Tiraboschi
            rhn-support-dbasunag Debarati Basu-Nag
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: