2022-03-08T10:42:49.585402684Z W0308 10:42:49.584942       1 etcd_env.go:298] cipher is not supported for use with etcd: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"
2022-03-08T10:42:49.585402684Z W0308 10:42:49.585101       1 etcd_env.go:298] cipher is not supported for use with etcd: "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
2022-03-08T10:42:49.585688854Z I0308 10:42:49.585662       1 observer_polling.go:52] Starting from specified content for file "/var/run/secrets/serving-cert/tls.crt"
2022-03-08T10:42:49.585713914Z I0308 10:42:49.585701       1 observer_polling.go:52] Starting from specified content for file "/var/run/secrets/serving-cert/tls.key"
2022-03-08T10:42:49.585764573Z I0308 10:42:49.585750       1 observer_polling.go:159] Starting file observer
2022-03-08T10:42:49.585897951Z I0308 10:42:49.585887       1 observer_polling.go:135] File observer successfully synced
2022-03-08T10:42:49.585994791Z I0308 10:42:49.585967       1 cmd.go:200] Using service-serving-cert provided certificates
2022-03-08T10:42:49.586043828Z I0308 10:42:49.586029       1 observer_polling.go:74] Adding reactor for file "/var/run/secrets/serving-cert/tls.crt"
2022-03-08T10:42:49.586160306Z I0308 10:42:49.586138       1 observer_polling.go:74] Adding reactor for file "/var/run/secrets/serving-cert/tls.key"
2022-03-08T10:42:49.586256409Z I0308 10:42:49.586237       1 observer_polling.go:52] Starting from specified content for file "/var/run/configmaps/config/config.yaml"
2022-03-08T10:42:49.586985922Z I0308 10:42:49.586962       1 observer_polling.go:159] Starting file observer
2022-03-08T10:42:49.587130990Z I0308 10:42:49.587110       1 observer_polling.go:135] File observer successfully synced
2022-03-08T10:42:49.623309180Z I0308 10:42:49.623220       1 builder.go:238] openshift-cluster-etcd-operator version v0.0.0-alpha.0-603-g6a3f3b50-6a3f3b506c5e824b40cbca78bcc8f8c0219b5b4a
2022-03-08T10:42:49.624095902Z I0308 10:42:49.624044       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"
2022-03-08T10:42:49.996606701Z I0308 10:42:49.996497       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
2022-03-08T10:42:49.998608436Z I0308 10:42:49.998534       1 config.go:638] Not requested to run hook priority-and-fairness-config-consumer
2022-03-08T10:42:50.000444815Z W0308 10:42:50.000386       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
2022-03-08T10:42:50.000592541Z W0308 10:42:50.000572       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
2022-03-08T10:42:50.002125817Z I0308 10:42:50.002028       1 leaderelection.go:243] attempting to acquire leader lease  openshift-etcd-operator/openshift-cluster-etcd-operator-lock...
2022-03-08T10:42:50.009003801Z I0308 10:42:50.008907       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
2022-03-08T10:42:50.009003801Z I0308 10:42:50.008951       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
2022-03-08T10:42:50.009003801Z I0308 10:42:50.008951       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2022-03-08T10:42:50.009003801Z I0308 10:42:50.008984       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2022-03-08T10:42:50.009093090Z I0308 10:42:50.008984       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2022-03-08T10:42:50.009093090Z I0308 10:42:50.009023       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2022-03-08T10:42:50.009142485Z I0308 10:42:50.009098       1 reflector.go:207] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:42:50.009142485Z I0308 10:42:50.009115       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:42:50.009142485Z I0308 10:42:50.009123       1 reflector.go:207] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:42:50.009142485Z I0308 10:42:50.009131       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:42:50.009394152Z I0308 10:42:50.009100       1 reflector.go:207] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:42:50.009394152Z I0308 10:42:50.009376       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:42:50.009843466Z I0308 10:42:50.009817       1 dynamic_serving_content.go:130] Starting serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key
2022-03-08T10:42:50.010049687Z I0308 10:42:50.010016       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-etcd-operator.svc" [serving] validServingFor=[metrics.openshift-etcd-operator.svc,metrics.openshift-etcd-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1646735940" (2022-03-08 10:39:21 +0000 UTC to 2024-03-07 10:39:22 +0000 UTC (now=2022-03-08 10:42:50.009987921 +0000 UTC))
2022-03-08T10:42:50.010342222Z I0308 10:42:50.010305       1 leaderelection.go:342] lock is held by etcd-operator-859798c7f4-kb5lq_02b0f5d9-9c2c-41fc-ad32-5e15a246e928 and has not yet expired
2022-03-08T10:42:50.010342222Z I0308 10:42:50.010324       1 leaderelection.go:248] failed to acquire lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
2022-03-08T10:42:50.011666684Z I0308 10:42:50.011614       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1646736169" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1646736169" (2022-03-08 09:42:49 +0000 UTC to 2023-03-08 09:42:49 +0000 UTC (now=2022-03-08 10:42:50.010276476 +0000 UTC))
2022-03-08T10:42:50.011699792Z I0308 10:42:50.011672       1 secure_serving.go:197] Serving securely on [::]:8443
2022-03-08T10:42:50.011699792Z I0308 10:42:50.011695       1 tlsconfig.go:240] Starting DynamicServingCertificateController
2022-03-08T10:42:50.109223706Z I0308 10:42:50.109148       1 shared_informer.go:270] caches populated
2022-03-08T10:42:50.109223706Z I0308 10:42:50.109190       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
2022-03-08T10:42:50.109223706Z I0308 10:42:50.109203       1 shared_informer.go:270] caches populated
2022-03-08T10:42:50.109268820Z I0308 10:42:50.109224       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
2022-03-08T10:42:50.109622181Z I0308 10:42:50.109582       1 tlsconfig.go:178] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2022-03-08 10:10:28 +0000 UTC to 2022-03-09 10:10:28 +0000 UTC (now=2022-03-08 10:42:50.109563108 +0000 UTC))
2022-03-08T10:42:50.109894709Z I0308 10:42:50.109832       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-etcd-operator.svc" [serving] validServingFor=[metrics.openshift-etcd-operator.svc,metrics.openshift-etcd-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1646735940" (2022-03-08 10:39:21 +0000 UTC to 2024-03-07 10:39:22 +0000 UTC (now=2022-03-08 10:42:50.109824352 +0000 UTC))
2022-03-08T10:42:50.110017614Z I0308 10:42:50.110002       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1646736169" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1646736169" (2022-03-08 09:42:49 +0000 UTC to 2023-03-08 09:42:49 +0000 UTC (now=2022-03-08 10:42:50.10999511 +0000 UTC))
2022-03-08T10:42:50.110088526Z I0308 10:42:50.110046       1 shared_informer.go:270] caches populated
2022-03-08T10:42:50.110121234Z I0308 10:42:50.110107       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
2022-03-08T10:42:50.110981238Z I0308 10:42:50.110931       1 tlsconfig.go:178] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "admin-kubeconfig-signer" [] issuer="<self>" (2022-03-08 10:10:26 +0000 UTC to 2032-03-05 10:10:26 +0000 UTC (now=2022-03-08 10:42:50.110918564 +0000 UTC))
2022-03-08T10:42:50.110981238Z I0308 10:42:50.110952       1 tlsconfig.go:178] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-signer" [] issuer="<self>" (2022-03-08 10:10:30 +0000 UTC to 2022-03-09 10:10:30 +0000 UTC (now=2022-03-08 10:42:50.110947122 +0000 UTC))
2022-03-08T10:42:50.110981238Z I0308 10:42:50.110965       1 tlsconfig.go:178] loaded client CA [2/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-control-plane-signer" [] issuer="<self>" (2022-03-08 10:10:30 +0000 UTC to 2023-03-08 10:10:30 +0000 UTC (now=2022-03-08 10:42:50.110960337 +0000 UTC))
2022-03-08T10:42:50.110981238Z I0308 10:42:50.110974       1 tlsconfig.go:178] loaded client CA [3/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-apiserver-to-kubelet-signer" [] issuer="<self>" (2022-03-08 10:10:31 +0000 UTC to 2023-03-08 10:10:31 +0000 UTC (now=2022-03-08 10:42:50.110969993 +0000 UTC))
2022-03-08T10:42:50.111006267Z I0308 10:42:50.110993       1 tlsconfig.go:178] loaded client CA [4/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="<self>" (2022-03-08 10:10:27 +0000 UTC to 2032-03-05 10:10:27 +0000 UTC (now=2022-03-08 10:42:50.110979195 +0000 UTC))
2022-03-08T10:42:50.111016364Z I0308 10:42:50.111003       1 tlsconfig.go:178] loaded client CA [5/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-csr-signer_@1646735926" [] issuer="kubelet-signer" (2022-03-08 10:38:45 +0000 UTC to 2022-03-09 10:10:30 +0000 UTC (now=2022-03-08 10:42:50.110999247 +0000 UTC))
2022-03-08T10:42:50.111027526Z I0308 10:42:50.111016       1 tlsconfig.go:178] loaded client CA [6/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "openshift-kube-apiserver-operator_node-system-admin-signer@1646735931" [] issuer="<self>" (2022-03-08 10:38:50 +0000 UTC to 2023-03-08 10:38:51 +0000 UTC (now=2022-03-08 10:42:50.111008469 +0000 UTC))
2022-03-08T10:42:50.111081023Z I0308 10:42:50.111027       1 tlsconfig.go:178] loaded client CA [7/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2022-03-08 10:10:28 +0000 UTC to 2022-03-09 10:10:28 +0000 UTC (now=2022-03-08 10:42:50.111023021 +0000 UTC))
2022-03-08T10:42:50.112525634Z I0308 10:42:50.112431       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-etcd-operator.svc" [serving] validServingFor=[metrics.openshift-etcd-operator.svc,metrics.openshift-etcd-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1646735940" (2022-03-08 10:39:21 +0000 UTC to 2024-03-07 10:39:22 +0000 UTC (now=2022-03-08 10:42:50.112418735 +0000 UTC))
2022-03-08T10:42:50.112749351Z I0308 10:42:50.112719       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1646736169" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1646736169" (2022-03-08 09:42:49 +0000 UTC to 2023-03-08 09:42:49 +0000 UTC (now=2022-03-08 10:42:50.112706915 +0000 UTC))
2022-03-08T10:43:10.471422453Z I0308 10:43:10.471333       1 leaderelection.go:342] lock is held by etcd-operator-859798c7f4-kb5lq_02b0f5d9-9c2c-41fc-ad32-5e15a246e928 and has not yet expired
2022-03-08T10:43:10.471422453Z I0308 10:43:10.471363       1 leaderelection.go:248] failed to acquire lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
2022-03-08T10:43:23.175495517Z I0308 10:43:23.175408       1 leaderelection.go:342] lock is held by etcd-operator-859798c7f4-kb5lq_02b0f5d9-9c2c-41fc-ad32-5e15a246e928 and has not yet expired
2022-03-08T10:43:23.175495517Z I0308 10:43:23.175440       1 leaderelection.go:248] failed to acquire lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
2022-03-08T10:43:37.398579645Z I0308 10:43:37.398534       1 leaderelection.go:342] lock is held by etcd-operator-859798c7f4-kb5lq_02b0f5d9-9c2c-41fc-ad32-5e15a246e928 and has not yet expired
2022-03-08T10:43:37.398646018Z I0308 10:43:37.398637       1 leaderelection.go:248] failed to acquire lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
2022-03-08T10:43:57.560790578Z I0308 10:43:57.560730       1 leaderelection.go:253] successfully acquired lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
2022-03-08T10:43:57.561358406Z I0308 10:43:57.561161       1 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-etcd-operator", Name:"openshift-cluster-etcd-operator-lock", UID:"09fdc970-c8c0-40be-92ab-e7e5cd5c6a6f", APIVersion:"v1", ResourceVersion:"16051", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' etcd-operator-859798c7f4-kb5lq_18fcd5e6-c8d0-4c4d-aba7-24ba745ea443 became leader
2022-03-08T10:43:57.580592312Z I0308 10:43:57.580539       1 base_controller.go:66] Waiting for caches to sync for LoggingSyncer
2022-03-08T10:43:57.581940966Z I0308 10:43:57.580719       1 base_controller.go:66] Waiting for caches to sync for TargetConfigController
2022-03-08T10:43:57.581997360Z I0308 10:43:57.580724       1 base_controller.go:66] Waiting for caches to sync for ClusterMemberController
2022-03-08T10:43:57.582018316Z I0308 10:43:57.580744       1 base_controller.go:66] Waiting for caches to sync for EtcdCertSignerController
2022-03-08T10:43:57.582084039Z I0308 10:43:57.580741       1 reflector.go:191] Reflector from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156 configured with expectedType of *unstructured.Unstructured with empty GroupVersionKind.
2022-03-08T10:43:57.582125471Z I0308 10:43:57.582112       1 reflector.go:207] Starting reflector *unstructured.Unstructured (12h0m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.582159287Z I0308 10:43:57.580757       1 base_controller.go:66] Waiting for caches to sync for HostEndpoints2Controller
2022-03-08T10:43:57.582184776Z I0308 10:43:57.582150       1 reflector.go:243] Listing and watching *unstructured.Unstructured from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.582283414Z I0308 10:43:57.580762       1 base_controller.go:66] Waiting for caches to sync for EtcdEndpointsController
2022-03-08T10:43:57.582326285Z I0308 10:43:57.580772       1 base_controller.go:66] Waiting for caches to sync for ResourceSyncController
2022-03-08T10:43:57.582326285Z I0308 10:43:57.580949       1 base_controller.go:66] Waiting for caches to sync for StatusSyncer_etcd
2022-03-08T10:43:57.582339718Z I0308 10:43:57.580956       1 base_controller.go:66] Waiting for caches to sync for ConfigObserver
2022-03-08T10:43:57.582348296Z I0308 10:43:57.581095       1 base_controller.go:66] Waiting for caches to sync for InstallerController
2022-03-08T10:43:57.582356716Z I0308 10:43:57.581106       1 base_controller.go:66] Waiting for caches to sync for EtcdMembersController
2022-03-08T10:43:57.582365458Z I0308 10:43:57.582358       1 shared_informer.go:270] caches populated
2022-03-08T10:43:57.582374125Z I0308 10:43:57.582364       1 base_controller.go:72] Caches are synced for EtcdMembersController 
2022-03-08T10:43:57.582374125Z I0308 10:43:57.582369       1 base_controller.go:109] Starting #1 worker of EtcdMembersController controller ...
2022-03-08T10:43:57.582398179Z I0308 10:43:57.581111       1 base_controller.go:66] Waiting for caches to sync for BootstrapTeardownController
2022-03-08T10:43:57.582427178Z I0308 10:43:57.581120       1 base_controller.go:66] Waiting for caches to sync for UnsupportedConfigOverridesController
2022-03-08T10:43:57.582457315Z I0308 10:43:57.581127       1 base_controller.go:66] Waiting for caches to sync for ScriptController
2022-03-08T10:43:57.582496370Z I0308 10:43:57.581140       1 envvarcontroller.go:161] Starting EnvVarController
2022-03-08T10:43:57.582515731Z I0308 10:43:57.581153       1 base_controller.go:66] Waiting for caches to sync for RevisionController
2022-03-08T10:43:57.582538944Z I0308 10:43:57.581169       1 base_controller.go:66] Waiting for caches to sync for RemoveStaleConditionsController
2022-03-08T10:43:57.582558243Z I0308 10:43:57.581266       1 base_controller.go:66] Waiting for caches to sync for StaticResourceController
2022-03-08T10:43:57.582578397Z I0308 10:43:57.581273       1 base_controller.go:66] Waiting for caches to sync for FSyncController
2022-03-08T10:43:57.582598571Z I0308 10:43:57.582590       1 shared_informer.go:270] caches populated
2022-03-08T10:43:57.582615234Z I0308 10:43:57.582608       1 base_controller.go:72] Caches are synced for FSyncController 
2022-03-08T10:43:57.582632442Z I0308 10:43:57.582625       1 base_controller.go:109] Starting #1 worker of FSyncController controller ...
2022-03-08T10:43:57.582664409Z I0308 10:43:57.581327       1 base_controller.go:66] Waiting for caches to sync for PruneController
2022-03-08T10:43:57.582904394Z I0308 10:43:57.581335       1 base_controller.go:66] Waiting for caches to sync for InstallerStateController
2022-03-08T10:43:57.582928846Z I0308 10:43:57.581341       1 base_controller.go:66] Waiting for caches to sync for StaticPodStateController
2022-03-08T10:43:57.582974131Z I0308 10:43:57.581363       1 reflector.go:207] Starting reflector *v1.ConfigMap (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583000521Z I0308 10:43:57.582990       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583045712Z I0308 10:43:57.581391       1 reflector.go:207] Starting reflector *v1.Etcd (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583045712Z I0308 10:43:57.583040       1 reflector.go:243] Listing and watching *v1.Etcd from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583199530Z I0308 10:43:57.581406       1 reflector.go:207] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583199530Z I0308 10:43:57.583190       1 reflector.go:243] Listing and watching *v1.Secret from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583270845Z I0308 10:43:57.581420       1 reflector.go:207] Starting reflector *v1.Endpoints (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583270845Z I0308 10:43:57.583256       1 reflector.go:243] Listing and watching *v1.Endpoints from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583316892Z I0308 10:43:57.581435       1 reflector.go:207] Starting reflector *v1.Pod (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583316892Z I0308 10:43:57.583310       1 reflector.go:243] Listing and watching *v1.Pod from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583363283Z I0308 10:43:57.581449       1 reflector.go:207] Starting reflector *v1.Namespace (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583363283Z I0308 10:43:57.583357       1 reflector.go:243] Listing and watching *v1.Namespace from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583437019Z I0308 10:43:57.581507       1 reflector.go:207] Starting reflector *v1.ServiceAccount (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583437019Z I0308 10:43:57.583431       1 reflector.go:243] Listing and watching *v1.ServiceAccount from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583521339Z I0308 10:43:57.581519       1 base_controller.go:66] Waiting for caches to sync for StaticResourceController
2022-03-08T10:43:57.583521339Z I0308 10:43:57.581521       1 reflector.go:207] Starting reflector *v1.ConfigMap (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583534912Z I0308 10:43:57.583520       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583575275Z I0308 10:43:57.581530       1 base_controller.go:66] Waiting for caches to sync for NodeController
2022-03-08T10:43:57.583575275Z I0308 10:43:57.581535       1 reflector.go:207] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583585708Z I0308 10:43:57.583574       1 reflector.go:243] Listing and watching *v1.Secret from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583626375Z I0308 10:43:57.581553       1 base_controller.go:66] Waiting for caches to sync for UnsupportedConfigOverridesController
2022-03-08T10:43:57.583626375Z I0308 10:43:57.581574       1 reflector.go:207] Starting reflector *v1.ConfigMap (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.583639863Z I0308 10:43:57.583627       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.584869923Z I0308 10:43:57.582991       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ReportEtcdMembersErrorUpdatingStatus' etcds.operator.openshift.io "cluster" not found
2022-03-08T10:43:57.584915731Z I0308 10:43:57.581605       1 reflector.go:207] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.584950869Z I0308 10:43:57.584937       1 reflector.go:243] Listing and watching *v1.Secret from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585134867Z I0308 10:43:57.581590       1 reflector.go:207] Starting reflector *v1.ConfigMap (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585134867Z I0308 10:43:57.585090       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585156843Z I0308 10:43:57.581631       1 reflector.go:207] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585156843Z I0308 10:43:57.585144       1 reflector.go:243] Listing and watching *v1.Secret from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585185710Z I0308 10:43:57.581646       1 reflector.go:207] Starting reflector *v1.Node (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585185710Z I0308 10:43:57.585181       1 reflector.go:243] Listing and watching *v1.Node from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585223207Z I0308 10:43:57.581662       1 reflector.go:207] Starting reflector *v1.ClusterRoleBinding (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585223207Z I0308 10:43:57.585218       1 reflector.go:243] Listing and watching *v1.ClusterRoleBinding from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585347773Z I0308 10:43:57.581783       1 reflector.go:207] Starting reflector *v1.ClusterOperator (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585347773Z I0308 10:43:57.585334       1 reflector.go:243] Listing and watching *v1.ClusterOperator from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585394117Z I0308 10:43:57.581791       1 reflector.go:207] Starting reflector *v1.Network (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585394117Z I0308 10:43:57.585389       1 reflector.go:243] Listing and watching *v1.Network from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585485882Z I0308 10:43:57.581806       1 reflector.go:207] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585485882Z I0308 10:43:57.585478       1 reflector.go:243] Listing and watching *v1.Secret from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585518331Z I0308 10:43:57.581821       1 reflector.go:207] Starting reflector *v1.ConfigMap (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585518331Z I0308 10:43:57.585513       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585545931Z I0308 10:43:57.581824       1 reflector.go:207] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585545931Z I0308 10:43:57.585541       1 reflector.go:243] Listing and watching *v1.Secret from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585573471Z I0308 10:43:57.581885       1 reflector.go:207] Starting reflector *v1.Infrastructure (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585573471Z I0308 10:43:57.585569       1 reflector.go:243] Listing and watching *v1.Infrastructure from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585632150Z E0308 10:43:57.582510       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:57.585646258Z I0308 10:43:57.581618       1 reflector.go:207] Starting reflector *v1.ConfigMap (10m0s) from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.585646258Z I0308 10:43:57.585643       1 reflector.go:243] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.19.0/tools/cache/reflector.go:156
2022-03-08T10:43:57.592865524Z E0308 10:43:57.592805       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:57.592994052Z I0308 10:43:57.592967       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ReportEtcdMembersErrorUpdatingStatus' etcds.operator.openshift.io "cluster" not found
2022-03-08T10:43:57.612452697Z E0308 10:43:57.612403       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:57.616946227Z I0308 10:43:57.616782       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ReportEtcdMembersErrorUpdatingStatus' etcds.operator.openshift.io "cluster" not found
2022-03-08T10:43:57.635414883Z E0308 10:43:57.635369       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:43:57.650373150Z E0308 10:43:57.650327       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:57.682447415Z I0308 10:43:57.682396       1 shared_informer.go:270] caches populated
2022-03-08T10:43:57.682530592Z I0308 10:43:57.682519       1 base_controller.go:72] Caches are synced for LoggingSyncer 
2022-03-08T10:43:57.682553484Z I0308 10:43:57.682545       1 base_controller.go:109] Starting #1 worker of LoggingSyncer controller ...
2022-03-08T10:43:57.683317098Z I0308 10:43:57.683271       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorLogLevelChange' Operator log level changed from "Debug" to "Normal"
2022-03-08T10:43:57.683531152Z I0308 10:43:57.683375       1 base_controller.go:72] Caches are synced for InstallerController 
2022-03-08T10:43:57.683566219Z I0308 10:43:57.683555       1 base_controller.go:109] Starting #1 worker of InstallerController controller ...
2022-03-08T10:43:57.684343604Z I0308 10:43:57.683395       1 base_controller.go:72] Caches are synced for StatusSyncer_etcd 
2022-03-08T10:43:57.684556107Z I0308 10:43:57.684539       1 base_controller.go:109] Starting #1 worker of StatusSyncer_etcd controller ...
2022-03-08T10:43:57.684605166Z I0308 10:43:57.683419       1 base_controller.go:72] Caches are synced for StaticPodStateController 
2022-03-08T10:43:57.685503168Z I0308 10:43:57.683435       1 base_controller.go:72] Caches are synced for UnsupportedConfigOverridesController 
2022-03-08T10:43:57.685569905Z I0308 10:43:57.683442       1 base_controller.go:72] Caches are synced for ScriptController 
2022-03-08T10:43:57.685609862Z I0308 10:43:57.683454       1 base_controller.go:72] Caches are synced for RevisionController 
2022-03-08T10:43:57.685647417Z I0308 10:43:57.683474       1 base_controller.go:72] Caches are synced for RemoveStaleConditionsController 
2022-03-08T10:43:57.685689764Z I0308 10:43:57.683482       1 base_controller.go:72] Caches are synced for StaticResourceController 
2022-03-08T10:43:57.685753428Z I0308 10:43:57.683489       1 base_controller.go:72] Caches are synced for PruneController 
2022-03-08T10:43:57.685791689Z I0308 10:43:57.683497       1 base_controller.go:72] Caches are synced for InstallerStateController 
2022-03-08T10:43:57.685828857Z I0308 10:43:57.684493       1 base_controller.go:72] Caches are synced for UnsupportedConfigOverridesController 
2022-03-08T10:43:57.685865948Z I0308 10:43:57.684506       1 base_controller.go:72] Caches are synced for StaticResourceController 
2022-03-08T10:43:57.685908612Z I0308 10:43:57.685424       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"EtcdMembersControllerDegraded: node lister not synced\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:43:57.686260218Z I0308 10:43:57.685544       1 base_controller.go:109] Starting #1 worker of StaticPodStateController controller ...
2022-03-08T10:43:57.686316143Z I0308 10:43:57.685589       1 base_controller.go:109] Starting #1 worker of UnsupportedConfigOverridesController controller ...
2022-03-08T10:43:57.687008781Z I0308 10:43:57.685627       1 base_controller.go:109] Starting #1 worker of ScriptController controller ...
2022-03-08T10:43:57.687048116Z I0308 10:43:57.685670       1 base_controller.go:109] Starting #1 worker of RevisionController controller ...
2022-03-08T10:43:57.688858516Z I0308 10:43:57.685726       1 base_controller.go:109] Starting #1 worker of RemoveStaleConditionsController controller ...
2022-03-08T10:43:57.688915111Z I0308 10:43:57.685773       1 base_controller.go:109] Starting #1 worker of StaticResourceController controller ...
2022-03-08T10:43:57.689254497Z I0308 10:43:57.685808       1 base_controller.go:109] Starting #1 worker of PruneController controller ...
2022-03-08T10:43:57.694686292Z I0308 10:43:57.685847       1 base_controller.go:109] Starting #1 worker of InstallerStateController controller ...
2022-03-08T10:43:57.694763112Z I0308 10:43:57.685886       1 base_controller.go:109] Starting #1 worker of UnsupportedConfigOverridesController controller ...
2022-03-08T10:43:57.695395424Z I0308 10:43:57.685926       1 base_controller.go:109] Starting #1 worker of StaticResourceController controller ...
2022-03-08T10:43:57.696398011Z E0308 10:43:57.696366       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:57.709030423Z I0308 10:43:57.708976       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:43:57.710678513Z E0308 10:43:57.710645       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:57.717248788Z E0308 10:43:57.717206       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:57.717555199Z E0308 10:43:57.717539       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:57.717988052Z I0308 10:43:57.717971       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:43:57.734332888Z I0308 10:43:57.734283       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:43:57.739198404Z E0308 10:43:57.739141       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:57.777122628Z E0308 10:43:57.777037       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:57.779936362Z E0308 10:43:57.779898       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:57.860641720Z E0308 10:43:57.860583       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:57.937673546Z E0308 10:43:57.937624       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.021431845Z E0308 10:43:58.021379       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:58.082566937Z I0308 10:43:58.082508       1 base_controller.go:72] Caches are synced for BootstrapTeardownController 
2022-03-08T10:43:58.082681023Z I0308 10:43:58.082663       1 base_controller.go:109] Starting #1 worker of BootstrapTeardownController controller ...
2022-03-08T10:43:58.113165238Z E0308 10:43:58.113114       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.113447787Z E0308 10:43:58.113431       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.113796738Z E0308 10:43:58.113775       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:58.114721645Z I0308 10:43:58.114692       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:43:58.118561854Z E0308 10:43:58.118530       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.126995236Z I0308 10:43:58.126943       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:43:58.139166780Z E0308 10:43:58.139111       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.179832277Z E0308 10:43:58.179768       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.258321327Z E0308 10:43:58.258276       1 base_controller.go:250] "EtcdMembersController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.260350512Z E0308 10:43:58.260315       1 base_controller.go:250] "BootstrapTeardownController" controller failed to sync "key", err: node lister not synced
2022-03-08T10:43:58.342255914Z E0308 10:43:58.342203       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:58.430088350Z WARNING: 2022/03/08 10:43:58 grpc: addrConn.createTransport failed to connect to {https://10.0.89.186:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.89.186:2379: connect: connection refused". Reconnecting...
2022-03-08T10:43:58.479015415Z E0308 10:43:58.478951       1 base_controller.go:250] "ScriptController" controller failed to sync "key", err: "configmap/etcd-pod": missing env var values
2022-03-08T10:43:58.482880008Z I0308 10:43:58.482819       1 base_controller.go:72] Caches are synced for TargetConfigController 
2022-03-08T10:43:58.482975250Z I0308 10:43:58.482959       1 base_controller.go:109] Starting #1 worker of TargetConfigController controller ...
2022-03-08T10:43:58.483046220Z I0308 10:43:58.483030       1 base_controller.go:72] Caches are synced for ClusterMemberController 
2022-03-08T10:43:58.483178933Z I0308 10:43:58.483150       1 base_controller.go:109] Starting #1 worker of ClusterMemberController controller ...
2022-03-08T10:43:58.483430035Z E0308 10:43:58.483413       1 base_controller.go:250] "TargetConfigController" controller failed to sync "key", err: missing env var values
2022-03-08T10:43:58.484937309Z WARNING: 2022/03/08 10:43:58 grpc: addrConn.createTransport failed to connect to {https://10.0.89.186:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.89.186:2379: connect: connection refused". Reconnecting...
2022-03-08T10:43:58.484990735Z I0308 10:43:58.484946       1 envvarcontroller.go:167] caches synced
2022-03-08T10:43:58.485018326Z I0308 10:43:58.484971       1 base_controller.go:72] Caches are synced for EtcdEndpointsController 
2022-03-08T10:43:58.485039915Z I0308 10:43:58.485031       1 base_controller.go:109] Starting #1 worker of EtcdEndpointsController controller ...
2022-03-08T10:43:58.485984533Z I0308 10:43:58.485957       1 base_controller.go:72] Caches are synced for NodeController 
2022-03-08T10:43:58.486021786Z I0308 10:43:58.486014       1 base_controller.go:109] Starting #1 worker of NodeController controller ...
2022-03-08T10:43:58.486250619Z I0308 10:43:58.486218       1 base_controller.go:72] Caches are synced for HostEndpoints2Controller 
2022-03-08T10:43:58.486263256Z I0308 10:43:58.486245       1 base_controller.go:109] Starting #1 worker of HostEndpoints2Controller controller ...
2022-03-08T10:43:58.523100317Z WARNING: 2022/03/08 10:43:58 grpc: addrConn.createTransport failed to connect to {https://10.0.89.186:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.89.186:2379: connect: connection refused". Reconnecting...
2022-03-08T10:43:58.526643423Z I0308 10:43:58.526571       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:43:58.564438169Z I0308 10:43:58.547083       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:43:58.577261217Z I0308 10:43:58.577185       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/etcd-endpoints -n openshift-etcd:
2022-03-08T10:43:58.603203574Z I0308 10:43:58.603118       1 request.go:645] Throttling request took 1.017575414s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-config/secrets?limit=500&resourceVersion=0
2022-03-08T10:43:58.682250262Z I0308 10:43:58.682185       1 base_controller.go:72] Caches are synced for EtcdCertSignerController 
2022-03-08T10:43:58.682325178Z I0308 10:43:58.682314       1 base_controller.go:109] Starting #1 worker of EtcdCertSignerController controller ...
2022-03-08T10:43:58.882562395Z I0308 10:43:58.882486       1 base_controller.go:72] Caches are synced for ConfigObserver 
2022-03-08T10:43:58.882638363Z I0308 10:43:58.882627       1 base_controller.go:109] Starting #1 worker of ConfigObserver controller ...
2022-03-08T10:43:58.951123248Z I0308 10:43:58.951041       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:43:58.966830166Z I0308 10:43:58.966780       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available"
2022-03-08T10:43:58.977172941Z I0308 10:43:58.977123       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"ScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:43:58.988665984Z I0308 10:43:58.988226       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "ScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:43:59.282543541Z I0308 10:43:59.282484       1 base_controller.go:72] Caches are synced for ResourceSyncController 
2022-03-08T10:43:59.282543541Z I0308 10:43:59.282520       1 base_controller.go:109] Starting #1 worker of ResourceSyncController controller ...
2022-03-08T10:43:59.425390221Z E0308 10:43:59.425328       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:43:59.783155606Z I0308 10:43:59.783014       1 request.go:645] Throttling request took 2.096334202s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:00.783272957Z I0308 10:44:00.783203       1 request.go:645] Throttling request took 2.297333016s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T10:44:00.837816083Z I0308 10:44:00.836360       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:41:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:44:00.862508600Z I0308 10:44:00.862426       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "ScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:44:01.199591778Z E0308 10:44:01.199537       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:01.983097956Z I0308 10:44:01.983002       1 request.go:645] Throttling request took 1.794430451s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T10:44:02.797887252Z E0308 10:44:02.797676       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:03.183082282Z I0308 10:44:03.182991       1 request.go:645] Throttling request took 1.594791756s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:03.997772911Z I0308 10:44:03.997695       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/etcd-scripts -n openshift-etcd:
2022-03-08T10:44:03.997772911Z cause by changes in data.etcd.env
2022-03-08T10:44:04.383158533Z I0308 10:44:04.382990       1 request.go:645] Throttling request took 1.544804085s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/secrets
2022-03-08T10:44:04.401536746Z E0308 10:44:04.401428       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:05.383210295Z I0308 10:44:05.383124       1 request.go:645] Throttling request took 1.387292775s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T10:44:05.800447768Z E0308 10:44:05.800369       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:06.583093642Z I0308 10:44:06.582988       1 request.go:645] Throttling request took 1.395404578s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T10:44:07.211643157Z E0308 10:44:07.211259       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:07.588148743Z I0308 10:44:07.586595       1 request.go:645] Throttling request took 1.396649838s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:08.203042646Z I0308 10:44:08.202947       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/etcd-pod -n openshift-etcd:
2022-03-08T10:44:08.203042646Z cause by changes in data.pod.yaml
2022-03-08T10:44:08.215832863Z I0308 10:44:08.215703       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RevisionTriggered' new revision 3 triggered by "configmap/etcd-pod has changed"
2022-03-08T10:44:08.602370386Z E0308 10:44:08.602274       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:09.026877101Z I0308 10:44:09.026821       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/revision-status-3 -n openshift-etcd because it was missing
2022-03-08T10:44:09.414734854Z I0308 10:44:09.414618       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/restore-etcd-pod -n openshift-etcd:
2022-03-08T10:44:09.414734854Z cause by changes in data.pod.yaml
2022-03-08T10:44:09.627234067Z I0308 10:44:09.627173       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/etcd-pod-3 -n openshift-etcd because it was missing
2022-03-08T10:44:09.847968473Z E0308 10:44:09.847888       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:10.616238315Z I0308 10:44:10.616134       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/config-3 -n openshift-etcd because it was missing
2022-03-08T10:44:11.400051050Z I0308 10:44:11.399958       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/etcd-serving-ca-3 -n openshift-etcd because it was missing
2022-03-08T10:44:12.008578063Z E0308 10:44:12.008487       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:12.408010970Z I0308 10:44:12.407861       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/etcd-peer-client-ca-3 -n openshift-etcd because it was missing
2022-03-08T10:44:13.007303978Z I0308 10:44:13.007208       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/etcd-metrics-proxy-serving-ca-3 -n openshift-etcd because it was missing
2022-03-08T10:44:13.405648066Z I0308 10:44:13.405573       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/etcd-metrics-proxy-client-ca-3 -n openshift-etcd because it was missing
2022-03-08T10:44:13.805156481Z I0308 10:44:13.805103       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/etcd-all-peer-3 -n openshift-etcd because it was missing
2022-03-08T10:44:14.213592329Z I0308 10:44:14.213451       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/etcd-all-serving-3 -n openshift-etcd because it was missing
2022-03-08T10:44:14.645107415Z I0308 10:44:14.640154       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/etcd-all-serving-metrics-3 -n openshift-etcd because it was missing
2022-03-08T10:44:14.672112520Z I0308 10:44:14.665210       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RevisionCreate' Revision 2 created because configmap/etcd-pod has changed
2022-03-08T10:44:14.699427782Z WARNING: 2022/03/08 10:44:14 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2022-03-08T10:44:15.028191826Z E0308 10:44:15.028105       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:15.783094417Z I0308 10:44:15.782991       1 request.go:645] Throttling request took 1.120949424s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:19.398692885Z I0308 10:44:19.398628       1 installer_controller.go:350] node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest and needs new revision 3
2022-03-08T10:44:19.398692885Z I0308 10:44:19.398678       1 installer_controller.go:359] "master-2.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:44:19.398692885Z  NodeName: (string) (len=49) "master-2.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:44:19.398692885Z  CurrentRevision: (int32) 2,
2022-03-08T10:44:19.398692885Z  TargetRevision: (int32) 3,
2022-03-08T10:44:19.398692885Z  LastFailedRevision: (int32) 0,
2022-03-08T10:44:19.398692885Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:44:19.398692885Z }
2022-03-08T10:44:19.413314588Z I0308 10:44:19.413185       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeTargetRevisionChanged' Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
2022-03-08T10:44:19.418698480Z I0308 10:44:19.418638       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:44:19.435139684Z I0308 10:44:19.435086       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available"
2022-03-08T10:44:20.583007985Z I0308 10:44:20.582948       1 request.go:645] Throttling request took 1.163490861s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:21.201957407Z E0308 10:44:21.201890       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:21.583111876Z I0308 10:44:21.583022       1 request.go:645] Throttling request took 1.191055623s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:22.040566314Z I0308 10:44:22.040493       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
2022-03-08T10:44:22.994735073Z I0308 10:44:22.994674       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:44:23.183273376Z I0308 10:44:23.183174       1 request.go:645] Throttling request took 1.141009574s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:44:24.405254037Z I0308 10:44:24.405007       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:44:25.390676047Z I0308 10:44:25.390623       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:44:27.501889903Z I0308 10:44:27.501824       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:44:30.338540004Z WARNING: 2022/03/08 10:44:30 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:31.340434794Z WARNING: 2022/03/08 10:44:31 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:31.460634452Z E0308 10:44:31.460575       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:31.514165314Z WARNING: 2022/03/08 10:44:31 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:31.559809960Z I0308 10:44:31.559530       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:44:32.517713202Z WARNING: 2022/03/08 10:44:32 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:32.929131777Z WARNING: 2022/03/08 10:44:32 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:34.263998271Z WARNING: 2022/03/08 10:44:34 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:35.849564168Z WARNING: 2022/03/08 10:44:35 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:37.235692324Z WARNING: 2022/03/08 10:44:37 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:39.426114577Z WARNING: 2022/03/08 10:44:39 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:41.387324567Z WARNING: 2022/03/08 10:44:41 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:43.946654809Z I0308 10:44:43.946606       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:44:46.459757147Z WARNING: 2022/03/08 10:44:46 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:46.513787156Z I0308 10:44:46.513711       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:44:46.519367286Z WARNING: 2022/03/08 10:44:46 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:47.339380605Z WARNING: 2022/03/08 10:44:47 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:47.521188961Z WARNING: 2022/03/08 10:44:47 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:48.342964662Z WARNING: 2022/03/08 10:44:48 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:48.994979292Z WARNING: 2022/03/08 10:44:48 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:49.828137170Z WARNING: 2022/03/08 10:44:49 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:50.379355360Z WARNING: 2022/03/08 10:44:50 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:51.380413491Z WARNING: 2022/03/08 10:44:51 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:51.905261157Z WARNING: 2022/03/08 10:44:51 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:51.953202797Z E0308 10:44:51.953130       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:53.061186814Z WARNING: 2022/03/08 10:44:53 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:55.336981137Z WARNING: 2022/03/08 10:44:55 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:55.432238629Z WARNING: 2022/03/08 10:44:55 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:56.435448352Z WARNING: 2022/03/08 10:44:56 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:57.432740332Z WARNING: 2022/03/08 10:44:57 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:57.584645650Z WARNING: 2022/03/08 10:44:57 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:57.598041123Z WARNING: 2022/03/08 10:44:57 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:57.600455204Z E0308 10:44:57.600412       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:44:58.592084730Z WARNING: 2022/03/08 10:44:58 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:44:58.602259920Z WARNING: 2022/03/08 10:44:58 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:00.061747081Z WARNING: 2022/03/08 10:45:00 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:00.421127750Z WARNING: 2022/03/08 10:45:00 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:01.519498410Z I0308 10:45:01.519371       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:01.524343050Z WARNING: 2022/03/08 10:45:01 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:02.388050047Z WARNING: 2022/03/08 10:45:02 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:02.525558442Z WARNING: 2022/03/08 10:45:02 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:03.486454837Z WARNING: 2022/03/08 10:45:03 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:03.905231650Z WARNING: 2022/03/08 10:45:03 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:06.677829842Z WARNING: 2022/03/08 10:45:06 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:06.721722744Z WARNING: 2022/03/08 10:45:06 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:07.692252739Z WARNING: 2022/03/08 10:45:07 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:11.111354246Z WARNING: 2022/03/08 10:45:11 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:12.665566960Z I0308 10:45:12.665511       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:12.676249471Z WARNING: 2022/03/08 10:45:12 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:12.759994778Z I0308 10:45:12.759925       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:12.760816284Z I0308 10:45:12.760747       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:12.776526382Z I0308 10:45:12.776431       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:12.776672407Z I0308 10:45:12.776645       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:12.786395843Z WARNING: 2022/03/08 10:45:12 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:12.801827736Z I0308 10:45:12.801777       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:12.802112161Z I0308 10:45:12.802081       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:12.814490378Z I0308 10:45:12.814405       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:13.799137190Z WARNING: 2022/03/08 10:45:13 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:13.829002735Z I0308 10:45:13.828922       1 request.go:645] Throttling request took 1.051184612s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T10:45:14.433252166Z I0308 10:45:14.433198       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:45:14.831478254Z I0308 10:45:14.831408       1 request.go:645] Throttling request took 1.395910036s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:15.597609651Z WARNING: 2022/03/08 10:45:15 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:16.029036432Z I0308 10:45:16.028966       1 request.go:645] Throttling request took 1.389016531s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T10:45:16.523846002Z I0308 10:45:16.523739       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:16.526784387Z WARNING: 2022/03/08 10:45:16 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:17.029028503Z I0308 10:45:17.028949       1 request.go:645] Throttling request took 1.395908678s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/restore-etcd-pod
2022-03-08T10:45:17.527846499Z WARNING: 2022/03/08 10:45:17 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:18.031655822Z I0308 10:45:18.029026       1 request.go:645] Throttling request took 1.19390108s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:45:18.441673356Z I0308 10:45:18.441624       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:45:18.654347928Z WARNING: 2022/03/08 10:45:18 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:18.863403420Z I0308 10:45:18.859550       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:18.876563460Z I0308 10:45:18.876301       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:19.040124819Z WARNING: 2022/03/08 10:45:19 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:20.029008729Z I0308 10:45:20.028942       1 request.go:645] Throttling request took 1.169119289s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:20.582743569Z WARNING: 2022/03/08 10:45:20 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:21.583892132Z WARNING: 2022/03/08 10:45:21 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:21.643880390Z I0308 10:45:21.643786       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:45:21.683853360Z WARNING: 2022/03/08 10:45:21 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:23.026012235Z WARNING: 2022/03/08 10:45:23 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:23.636165194Z I0308 10:45:23.635947       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:45:25.178576172Z WARNING: 2022/03/08 10:45:25 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:25.253256798Z WARNING: 2022/03/08 10:45:25 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:26.629292235Z WARNING: 2022/03/08 10:45:26 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:27.630275235Z WARNING: 2022/03/08 10:45:27 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:29.138977034Z WARNING: 2022/03/08 10:45:29 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:31.527030217Z I0308 10:45:31.526841       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:31.529206156Z WARNING: 2022/03/08 10:45:31 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:31.734405181Z WARNING: 2022/03/08 10:45:31 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:32.534136565Z WARNING: 2022/03/08 10:45:32 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:32.928801620Z E0308 10:45:32.928733       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:45:34.120260684Z WARNING: 2022/03/08 10:45:34 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.219:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:36.526919753Z I0308 10:45:36.526714       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0x827eb816f33178d7, Name:"master-2.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.94.219:2380"}, ClientURLs:[]string{"https://10.0.94.219:2379"}, IsLearner:false}
2022-03-08T10:45:38.419036959Z I0308 10:45:38.418946       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:45:38.433344690Z I0308 10:45:38.433288       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0x827eb816f33178d7, Name:"master-2.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.94.219:2380"}, ClientURLs:[]string{"https://10.0.94.219:2379"}, IsLearner:false}
2022-03-08T10:45:39.570043933Z I0308 10:45:39.569989       1 installer_controller.go:318] "master-2.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:45:39.570043933Z  NodeName: (string) (len=49) "master-2.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:45:39.570043933Z  CurrentRevision: (int32) 3,
2022-03-08T10:45:39.570043933Z  TargetRevision: (int32) 0,
2022-03-08T10:45:39.570043933Z  LastFailedRevision: (int32) 0,
2022-03-08T10:45:39.570043933Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:45:39.570043933Z }
2022-03-08T10:45:39.570043933Z  because static pod is ready
2022-03-08T10:45:39.590786405Z I0308 10:45:39.590709       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeCurrentRevisionChanged' Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because static pod is ready
2022-03-08T10:45:39.591398871Z I0308 10:45:39.591328       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:39.608831984Z {"level":"warn","ts":"2022-03-08T10:45:39.608Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":0,"error":"rpc error: code = Unavailable desc = transport is closing"}
2022-03-08T10:45:39.608831984Z {"level":"warn","ts":"2022-03-08T10:45:39.608Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":1,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.612879787Z I0308 10:45:39.612816       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:39.614234015Z I0308 10:45:39.614149       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3" to "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3",Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:39.624111414Z I0308 10:45:39.624053       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:39.624986394Z I0308 10:45:39.624939       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3" to "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3",Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:39.643329969Z I0308 10:45:39.643267       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:39.643654951Z I0308 10:45:39.643609       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:39.644295825Z {"level":"warn","ts":"2022-03-08T10:45:39.643Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":2,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.644295825Z {"level":"warn","ts":"2022-03-08T10:45:39.643Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":3,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.660328723Z I0308 10:45:39.659820       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:39.667739589Z {"level":"warn","ts":"2022-03-08T10:45:39.667Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":4,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.667778815Z {"level":"warn","ts":"2022-03-08T10:45:39.667Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":5,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.695380225Z {"level":"warn","ts":"2022-03-08T10:45:39.695Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":6,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.695380225Z {"level":"warn","ts":"2022-03-08T10:45:39.695Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":7,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.719652634Z {"level":"warn","ts":"2022-03-08T10:45:39.719Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":8,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.719652634Z {"level":"warn","ts":"2022-03-08T10:45:39.719Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":9,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.746958363Z {"level":"warn","ts":"2022-03-08T10:45:39.746Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":10,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.746958363Z {"level":"warn","ts":"2022-03-08T10:45:39.746Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":11,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.774046517Z {"level":"warn","ts":"2022-03-08T10:45:39.773Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":12,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.774046517Z {"level":"warn","ts":"2022-03-08T10:45:39.774Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":13,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.797992965Z {"level":"warn","ts":"2022-03-08T10:45:39.797Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":14,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.797992965Z {"level":"warn","ts":"2022-03-08T10:45:39.797Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":15,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.825714437Z {"level":"warn","ts":"2022-03-08T10:45:39.825Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":16,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.825714437Z {"level":"warn","ts":"2022-03-08T10:45:39.825Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":17,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.849342223Z {"level":"warn","ts":"2022-03-08T10:45:39.849Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":18,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.849342223Z {"level":"warn","ts":"2022-03-08T10:45:39.849Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":19,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.876853868Z {"level":"warn","ts":"2022-03-08T10:45:39.876Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":20,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.876853868Z {"level":"warn","ts":"2022-03-08T10:45:39.876Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":21,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.901527205Z {"level":"warn","ts":"2022-03-08T10:45:39.901Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":22,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.901527205Z {"level":"warn","ts":"2022-03-08T10:45:39.901Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":23,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.928799760Z {"level":"warn","ts":"2022-03-08T10:45:39.928Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":24,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.928880403Z {"level":"warn","ts":"2022-03-08T10:45:39.928Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":25,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.953709010Z {"level":"warn","ts":"2022-03-08T10:45:39.953Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":26,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.953709010Z {"level":"warn","ts":"2022-03-08T10:45:39.953Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":27,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.978984355Z {"level":"warn","ts":"2022-03-08T10:45:39.978Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":28,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:39.978984355Z {"level":"warn","ts":"2022-03-08T10:45:39.978Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":29,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.003934674Z {"level":"warn","ts":"2022-03-08T10:45:40.003Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":30,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.003934674Z {"level":"warn","ts":"2022-03-08T10:45:40.003Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":31,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.031186076Z {"level":"warn","ts":"2022-03-08T10:45:40.031Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":32,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.031230685Z {"level":"warn","ts":"2022-03-08T10:45:40.031Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":33,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.056496924Z {"level":"warn","ts":"2022-03-08T10:45:40.056Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":34,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.056544492Z {"level":"warn","ts":"2022-03-08T10:45:40.056Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":35,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.081907197Z {"level":"warn","ts":"2022-03-08T10:45:40.081Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":36,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.081907197Z {"level":"warn","ts":"2022-03-08T10:45:40.081Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":37,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.107309143Z {"level":"warn","ts":"2022-03-08T10:45:40.107Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":38,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.107362393Z {"level":"warn","ts":"2022-03-08T10:45:40.107Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":39,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.131021108Z {"level":"warn","ts":"2022-03-08T10:45:40.130Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":40,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.131021108Z {"level":"warn","ts":"2022-03-08T10:45:40.130Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":41,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.155959275Z {"level":"warn","ts":"2022-03-08T10:45:40.155Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":42,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.155959275Z {"level":"warn","ts":"2022-03-08T10:45:40.155Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":43,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.180814134Z {"level":"warn","ts":"2022-03-08T10:45:40.180Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":44,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.180814134Z {"level":"warn","ts":"2022-03-08T10:45:40.180Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":45,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.206406921Z {"level":"warn","ts":"2022-03-08T10:45:40.206Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":46,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.206406921Z {"level":"warn","ts":"2022-03-08T10:45:40.206Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":47,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.232442674Z {"level":"warn","ts":"2022-03-08T10:45:40.232Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":48,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.232442674Z {"level":"warn","ts":"2022-03-08T10:45:40.232Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":49,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.262128313Z {"level":"warn","ts":"2022-03-08T10:45:40.259Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":50,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.262128313Z {"level":"warn","ts":"2022-03-08T10:45:40.259Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":51,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.284816141Z {"level":"warn","ts":"2022-03-08T10:45:40.284Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":52,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.284816141Z {"level":"warn","ts":"2022-03-08T10:45:40.284Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":53,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.311408389Z {"level":"warn","ts":"2022-03-08T10:45:40.311Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":54,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.311408389Z {"level":"warn","ts":"2022-03-08T10:45:40.311Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":55,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.336005806Z {"level":"warn","ts":"2022-03-08T10:45:40.335Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":56,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.336005806Z {"level":"warn","ts":"2022-03-08T10:45:40.335Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":57,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.364304382Z {"level":"warn","ts":"2022-03-08T10:45:40.363Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":58,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.364304382Z {"level":"warn","ts":"2022-03-08T10:45:40.363Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":59,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.387903393Z {"level":"warn","ts":"2022-03-08T10:45:40.387Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":60,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.387903393Z {"level":"warn","ts":"2022-03-08T10:45:40.387Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":61,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.411439609Z {"level":"warn","ts":"2022-03-08T10:45:40.411Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":62,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.411517662Z {"level":"warn","ts":"2022-03-08T10:45:40.411Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":63,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.437942554Z {"level":"warn","ts":"2022-03-08T10:45:40.437Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":64,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.437942554Z {"level":"warn","ts":"2022-03-08T10:45:40.437Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":65,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.461242546Z {"level":"warn","ts":"2022-03-08T10:45:40.461Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":66,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.461242546Z {"level":"warn","ts":"2022-03-08T10:45:40.461Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":67,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.486454264Z {"level":"warn","ts":"2022-03-08T10:45:40.486Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":68,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.486454264Z {"level":"warn","ts":"2022-03-08T10:45:40.486Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":69,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.511774690Z {"level":"warn","ts":"2022-03-08T10:45:40.511Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":70,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.511774690Z {"level":"warn","ts":"2022-03-08T10:45:40.511Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":71,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.539562206Z {"level":"warn","ts":"2022-03-08T10:45:40.539Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":72,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.539562206Z {"level":"warn","ts":"2022-03-08T10:45:40.539Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":73,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.566737782Z {"level":"warn","ts":"2022-03-08T10:45:40.566Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":74,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.566737782Z {"level":"warn","ts":"2022-03-08T10:45:40.566Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":75,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.593376769Z {"level":"warn","ts":"2022-03-08T10:45:40.593Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":76,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.593376769Z {"level":"warn","ts":"2022-03-08T10:45:40.593Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":77,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.618741258Z {"level":"warn","ts":"2022-03-08T10:45:40.618Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":78,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.618741258Z {"level":"warn","ts":"2022-03-08T10:45:40.618Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":79,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.644782471Z {"level":"warn","ts":"2022-03-08T10:45:40.644Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":80,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.644782471Z {"level":"warn","ts":"2022-03-08T10:45:40.644Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":81,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.668930698Z {"level":"warn","ts":"2022-03-08T10:45:40.668Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":82,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.668930698Z {"level":"warn","ts":"2022-03-08T10:45:40.668Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":83,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.694994838Z {"level":"warn","ts":"2022-03-08T10:45:40.693Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":84,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.694994838Z {"level":"warn","ts":"2022-03-08T10:45:40.693Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":85,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.719625925Z {"level":"warn","ts":"2022-03-08T10:45:40.719Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":86,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.719625925Z {"level":"warn","ts":"2022-03-08T10:45:40.719Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":87,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.743098905Z {"level":"warn","ts":"2022-03-08T10:45:40.742Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":88,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.743098905Z {"level":"warn","ts":"2022-03-08T10:45:40.743Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":89,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.767771555Z {"level":"warn","ts":"2022-03-08T10:45:40.767Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":90,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.767771555Z {"level":"warn","ts":"2022-03-08T10:45:40.767Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":91,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.780033824Z I0308 10:45:40.779920       1 request.go:645] Throttling request took 1.18297283s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:40.792925259Z {"level":"warn","ts":"2022-03-08T10:45:40.792Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":92,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.793034254Z {"level":"warn","ts":"2022-03-08T10:45:40.793Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":93,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.820222776Z {"level":"warn","ts":"2022-03-08T10:45:40.820Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":94,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.820377005Z {"level":"warn","ts":"2022-03-08T10:45:40.820Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":95,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.846172191Z {"level":"warn","ts":"2022-03-08T10:45:40.845Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":96,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.846172191Z {"level":"warn","ts":"2022-03-08T10:45:40.846Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":97,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.873508203Z {"level":"warn","ts":"2022-03-08T10:45:40.873Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":98,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.873508203Z {"level":"warn","ts":"2022-03-08T10:45:40.873Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-2841c53c-21f4-4763-931d-9f98a1deb4b8/10.0.94.219:2379","attempt":99,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
2022-03-08T10:45:40.894033476Z E0308 10:45:40.893959       1 base_controller.go:250] "ClusterMemberController" controller failed to sync "key", err: rpc error: code = Canceled desc = grpc: the client connection is closing
2022-03-08T10:45:40.902145753Z I0308 10:45:40.902045       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"ClusterMemberControllerDegraded: rpc error: code = Canceled desc = grpc: the client connection is closing\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:40.962260947Z I0308 10:45:40.962167       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "ClusterMemberControllerDegraded: rpc error: code = Canceled desc = grpc: the client connection is closing\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:40.981672433Z I0308 10:45:40.981621       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:45:41.000767754Z I0308 10:45:41.000457       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "ClusterMemberControllerDegraded: rpc error: code = Canceled desc = grpc: the client connection is closing\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:45:41.980014966Z I0308 10:45:41.979873       1 request.go:645] Throttling request took 1.396038632s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T10:45:42.207048349Z I0308 10:45:42.206992       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
2022-03-08T10:45:43.180015656Z I0308 10:45:43.179930       1 request.go:645] Throttling request took 1.395105233s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:44.379963845Z I0308 10:45:44.379875       1 request.go:645] Throttling request took 1.193401179s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:45.579928556Z I0308 10:45:45.579852       1 request.go:645] Throttling request took 1.186655542s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:46.584730720Z I0308 10:45:46.584653       1 installer_controller.go:350] node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest and needs new revision 3
2022-03-08T10:45:46.584779182Z I0308 10:45:46.584733       1 installer_controller.go:359] "master-0.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:45:46.584779182Z  NodeName: (string) (len=49) "master-0.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:45:46.584779182Z  CurrentRevision: (int32) 2,
2022-03-08T10:45:46.584779182Z  TargetRevision: (int32) 3,
2022-03-08T10:45:46.584779182Z  LastFailedRevision: (int32) 0,
2022-03-08T10:45:46.584779182Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:45:46.584779182Z }
2022-03-08T10:45:46.604651345Z I0308 10:45:46.604560       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeTargetRevisionChanged' Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
2022-03-08T10:45:47.779958927Z I0308 10:45:47.779857       1 request.go:645] Throttling request took 1.175830393s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:45:48.779979757Z I0308 10:45:48.779892       1 request.go:645] Throttling request took 1.192182265s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-pod
2022-03-08T10:45:48.995458236Z I0308 10:45:48.995363       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
2022-03-08T10:45:49.979982539Z I0308 10:45:49.979875       1 request.go:645] Throttling request took 1.191331413s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/restore-etcd-pod
2022-03-08T10:45:50.186290997Z I0308 10:45:50.186182       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:45:51.786131379Z I0308 10:45:51.785930       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:45:52.800614781Z I0308 10:45:52.800553       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:45:54.337717694Z I0308 10:45:54.337663       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:45:55.385435565Z I0308 10:45:55.385362       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:45:56.421223024Z WARNING: 2022/03/08 10:45:56 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:56.759360133Z WARNING: 2022/03/08 10:45:56 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:56.772044622Z WARNING: 2022/03/08 10:45:56 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:56.836593366Z WARNING: 2022/03/08 10:45:56 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:57.185092328Z I0308 10:45:57.185015       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:45:57.583973864Z WARNING: 2022/03/08 10:45:57 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:57.594767997Z WARNING: 2022/03/08 10:45:57 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:57.604392717Z E0308 10:45:57.604320       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:45:57.772698822Z WARNING: 2022/03/08 10:45:57 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:58.584254457Z WARNING: 2022/03/08 10:45:58 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:58.595567902Z WARNING: 2022/03/08 10:45:58 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:45:59.415532500Z WARNING: 2022/03/08 10:45:59 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:00.264893322Z WARNING: 2022/03/08 10:46:00 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:00.356788116Z WARNING: 2022/03/08 10:46:00 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:01.601525399Z WARNING: 2022/03/08 10:46:01 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:02.829082473Z WARNING: 2022/03/08 10:46:02 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:03.039329716Z WARNING: 2022/03/08 10:46:03 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:05.619821684Z WARNING: 2022/03/08 10:46:05 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:06.321619751Z WARNING: 2022/03/08 10:46:06 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:06.565128065Z WARNING: 2022/03/08 10:46:06 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:07.267923313Z I0308 10:46:07.267840       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:46:11.772521713Z I0308 10:46:11.772471       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:11.776737838Z WARNING: 2022/03/08 10:46:11 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:11.801175493Z WARNING: 2022/03/08 10:46:11 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:12.448603803Z WARNING: 2022/03/08 10:46:12 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:12.614407979Z I0308 10:46:12.614342       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:12.635544343Z WARNING: 2022/03/08 10:46:12 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:12.643829344Z I0308 10:46:12.643784       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:46:12.643894803Z I0308 10:46:12.643823       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:12.673116773Z I0308 10:46:12.673037       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:46:12.777360387Z WARNING: 2022/03/08 10:46:12 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:13.420107311Z I0308 10:46:13.420011       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:46:13.636662003Z WARNING: 2022/03/08 10:46:13 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:13.812815024Z I0308 10:46:13.812720       1 request.go:645] Throttling request took 1.139055554s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/restore-etcd-pod
2022-03-08T10:46:14.503390335Z WARNING: 2022/03/08 10:46:14 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:15.006608749Z WARNING: 2022/03/08 10:46:15 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:16.007326580Z WARNING: 2022/03/08 10:46:16 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:16.618920824Z I0308 10:46:16.618863       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:46:16.764223824Z WARNING: 2022/03/08 10:46:16 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:17.637138044Z WARNING: 2022/03/08 10:46:17 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:19.749882270Z WARNING: 2022/03/08 10:46:19 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:21.030878031Z WARNING: 2022/03/08 10:46:21 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:23.497632070Z WARNING: 2022/03/08 10:46:23 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:26.776673584Z I0308 10:46:26.776615       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:26.777912805Z WARNING: 2022/03/08 10:46:26 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:26.789870378Z WARNING: 2022/03/08 10:46:26 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:26.859748721Z WARNING: 2022/03/08 10:46:26 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:27.790570481Z WARNING: 2022/03/08 10:46:27 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:27.860309888Z WARNING: 2022/03/08 10:46:27 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:29.368316431Z WARNING: 2022/03/08 10:46:29 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:29.373114041Z WARNING: 2022/03/08 10:46:29 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:29.889221695Z WARNING: 2022/03/08 10:46:29 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:30.889806583Z WARNING: 2022/03/08 10:46:30 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:31.517179154Z WARNING: 2022/03/08 10:46:31 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:32.385711916Z WARNING: 2022/03/08 10:46:32 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:32.386162646Z I0308 10:46:32.386125       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:32.405680806Z I0308 10:46:32.405612       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:46:32.709646861Z I0308 10:46:32.709347       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:46:33.386326196Z WARNING: 2022/03/08 10:46:33 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:33.482396403Z I0308 10:46:33.482287       1 request.go:645] Throttling request took 1.119984753s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:34.682360973Z I0308 10:46:34.682271       1 request.go:645] Throttling request took 1.587860702s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T10:46:34.883020522Z WARNING: 2022/03/08 10:46:34 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:35.059910255Z WARNING: 2022/03/08 10:46:35 grpc: addrConn.createTransport failed to connect to {https://10.0.94.119:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.94.119:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:35.882377907Z I0308 10:46:35.882313       1 request.go:645] Throttling request took 1.389764052s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-pod
2022-03-08T10:46:36.882475566Z I0308 10:46:36.882324       1 request.go:645] Throttling request took 1.388731999s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:46:37.088643848Z I0308 10:46:37.088556       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:46:39.105612819Z I0308 10:46:39.105532       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:46:35 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:39.115349519Z I0308 10:46:39.115269       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:46:35 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:46:39.287810013Z I0308 10:46:39.287710       1 installer_controller.go:331] "master-0.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:46:40.282370202Z I0308 10:46:40.282274       1 request.go:645] Throttling request took 1.173878824s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/installer-sa
2022-03-08T10:46:41.283603519Z I0308 10:46:41.283499       1 request.go:645] Throttling request took 1.196051037s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:41.791624263Z I0308 10:46:41.789897       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:42.482434731Z I0308 10:46:42.482305       1 request.go:645] Throttling request took 1.396149919s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/services/etcd
2022-03-08T10:46:42.925160852Z I0308 10:46:42.925022       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:42.941969726Z I0308 10:46:42.941894       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:46:35 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:46:43.087546794Z I0308 10:46:43.087436       1 installer_controller.go:318] "master-0.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:46:43.087546794Z  NodeName: (string) (len=49) "master-0.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:46:43.087546794Z  CurrentRevision: (int32) 3,
2022-03-08T10:46:43.087546794Z  TargetRevision: (int32) 0,
2022-03-08T10:46:43.087546794Z  LastFailedRevision: (int32) 0,
2022-03-08T10:46:43.087546794Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:46:43.087546794Z }
2022-03-08T10:46:43.087546794Z  because static pod is ready
2022-03-08T10:46:43.105998878Z I0308 10:46:43.105930       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeCurrentRevisionChanged' Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because static pod is ready
2022-03-08T10:46:43.106451511Z I0308 10:46:43.106366       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:43.143124871Z I0308 10:46:43.143035       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3" to "NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3",Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
2022-03-08T10:46:44.082371519Z I0308 10:46:44.082266       1 request.go:645] Throttling request took 1.156251602s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T10:46:45.082362817Z I0308 10:46:45.082277       1 request.go:645] Throttling request took 1.38439591s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:46.082407682Z I0308 10:46:46.082284       1 request.go:645] Throttling request took 1.394234025s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-pod
2022-03-08T10:46:48.896345451Z I0308 10:46:48.895978       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
2022-03-08T10:46:49.486703871Z I0308 10:46:49.486622       1 installer_controller.go:350] node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest and needs new revision 3
2022-03-08T10:46:49.486703871Z I0308 10:46:49.486680       1 installer_controller.go:359] "master-1.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:46:49.486703871Z  NodeName: (string) (len=49) "master-1.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:46:49.486703871Z  CurrentRevision: (int32) 2,
2022-03-08T10:46:49.486703871Z  TargetRevision: (int32) 3,
2022-03-08T10:46:49.486703871Z  LastFailedRevision: (int32) 0,
2022-03-08T10:46:49.486703871Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:46:49.486703871Z }
2022-03-08T10:46:49.576175577Z I0308 10:46:49.576098       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeTargetRevisionChanged' Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
2022-03-08T10:46:50.682378200Z I0308 10:46:50.682300       1 request.go:645] Throttling request took 1.170593884s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T10:46:51.682388013Z I0308 10:46:51.682321       1 request.go:645] Throttling request took 1.59530521s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:52.300357070Z I0308 10:46:52.300283       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
2022-03-08T10:46:52.882398605Z I0308 10:46:52.882319       1 request.go:645] Throttling request took 1.195297772s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:46:53.486676161Z I0308 10:46:53.486582       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:46:53.882460212Z I0308 10:46:53.882396       1 request.go:645] Throttling request took 1.192561694s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:46:55.286815123Z I0308 10:46:55.286740       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:46:56.687588187Z I0308 10:46:56.687486       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:46:57.648680474Z I0308 10:46:57.648573       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:57.672557307Z I0308 10:46:57.672441       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:46:57.672994439Z I0308 10:46:57.672959       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:46:57.696338411Z I0308 10:46:57.696285       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available"
2022-03-08T10:46:58.100871689Z E0308 10:46:58.100806       1 base_controller.go:250] "FSyncController" controller failed to sync "key", err: unable to retrieve prometheus-k8 bearer token
2022-03-08T10:46:58.682906497Z I0308 10:46:58.682814       1 request.go:645] Throttling request took 1.039107143s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/config
2022-03-08T10:46:59.286578424Z I0308 10:46:59.286454       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because 
2022-03-08T10:46:59.753587308Z WARNING: 2022/03/08 10:46:59 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:46:59.884762871Z I0308 10:46:59.882950       1 request.go:645] Throttling request took 1.5566746s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/installer-sa
2022-03-08T10:47:00.020757534Z WARNING: 2022/03/08 10:47:00 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:00.754157336Z WARNING: 2022/03/08 10:47:00 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:01.021620105Z WARNING: 2022/03/08 10:47:01 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:01.082347457Z I0308 10:47:01.082267       1 request.go:645] Throttling request took 1.391024401s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:02.128535310Z WARNING: 2022/03/08 10:47:02 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:02.282481087Z I0308 10:47:02.282337       1 request.go:645] Throttling request took 1.194951045s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:02.703300994Z WARNING: 2022/03/08 10:47:02 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:03.087985703Z I0308 10:47:03.087856       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:47:03.164576153Z WARNING: 2022/03/08 10:47:03 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:04.163861865Z WARNING: 2022/03/08 10:47:04 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:05.130838472Z WARNING: 2022/03/08 10:47:05 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:05.497285527Z I0308 10:47:05.497205       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:47:05.679295224Z WARNING: 2022/03/08 10:47:05 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:06.177716413Z WARNING: 2022/03/08 10:47:06 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:07.178779292Z WARNING: 2022/03/08 10:47:07 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:08.470497715Z WARNING: 2022/03/08 10:47:08 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:08.646177142Z WARNING: 2022/03/08 10:47:08 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:11.000870551Z I0308 10:47:11.000802       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because waiting for static pod of revision 3, found 2
2022-03-08T10:47:11.034642969Z WARNING: 2022/03/08 10:47:11 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:14.150393267Z WARNING: 2022/03/08 10:47:14 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:14.855741804Z WARNING: 2022/03/08 10:47:14 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:15.020134669Z I0308 10:47:15.019992       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:15.024505500Z WARNING: 2022/03/08 10:47:15 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:16.025430179Z WARNING: 2022/03/08 10:47:16 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:17.365080515Z WARNING: 2022/03/08 10:47:17 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:19.646636195Z WARNING: 2022/03/08 10:47:19 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:20.521448932Z WARNING: 2022/03/08 10:47:20 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:24.064116273Z WARNING: 2022/03/08 10:47:24 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:29.347811022Z WARNING: 2022/03/08 10:47:29 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:30.024357729Z I0308 10:47:30.024277       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'UnhealthyEtcdMember' unhealthy members: master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:30.026786111Z WARNING: 2022/03/08 10:47:30 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:30.038168238Z WARNING: 2022/03/08 10:47:30 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:30.324321378Z WARNING: 2022/03/08 10:47:30 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:31.039952254Z WARNING: 2022/03/08 10:47:31 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:31.325004739Z WARNING: 2022/03/08 10:47:31 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:32.697034548Z WARNING: 2022/03/08 10:47:32 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:32.937807902Z WARNING: 2022/03/08 10:47:32 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:34.364299521Z I0308 10:47:34.364228       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:47:34.423437153Z I0308 10:47:34.423391       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:47:34.435602801Z I0308 10:47:34.435528       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:47:35.359502787Z WARNING: 2022/03/08 10:47:35 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:35.542441222Z I0308 10:47:35.542136       1 request.go:645] Throttling request took 1.12668328s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:35.709075631Z WARNING: 2022/03/08 10:47:35 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.0.95.76:2379: connect: connection refused". Reconnecting...
2022-03-08T10:47:36.353315966Z I0308 10:47:36.352904       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:47:36.742148364Z I0308 10:47:36.742020       1 request.go:645] Throttling request took 1.175738388s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:37.941533208Z I0308 10:47:37.941429       1 request.go:645] Throttling request took 1.502775631s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/installer-sa
2022-03-08T10:47:38.193106569Z I0308 10:47:38.192864       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:47:38.222967392Z I0308 10:47:38.221826       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:47:38.943361799Z I0308 10:47:38.941726       1 request.go:645] Throttling request took 1.386579268s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/config
2022-03-08T10:47:40.140833711Z I0308 10:47:40.140745       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:40.141918171Z I0308 10:47:40.141804       1 request.go:645] Throttling request took 1.51317328s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:47:40.148637592Z WARNING: 2022/03/08 10:47:40 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2022-03-08T10:47:40.184426128Z I0308 10:47:40.184368       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:40.309406716Z I0308 10:47:40.306748       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:40.330780361Z I0308 10:47:40.330675       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:40.745852750Z I0308 10:47:40.745797       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:47:41.246967207Z I0308 10:47:41.246896       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:41.341530947Z I0308 10:47:41.341430       1 request.go:645] Throttling request took 1.395830858s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:41.536214492Z WARNING: 2022/03/08 10:47:41 grpc: addrConn.createTransport failed to connect to {https://10.0.94.219:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2022-03-08T10:47:41.537582054Z I0308 10:47:41.537508       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:42.176896087Z I0308 10:47:42.175283       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:47:37 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:47:42.195219755Z I0308 10:47:42.194990       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:47:37 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:47:42.214166452Z I0308 10:47:42.213221       1 clustermembercontroller.go:170] skipping unready pod "etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com" because it is already an etcd member: &etcdserverpb.Member{ID:0xb983e031d3964a8, Name:"master-1.case03116926.lab.upshift.rdu2.redhat.com", PeerURLs:[]string{"https://10.0.95.76:2380"}, ClientURLs:[]string{"https://10.0.95.76:2379"}, IsLearner:false}
2022-03-08T10:47:42.541484579Z I0308 10:47:42.541362       1 request.go:645] Throttling request took 1.146538236s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:43.541513668Z I0308 10:47:43.541381       1 request.go:645] Throttling request took 1.372943167s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:44.541510806Z I0308 10:47:44.541433       1 request.go:645] Throttling request took 1.393878226s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:47:44.545228696Z I0308 10:47:44.545170       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 3, but has not made progress because static pod is pending
2022-03-08T10:47:45.742409539Z I0308 10:47:45.742278       1 request.go:645] Throttling request took 1.108435653s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:47:46.192600117Z I0308 10:47:46.192548       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:44:19Z","message":"NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:47:46.233243490Z I0308 10:47:46.229889       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:47:37 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
2022-03-08T10:47:46.746499170Z I0308 10:47:46.744161       1 request.go:645] Throttling request took 1.364347965s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/config
2022-03-08T10:47:47.942091683Z I0308 10:47:47.941287       1 request.go:645] Throttling request took 1.378026628s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T10:47:48.347569568Z I0308 10:47:48.347508       1 installer_controller.go:318] "master-1.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:47:48.347569568Z  NodeName: (string) (len=49) "master-1.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:47:48.347569568Z  CurrentRevision: (int32) 3,
2022-03-08T10:47:48.347569568Z  TargetRevision: (int32) 0,
2022-03-08T10:47:48.347569568Z  LastFailedRevision: (int32) 0,
2022-03-08T10:47:48.347569568Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:47:48.347569568Z }
2022-03-08T10:47:48.347569568Z  because static pod is ready
2022-03-08T10:47:48.367980987Z I0308 10:47:48.367929       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeCurrentRevisionChanged' Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because static pod is ready
2022-03-08T10:47:48.372622977Z I0308 10:47:48.372569       1 status_controller.go:172] clusteroperator/etcd diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:45Z","message":"NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:47:48Z","message":"NodeInstallerProgressing: 3 nodes are at revision 3\nEtcdMembersProgressing: No unstarted etcd members found","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:40:07Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:47Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:47:48.398896638Z I0308 10:47:48.398669       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/etcd changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 3\nEtcdMembersProgressing: No unstarted etcd members found"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available"
2022-03-08T10:47:49.542365280Z I0308 10:47:49.542316       1 request.go:645] Throttling request took 1.174589812s, request: PUT:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/revision-status-3
2022-03-08T10:47:49.564477651Z I0308 10:47:49.564407       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/revision-status-3 -n openshift-etcd:
2022-03-08T10:47:49.564477651Z cause by changes in data.status
2022-03-08T10:47:50.580487588Z I0308 10:47:50.575121       1 request.go:645] Throttling request took 1.427832031s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/config
2022-03-08T10:47:53.965607302Z I0308 10:47:53.965543       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"8520abe5-2170-45b9-b29e-f3df07b4ad2b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
2022-03-08T10:47:58.742766860Z I0308 10:47:58.741324       1 request.go:645] Throttling request took 1.041637134s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T10:47:59.742343308Z I0308 10:47:59.742222       1 request.go:645] Throttling request took 1.166740739s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T10:53:58.785200080Z I0308 10:53:58.785043       1 request.go:645] Throttling request took 1.093401503s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T10:53:59.985009073Z I0308 10:53:59.984937       1 request.go:645] Throttling request took 1.19494334s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T10:58:36.976791605Z I0308 10:58:36.976676       1 request.go:645] Throttling request took 1.156831021s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T11:03:58.789958939Z I0308 11:03:58.789797       1 request.go:645] Throttling request took 1.094697637s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T11:03:59.791483614Z I0308 11:03:59.790052       1 request.go:645] Throttling request took 1.39047609s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T11:04:36.616072353Z I0308 11:04:36.615744       1 request.go:645] Throttling request took 1.03977064s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts?resourceVersion=23007
2022-03-08T11:04:37.815267815Z I0308 11:04:37.815196       1 request.go:645] Throttling request took 1.195620572s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/installer-sa
2022-03-08T11:11:15.892608028Z I0308 11:11:15.891987       1 request.go:645] Throttling request took 1.056276656s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T11:11:17.092227326Z I0308 11:11:17.092160       1 request.go:645] Throttling request took 1.394457177s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T11:13:58.799040383Z I0308 11:13:58.798911       1 request.go:645] Throttling request took 1.020397771s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T11:13:59.799032193Z I0308 11:13:59.798961       1 request.go:645] Throttling request took 1.386622634s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T11:23:58.813398159Z I0308 11:23:58.813239       1 request.go:645] Throttling request took 1.030659916s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T11:23:59.813371920Z I0308 11:23:59.813300       1 request.go:645] Throttling request took 1.393079509s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T11:33:58.817733812Z I0308 11:33:58.817603       1 request.go:645] Throttling request took 1.033299888s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T11:33:59.818032787Z I0308 11:33:59.817944       1 request.go:645] Throttling request took 1.390995897s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T11:43:58.818026321Z I0308 11:43:58.817868       1 request.go:645] Throttling request took 1.030591909s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-scripts
2022-03-08T11:43:59.817970509Z I0308 11:43:59.817913       1 request.go:645] Throttling request took 1.394769912s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/etcd-sa
2022-03-08T11:53:58.449037235Z WARNING: 2022/03/08 11:53:58 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2022-03-08T11:53:58.819933266Z I0308 11:53:58.819839       1 request.go:645] Throttling request took 1.074121681s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T11:53:59.819963432Z I0308 11:53:59.819892       1 request.go:645] Throttling request took 1.389110651s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T12:03:58.820333777Z I0308 12:03:58.820222       1 request.go:645] Throttling request took 1.027220691s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/serviceaccounts/installer-sa
2022-03-08T12:08:58.508658140Z WARNING: 2022/03/08 12:08:58 grpc: addrConn.createTransport failed to connect to {https://10.0.95.76:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2022-03-08T12:13:58.838799579Z I0308 12:13:58.838671       1 request.go:645] Throttling request took 1.089448504s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T12:14:00.038697289Z I0308 12:14:00.038638       1 request.go:645] Throttling request took 1.589490095s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T12:23:58.820949675Z I0308 12:23:58.820863       1 request.go:645] Throttling request took 1.069540374s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T12:23:59.820937476Z I0308 12:23:59.820874       1 request.go:645] Throttling request took 1.387332871s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-pod
2022-03-08T12:33:58.821285846Z I0308 12:33:58.821172       1 request.go:645] Throttling request took 1.067849948s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T12:34:00.021148563Z I0308 12:34:00.021088       1 request.go:645] Throttling request took 1.591655649s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/etcd-pod
2022-03-08T12:43:58.821544536Z I0308 12:43:58.821439       1 request.go:645] Throttling request took 1.066746012s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T12:43:59.821613231Z I0308 12:43:59.821492       1 request.go:645] Throttling request took 1.592739399s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T12:53:58.822421330Z I0308 12:53:58.822274       1 request.go:645] Throttling request took 1.065223116s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T12:53:59.823109862Z I0308 12:53:59.823015       1 request.go:645] Throttling request took 1.595225371s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T13:03:58.822766786Z I0308 13:03:58.822655       1 request.go:645] Throttling request took 1.062695182s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T13:04:00.022270829Z I0308 13:04:00.022179       1 request.go:645] Throttling request took 1.583646082s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T13:13:58.822435807Z I0308 13:13:58.822308       1 request.go:645] Throttling request took 1.056595975s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T13:14:00.022489707Z I0308 13:14:00.022417       1 request.go:645] Throttling request took 1.593845039s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/configmaps/config
2022-03-08T13:23:58.822566893Z I0308 13:23:58.822389       1 request.go:645] Throttling request took 1.0554958s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T13:24:00.022448097Z I0308 13:24:00.022343       1 request.go:645] Throttling request took 1.594071468s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods?labelSelector=app%3Dinstaller
2022-03-08T13:33:58.826239157Z I0308 13:33:58.825897       1 request.go:645] Throttling request took 1.057238968s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T13:33:59.826105627Z I0308 13:33:59.826009       1 request.go:645] Throttling request took 1.589028498s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T13:43:58.843554220Z I0308 13:43:58.843421       1 request.go:645] Throttling request took 1.072703262s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd
2022-03-08T13:43:59.843510148Z I0308 13:43:59.843419       1 request.go:645] Throttling request took 1.592708948s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-etcd/pods/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
