2022-03-08T10:40:01.733231437Z I0308 10:40:01.733080       1 cmd.go:200] Using service-serving-cert provided certificates
2022-03-08T10:40:01.733936108Z I0308 10:40:01.733904       1 observer_polling.go:159] Starting file observer
2022-03-08T10:40:01.758799866Z I0308 10:40:01.758735       1 builder.go:238] kube-apiserver-operator version v4.0.0-alpha.0-1136-g0a8f1f99-0a8f1f99f98b4637cfb5a22c61c92264687da859
2022-03-08T10:40:02.150213853Z I0308 10:40:02.147134       1 leaderelection.go:243] attempting to acquire leader lease  openshift-kube-apiserver-operator/kube-apiserver-operator-lock...
2022-03-08T10:40:02.150554598Z W0308 10:40:02.150528       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
2022-03-08T10:40:02.150589379Z W0308 10:40:02.150579       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159226       1 secure_serving.go:197] Serving securely on [::]:8443
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159282       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159297       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159318       1 dynamic_serving_content.go:130] Starting serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159345       1 tlsconfig.go:240] Starting DynamicServingCertificateController
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159426       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159434       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159447       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2022-03-08T10:40:02.161521048Z I0308 10:40:02.159453       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2022-03-08T10:40:02.161521048Z I0308 10:40:02.160602       1 leaderelection.go:253] successfully acquired lease openshift-kube-apiserver-operator/kube-apiserver-operator-lock
2022-03-08T10:40:02.161680859Z I0308 10:40:02.161616       1 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator-lock", UID:"d0ca80be-c43e-4831-8d50-0e1008098938", APIVersion:"v1", ResourceVersion:"9905", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' kube-apiserver-operator-c74696d8d-gsntz_12b45cd8-f925-45de-b00e-7572ff5be42c became leader
2022-03-08T10:40:02.167434339Z E0308 10:40:02.166917       1 static_resource_controller.go:118] missing informer for namespace "kube-system"; no dynamic wiring added, time-based only.
2022-03-08T10:40:02.186179597Z I0308 10:40:02.184167       1 base_controller.go:66] Waiting for caches to sync for ConnectivityCheckController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185591       1 certrotationcontroller.go:654] Starting CertRotation
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185600       1 certrotationcontroller.go:619] Waiting for CertRotation
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185617       1 base_controller.go:66] Waiting for caches to sync for EncryptionConditionController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185627       1 base_controller.go:66] Waiting for caches to sync for FeatureUpgradeableController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185636       1 base_controller.go:66] Waiting for caches to sync for WebhookAuthenticatorUpgradeableController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185644       1 base_controller.go:66] Waiting for caches to sync for CertRotationTimeUpgradeableController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185654       1 termination_observer.go:145] Starting TerminationObserver
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185663       1 base_controller.go:66] Waiting for caches to sync for EventWatchController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185671       1 base_controller.go:66] Waiting for caches to sync for BoundSATokenSignerController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185684       1 base_controller.go:66] Waiting for caches to sync for RemoveStaleConditionsController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185721       1 base_controller.go:66] Waiting for caches to sync for StatusSyncer_kube-apiserver
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185753       1 base_controller.go:66] Waiting for caches to sync for EncryptionKeyController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185762       1 base_controller.go:66] Waiting for caches to sync for EncryptionStateController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185770       1 base_controller.go:66] Waiting for caches to sync for EncryptionPruneController
2022-03-08T10:40:02.186179597Z I0308 10:40:02.185780       1 base_controller.go:66] Waiting for caches to sync for EncryptionMigrationController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.187856       1 base_controller.go:66] Waiting for caches to sync for LoggingSyncer
2022-03-08T10:40:02.192699250Z I0308 10:40:02.187890       1 base_controller.go:66] Waiting for caches to sync for RevisionController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.187917       1 base_controller.go:66] Waiting for caches to sync for ResourceSyncController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.187984       1 base_controller.go:66] Waiting for caches to sync for TargetConfigController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.187998       1 base_controller.go:66] Waiting for caches to sync for NodeKubeconfigController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188012       1 base_controller.go:66] Waiting for caches to sync for ConfigObserver
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188024       1 base_controller.go:66] Waiting for caches to sync for InstallerController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188037       1 base_controller.go:66] Waiting for caches to sync for InstallerStateController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188048       1 base_controller.go:66] Waiting for caches to sync for StaticPodStateController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188062       1 base_controller.go:66] Waiting for caches to sync for PruneController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188071       1 base_controller.go:66] Waiting for caches to sync for NodeController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188225       1 base_controller.go:66] Waiting for caches to sync for StaticResourceController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188248       1 base_controller.go:66] Waiting for caches to sync for UnsupportedConfigOverridesController
2022-03-08T10:40:02.192699250Z I0308 10:40:02.188358       1 base_controller.go:66] Waiting for caches to sync for StaticResourceController
2022-03-08T10:40:02.262872472Z I0308 10:40:02.259676       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
2022-03-08T10:40:02.262872472Z I0308 10:40:02.259694       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
2022-03-08T10:40:02.262872472Z I0308 10:40:02.259705       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
2022-03-08T10:40:02.297011025Z I0308 10:40:02.296965       1 base_controller.go:72] Caches are synced for UnsupportedConfigOverridesController 
2022-03-08T10:40:02.297011025Z I0308 10:40:02.296996       1 base_controller.go:109] Starting #1 worker of UnsupportedConfigOverridesController controller ...
2022-03-08T10:40:02.297766632Z I0308 10:40:02.297745       1 base_controller.go:72] Caches are synced for LoggingSyncer 
2022-03-08T10:40:02.297766632Z I0308 10:40:02.297760       1 base_controller.go:109] Starting #1 worker of LoggingSyncer controller ...
2022-03-08T10:40:02.298097178Z I0308 10:40:02.298072       1 servicehostname.go:40] syncing servicenetwork hostnames: [172.30.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local openshift openshift.default openshift.default.svc openshift.default.svc.cluster.local]
2022-03-08T10:40:02.298114425Z I0308 10:40:02.298099       1 externalloadbalancer.go:27] syncing external loadbalancer hostnames: api.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:02.298114425Z I0308 10:40:02.298105       1 internalloadbalancer.go:27] syncing internal loadbalancer hostnames: api-int.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:02.298114425Z I0308 10:40:02.298109       1 certrotationcontroller.go:637] Finished waiting for CertRotation
2022-03-08T10:40:02.298174232Z I0308 10:40:02.298164       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.298174232Z I0308 10:40:02.298171       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:02.298181337Z I0308 10:40:02.298175       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:02.298222606Z I0308 10:40:02.298211       1 base_controller.go:72] Caches are synced for FeatureUpgradeableController 
2022-03-08T10:40:02.298222606Z I0308 10:40:02.298219       1 base_controller.go:109] Starting #1 worker of FeatureUpgradeableController controller ...
2022-03-08T10:40:02.298498471Z I0308 10:40:02.298488       1 base_controller.go:72] Caches are synced for WebhookAuthenticatorUpgradeableController 
2022-03-08T10:40:02.298498471Z I0308 10:40:02.298496       1 base_controller.go:109] Starting #1 worker of WebhookAuthenticatorUpgradeableController controller ...
2022-03-08T10:40:02.298636275Z I0308 10:40:02.298620       1 externalloadbalancer.go:27] syncing external loadbalancer hostnames: api.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:02.298718493Z I0308 10:40:02.298711       1 base_controller.go:72] Caches are synced for RemoveStaleConditionsController 
2022-03-08T10:40:02.298742892Z I0308 10:40:02.298734       1 base_controller.go:109] Starting #1 worker of RemoveStaleConditionsController controller ...
2022-03-08T10:40:02.298777085Z I0308 10:40:02.298766       1 base_controller.go:72] Caches are synced for PruneController 
2022-03-08T10:40:02.298793779Z I0308 10:40:02.298787       1 base_controller.go:109] Starting #1 worker of PruneController controller ...
2022-03-08T10:40:02.299332456Z I0308 10:40:02.299143       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299410514Z I0308 10:40:02.299392       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299428922Z I0308 10:40:02.299418       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299452339Z I0308 10:40:02.299443       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299473426Z I0308 10:40:02.299457       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299480450Z I0308 10:40:02.299474       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299498253Z I0308 10:40:02.299489       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299529999Z I0308 10:40:02.299519       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299545787Z I0308 10:40:02.299537       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299553663Z I0308 10:40:02.299549       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299570260Z I0308 10:40:02.299562       1 base_controller.go:66] Waiting for caches to sync for CertRotationController
2022-03-08T10:40:02.299589669Z I0308 10:40:02.299511       1 servicehostname.go:40] syncing servicenetwork hostnames: [172.30.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local openshift openshift.default openshift.default.svc openshift.default.svc.cluster.local]
2022-03-08T10:40:02.300231653Z I0308 10:40:02.300169       1 prune_controller.go:134] no revision IDs currently eligible to prune
2022-03-08T10:40:02.300266514Z I0308 10:40:02.300257       1 prune_controller.go:286] No excluded revisions to prune, skipping
2022-03-08T10:40:02.313140683Z I0308 10:40:02.313096       1 prune_controller.go:134] no revision IDs currently eligible to prune
2022-03-08T10:40:02.313140683Z I0308 10:40:02.313117       1 prune_controller.go:286] No excluded revisions to prune, skipping
2022-03-08T10:40:02.384402399Z I0308 10:40:02.384357       1 base_controller.go:72] Caches are synced for ConnectivityCheckController 
2022-03-08T10:40:02.384453429Z I0308 10:40:02.384442       1 base_controller.go:109] Starting #1 worker of ConnectivityCheckController controller ...
2022-03-08T10:40:02.385797394Z I0308 10:40:02.385754       1 base_controller.go:72] Caches are synced for StatusSyncer_kube-apiserver 
2022-03-08T10:40:02.385886540Z I0308 10:40:02.385833       1 base_controller.go:109] Starting #1 worker of StatusSyncer_kube-apiserver controller ...
2022-03-08T10:40:02.588215728Z I0308 10:40:02.588093       1 base_controller.go:72] Caches are synced for NodeController 
2022-03-08T10:40:02.588215728Z I0308 10:40:02.588118       1 base_controller.go:109] Starting #1 worker of NodeController controller ...
2022-03-08T10:40:02.785931051Z I0308 10:40:02.785797       1 base_controller.go:72] Caches are synced for CertRotationTimeUpgradeableController 
2022-03-08T10:40:02.785931051Z I0308 10:40:02.785823       1 base_controller.go:109] Starting #1 worker of CertRotationTimeUpgradeableController controller ...
2022-03-08T10:40:03.378582945Z I0308 10:40:03.378470       1 request.go:645] Throttling request took 1.191709283s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-config-managed/secrets?limit=500&resourceVersion=0
2022-03-08T10:40:03.399722345Z I0308 10:40:03.399638       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.399722345Z I0308 10:40:03.399666       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.399722345Z I0308 10:40:03.399690       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.399722345Z I0308 10:40:03.399694       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.586358857Z I0308 10:40:03.585813       1 base_controller.go:72] Caches are synced for BoundSATokenSignerController 
2022-03-08T10:40:03.586358857Z I0308 10:40:03.585928       1 base_controller.go:109] Starting #1 worker of BoundSATokenSignerController controller ...
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588045       1 base_controller.go:72] Caches are synced for NodeKubeconfigController 
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588068       1 base_controller.go:109] Starting #1 worker of NodeKubeconfigController controller ...
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588079       1 base_controller.go:72] Caches are synced for RevisionController 
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588093       1 base_controller.go:109] Starting #1 worker of RevisionController controller ...
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588093       1 base_controller.go:72] Caches are synced for ResourceSyncController 
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588106       1 base_controller.go:109] Starting #1 worker of ResourceSyncController controller ...
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588109       1 base_controller.go:72] Caches are synced for ConfigObserver 
2022-03-08T10:40:03.588123042Z I0308 10:40:03.588113       1 base_controller.go:109] Starting #1 worker of ConfigObserver controller ...
2022-03-08T10:40:03.588814235Z I0308 10:40:03.588766       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RevisionTriggered' new revision 2 triggered by "configmap \"kube-apiserver-pod-1\" not found"
2022-03-08T10:40:03.599415466Z I0308 10:40:03.599310       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.599415466Z I0308 10:40:03.599338       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.599483261Z I0308 10:40:03.599459       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.599483261Z I0308 10:40:03.599468       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.599483261Z I0308 10:40:03.599469       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.599493087Z I0308 10:40:03.599485       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600530       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600553       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600573       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600578       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600614       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600620       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600636       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600640       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600655       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600660       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600675       1 base_controller.go:72] Caches are synced for CertRotationController 
2022-03-08T10:40:03.600815623Z I0308 10:40:03.600679       1 base_controller.go:109] Starting #1 worker of CertRotationController controller ...
2022-03-08T10:40:03.807786901Z E0308 10:40:03.807696       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:03.813141019Z E0308 10:40:03.813077       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:03.813809946Z E0308 10:40:03.813782       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:03.817910086Z I0308 10:40:03.817807       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:03.835279090Z E0308 10:40:03.835168       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:03.839749615Z I0308 10:40:03.839647       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
2022-03-08T10:40:03.841354898Z E0308 10:40:03.841138       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:03.843698057Z I0308 10:40:03.843647       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:03.876155325Z E0308 10:40:03.876067       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:03.886002625Z I0308 10:40:03.885926       1 base_controller.go:72] Caches are synced for EventWatchController 
2022-03-08T10:40:03.886002625Z I0308 10:40:03.885967       1 base_controller.go:109] Starting #1 worker of EventWatchController controller ...
2022-03-08T10:40:04.037542838Z E0308 10:40:04.037464       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:04.188243940Z I0308 10:40:04.188164       1 base_controller.go:72] Caches are synced for TargetConfigController 
2022-03-08T10:40:04.188243940Z I0308 10:40:04.188199       1 base_controller.go:109] Starting #1 worker of TargetConfigController controller ...
2022-03-08T10:40:04.188368714Z I0308 10:40:04.188306       1 base_controller.go:72] Caches are synced for StaticResourceController 
2022-03-08T10:40:04.188379992Z I0308 10:40:04.188367       1 base_controller.go:109] Starting #1 worker of StaticResourceController controller ...
2022-03-08T10:40:04.188468983Z I0308 10:40:04.188428       1 base_controller.go:72] Caches are synced for StaticResourceController 
2022-03-08T10:40:04.188468983Z I0308 10:40:04.188442       1 base_controller.go:109] Starting #1 worker of StaticResourceController controller ...
2022-03-08T10:40:04.358670499Z E0308 10:40:04.358562       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:04.385908438Z I0308 10:40:04.385743       1 base_controller.go:72] Caches are synced for EncryptionConditionController 
2022-03-08T10:40:04.385908438Z I0308 10:40:04.385789       1 base_controller.go:109] Starting #1 worker of EncryptionConditionController controller ...
2022-03-08T10:40:04.385908438Z I0308 10:40:04.385885       1 base_controller.go:72] Caches are synced for EncryptionStateController 
2022-03-08T10:40:04.385908438Z I0308 10:40:04.385899       1 base_controller.go:109] Starting #1 worker of EncryptionStateController controller ...
2022-03-08T10:40:04.385990854Z I0308 10:40:04.385885       1 base_controller.go:72] Caches are synced for EncryptionMigrationController 
2022-03-08T10:40:04.385990854Z I0308 10:40:04.385914       1 base_controller.go:72] Caches are synced for EncryptionKeyController 
2022-03-08T10:40:04.385990854Z I0308 10:40:04.385926       1 base_controller.go:109] Starting #1 worker of EncryptionKeyController controller ...
2022-03-08T10:40:04.385990854Z I0308 10:40:04.385900       1 base_controller.go:72] Caches are synced for EncryptionPruneController 
2022-03-08T10:40:04.385990854Z I0308 10:40:04.385946       1 base_controller.go:109] Starting #1 worker of EncryptionPruneController controller ...
2022-03-08T10:40:04.386029439Z I0308 10:40:04.385917       1 base_controller.go:109] Starting #1 worker of EncryptionMigrationController controller ...
2022-03-08T10:40:04.388213941Z I0308 10:40:04.388149       1 base_controller.go:72] Caches are synced for StaticPodStateController 
2022-03-08T10:40:04.388213941Z I0308 10:40:04.388161       1 base_controller.go:109] Starting #1 worker of StaticPodStateController controller ...
2022-03-08T10:40:04.388213941Z I0308 10:40:04.388167       1 base_controller.go:72] Caches are synced for InstallerStateController 
2022-03-08T10:40:04.388213941Z I0308 10:40:04.388179       1 base_controller.go:109] Starting #1 worker of InstallerStateController controller ...
2022-03-08T10:40:04.388264540Z I0308 10:40:04.388224       1 base_controller.go:72] Caches are synced for InstallerController 
2022-03-08T10:40:04.388264540Z I0308 10:40:04.388230       1 base_controller.go:109] Starting #1 worker of InstallerController controller ...
2022-03-08T10:40:04.388991275Z I0308 10:40:04.388940       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:04.778592733Z I0308 10:40:04.778482       1 request.go:645] Throttling request took 1.189910587s, request: POST:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps
2022-03-08T10:40:04.999908649Z E0308 10:40:04.999758       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:05.039228214Z E0308 10:40:05.039135       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:05.041481670Z I0308 10:40:05.040447       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-client-ca -n openshift-config-managed because it was missing
2022-03-08T10:40:05.041481670Z I0308 10:40:05.040978       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-pod-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:05.042590077Z I0308 10:40:05.042510       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
2022-03-08T10:40:05.046254604Z E0308 10:40:05.046212       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:05.780974641Z I0308 10:40:05.779307       1 request.go:645] Throttling request took 1.392772618s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:05.805272464Z E0308 10:40:05.804927       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:06.281100765Z E0308 10:40:06.281031       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:06.789463690Z I0308 10:40:06.789333       1 request.go:645] Throttling request took 1.586918864s, request: PUT:https://172.30.0.1:443/apis/operator.openshift.io/v1/kubeapiservers/cluster/status
2022-03-08T10:40:07.188433563Z I0308 10:40:07.188340       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/config-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:07.193193730Z E0308 10:40:07.193114       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:07.195981969Z E0308 10:40:07.195917       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:07.409820255Z E0308 10:40:07.404511       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:07.594456073Z E0308 10:40:07.594393       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:07.789476776Z I0308 10:40:07.789344       1 request.go:645] Throttling request took 1.321651772s, request: PUT:https://172.30.0.1:443/apis/operator.openshift.io/v1/kubeapiservers/cluster/status
2022-03-08T10:40:07.797497250Z E0308 10:40:07.797406       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:07.996068958Z E0308 10:40:07.996004       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:08.595007980Z E0308 10:40:08.594914       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:08.785958100Z I0308 10:40:08.785893       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-cert-syncer-kubeconfig-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:08.787435428Z E0308 10:40:08.786593       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:08.789462750Z I0308 10:40:08.789404       1 request.go:645] Throttling request took 1.737637191s, request: PUT:https://172.30.0.1:443/apis/operator.openshift.io/v1/kubeapiservers/cluster/status
2022-03-08T10:40:09.201710162Z E0308 10:40:09.201631       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:09.978465665Z I0308 10:40:09.978404       1 request.go:645] Throttling request took 1.787798904s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:10.181297514Z I0308 10:40:10.181234       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:10.595809512Z E0308 10:40:10.595735       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:10.597729183Z I0308 10:40:10.597666       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:10.978612369Z I0308 10:40:10.978472       1 request.go:645] Throttling request took 1.596620237s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/kube-apiserver-server-ca
2022-03-08T10:40:11.005493377Z E0308 10:40:11.005422       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:11.795124443Z E0308 10:40:11.795062       1 base_controller.go:250] "StaticResourceController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:11.989433112Z I0308 10:40:11.989347       1 request.go:645] Throttling request took 1.538415559s, request: PUT:https://172.30.0.1:443/apis/operator.openshift.io/v1/kubeapiservers/cluster/status
2022-03-08T10:40:12.187029845Z E0308 10:40:12.186979       1 base_controller.go:250] "NodeKubeconfigController" controller failed to sync "key", err: "secret/node-kubeconfigs": configmap "kube-apiserver-server-ca" not found
2022-03-08T10:40:12.591226863Z I0308 10:40:12.590276       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-server-ca -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:12.989454191Z I0308 10:40:12.989349       1 request.go:645] Throttling request took 1.342907095s, request: PUT:https://172.30.0.1:443/apis/operator.openshift.io/v1/kubeapiservers/cluster/status
2022-03-08T10:40:13.195676658Z E0308 10:40:13.195579       1 base_controller.go:250] "EncryptionKeyController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:13.608790624Z E0308 10:40:13.608684       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:13.985588775Z I0308 10:40:13.985514       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/bound-sa-token-signing-certs-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:14.178488727Z I0308 10:40:14.178381       1 request.go:645] Throttling request took 1.796541322s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:15.178578729Z I0308 10:40:15.178480       1 request.go:645] Throttling request took 1.99658475s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:15.195022339Z E0308 10:40:15.194945       1 base_controller.go:250] "StaticResourceController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:15.581601932Z I0308 10:40:15.581489       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:15.594653617Z E0308 10:40:15.594547       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:15.595619319Z I0308 10:40:15.595547       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:15.792483729Z I0308 10:40:15.791372       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-server-ca -n openshift-config-managed because it was missing
2022-03-08T10:40:15.987481856Z I0308 10:40:15.987346       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/etcd-serving-ca-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:16.178811324Z I0308 10:40:16.178570       1 request.go:645] Throttling request took 1.995538521s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:16.419617484Z E0308 10:40:16.419481       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:16.420582511Z I0308 10:40:16.420532       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:16.426937995Z I0308 10:40:16.426027       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:16.439637652Z I0308 10:40:16.436472       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
2022-03-08T10:40:16.763878005Z I0308 10:40:16.760542       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/node-kubeconfigs -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:17.205644806Z E0308 10:40:17.205586       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:17.207802882Z I0308 10:40:17.207669       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:17.378483618Z I0308 10:40:17.378419       1 request.go:645] Throttling request took 2.182949965s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver
2022-03-08T10:40:17.449308283Z I0308 10:40:17.449264       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:17.469916028Z I0308 10:40:17.469283       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
2022-03-08T10:40:17.796944525Z E0308 10:40:17.796895       1 base_controller.go:250] "CertRotationController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:17.990366055Z I0308 10:40:17.989804       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-server-ca-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:18.004905487Z E0308 10:40:18.004827       1 base_controller.go:250] "ConfigObserver" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:18.578530898Z I0308 10:40:18.578462       1 request.go:645] Throttling request took 1.768425429s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:18.678940746Z E0308 10:40:18.678894       1 base_controller.go:250] "EncryptionKeyController" controller failed to sync "key", err: Operation cannot be fulfilled on kubeapiservers.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:40:19.578539682Z I0308 10:40:19.578470       1 request.go:645] Throttling request took 1.993366591s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa
2022-03-08T10:40:19.812572359Z E0308 10:40:19.812509       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:19.816710848Z E0308 10:40:19.816641       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:19.817063077Z I0308 10:40:19.817033       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:19.818048560Z E0308 10:40:19.818013       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:19.818140718Z I0308 10:40:19.818119       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:19.818778270Z I0308 10:40:19.818643       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:19.861442645Z I0308 10:40:19.861375       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
2022-03-08T10:40:19.899856223Z E0308 10:40:19.897390       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:19.899856223Z I0308 10:40:19.897465       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:19.988636081Z I0308 10:40:19.988573       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kubelet-serving-ca-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:20.440781438Z E0308 10:40:20.440720       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:20.441096501Z I0308 10:40:20.441064       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:20.538912689Z E0308 10:40:20.538867       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:20.538945093Z I0308 10:40:20.538912       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:20.778527142Z I0308 10:40:20.778418       1 request.go:645] Throttling request took 2.087415373s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:20.810426849Z I0308 10:40:20.810340       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:20.810624971Z E0308 10:40:20.810553       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:21.402222843Z E0308 10:40:21.402172       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:21.402514501Z I0308 10:40:21.402485       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:21.778580341Z I0308 10:40:21.778451       1 request.go:645] Throttling request took 2.190206358s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa
2022-03-08T10:40:21.981829308Z I0308 10:40:21.981735       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:22.186193326Z I0308 10:40:22.185993       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/sa-token-signing-certs-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:22.778693406Z I0308 10:40:22.778588       1 request.go:645] Throttling request took 2.182823678s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:23.100003523Z I0308 10:40:23.099899       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RequiredInstallerResourcesMissing' configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
2022-03-08T10:40:23.100507143Z E0308 10:40:23.100468       1 base_controller.go:250] "InstallerController" controller failed to sync "key", err: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]
2022-03-08T10:40:23.978877761Z I0308 10:40:23.978397       1 request.go:645] Throttling request took 2.193429861s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa
2022-03-08T10:40:24.389962908Z I0308 10:40:24.389877       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapCreated' Created ConfigMap/kube-apiserver-audit-policies-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:24.979205079Z I0308 10:40:24.978677       1 request.go:645] Throttling request took 2.195351782s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:26.178474056Z I0308 10:40:26.178363       1 request.go:645] Throttling request took 1.996912091s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:26.386918095Z I0308 10:40:26.386104       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/etcd-client-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:27.378637045Z I0308 10:40:27.378559       1 request.go:645] Throttling request took 1.99707603s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:28.218414338Z I0308 10:40:28.218344       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:28.382616014Z I0308 10:40:28.380943       1 request.go:645] Throttling request took 1.99481733s, request: POST:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets
2022-03-08T10:40:28.405084108Z I0308 10:40:28.405023       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/kubelet-client-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:29.578503567Z I0308 10:40:29.578435       1 request.go:645] Throttling request took 1.995024079s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:30.200960209Z I0308 10:40:30.199035       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/localhost-recovery-serving-certkey-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:30.578863713Z I0308 10:40:30.578784       1 request.go:645] Throttling request took 1.701134266s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/trusted-ca-bundle
2022-03-08T10:40:31.778410657Z I0308 10:40:31.778361       1 request.go:645] Throttling request took 1.78865638s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs
2022-03-08T10:40:31.991105962Z I0308 10:40:31.991043       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/localhost-recovery-client-token-2 -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:32.009753177Z I0308 10:40:32.009705       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RevisionCreate' Revision 1 created because configmap "kube-apiserver-pod-1" not found
2022-03-08T10:40:32.778460229Z I0308 10:40:32.778387       1 request.go:645] Throttling request took 1.787745191s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:33.978504074Z I0308 10:40:33.978442       1 request.go:645] Throttling request took 1.969239488s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:33.982585229Z I0308 10:40:33.982539       1 installer_controller.go:350] node master-1.case03116926.lab.upshift.rdu2.redhat.com static pod not found and needs new revision 2
2022-03-08T10:40:33.982690550Z I0308 10:40:33.982677       1 installer_controller.go:359] "master-1.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:40:33.982690550Z  NodeName: (string) (len=49) "master-1.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:40:33.982690550Z  CurrentRevision: (int32) 0,
2022-03-08T10:40:33.982690550Z  TargetRevision: (int32) 2,
2022-03-08T10:40:33.982690550Z  LastFailedRevision: (int32) 0,
2022-03-08T10:40:33.982690550Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:40:33.982690550Z }
2022-03-08T10:40:33.996153001Z I0308 10:40:33.996089       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeTargetRevisionChanged' Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-1.case03116926.lab.upshift.rdu2.redhat.com static pod not found
2022-03-08T10:40:34.002114978Z I0308 10:40:34.002066       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:34.017039808Z I0308 10:40:34.016984       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
2022-03-08T10:40:34.059392833Z I0308 10:40:34.059341       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:34.071553837Z I0308 10:40:34.071456       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
2022-03-08T10:40:35.180122963Z I0308 10:40:35.178338       1 request.go:645] Throttling request took 2.19703136s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:36.178610382Z I0308 10:40:36.178450       1 request.go:645] Throttling request took 2.119495215s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:37.378475678Z I0308 10:40:37.378402       1 request.go:645] Throttling request took 2.194627021s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:38.181470463Z I0308 10:40:38.181385       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:38.391480758Z I0308 10:40:38.391390       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
2022-03-08T10:40:38.578450002Z I0308 10:40:38.578399       1 request.go:645] Throttling request took 1.994738057s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:39.578534219Z I0308 10:40:39.578428       1 request.go:645] Throttling request took 1.995832138s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:40.578920559Z I0308 10:40:40.578865       1 request.go:645] Throttling request took 2.187689565s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:40.587580320Z I0308 10:40:40.587499       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because 
2022-03-08T10:40:41.579027567Z I0308 10:40:41.578940       1 request.go:645] Throttling request took 2.196730696s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:42.778482793Z I0308 10:40:42.778411       1 request.go:645] Throttling request took 2.189571014s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:43.778874862Z I0308 10:40:43.778817       1 request.go:645] Throttling request took 2.197125894s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:44.381752622Z I0308 10:40:44.381696       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:44.979319605Z I0308 10:40:44.978733       1 request.go:645] Throttling request took 2.396232488s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dopenshift-kube-apiserver
2022-03-08T10:40:45.182056140Z I0308 10:40:45.182011       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because 
2022-03-08T10:40:46.178465413Z I0308 10:40:46.178361       1 request.go:645] Throttling request took 2.396676616s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:47.178479419Z I0308 10:40:47.178409       1 request.go:645] Throttling request took 2.196196561s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dopenshift-kube-apiserver
2022-03-08T10:40:48.178462964Z I0308 10:40:48.178400       1 request.go:645] Throttling request took 2.189800003s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:48.981813821Z I0308 10:40:48.981717       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because 
2022-03-08T10:40:49.378535895Z I0308 10:40:49.378425       1 request.go:645] Throttling request took 1.597414416s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/localhost-recovery-client-token
2022-03-08T10:40:50.381307319Z I0308 10:40:50.381168       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'MissingVersion' no image found for operand pod
2022-03-08T10:40:50.578415261Z I0308 10:40:50.578359       1 request.go:645] Throttling request took 1.396281289s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:51.578500959Z I0308 10:40:51.578397       1 request.go:645] Throttling request took 1.3961142s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:52.778423483Z I0308 10:40:52.778367       1 request.go:645] Throttling request took 1.620048354s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dinstaller
2022-03-08T10:40:53.779125546Z I0308 10:40:53.778800       1 request.go:645] Throttling request took 1.99664987s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:54.780761529Z I0308 10:40:54.779919       1 request.go:645] Throttling request took 2.198491074s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:55.979981277Z I0308 10:40:55.978300       1 request.go:645] Throttling request took 2.197172915s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:40:55.988239371Z I0308 10:40:55.988175       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/kube-apiserver version "kube-apiserver" changed from "" to "1.19.16"
2022-03-08T10:40:55.995071257Z I0308 10:40:55.988284       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/kube-apiserver version "operator" changed from "" to "4.6.52"
2022-03-08T10:40:56.058983500Z I0308 10:40:56.041550       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"versions":[{"name":"raw-internal","version":"4.6.52"},{"name":"kube-apiserver","version":"1.19.16"},{"name":"operator","version":"4.6.52"}]}}
2022-03-08T10:40:56.058983500Z I0308 10:40:56.057064       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}],"versions":[{"name":"raw-internal","version":"4.6.52"},{"name":"operator","version":"4.6.52"},{"name":"kube-apiserver","version":"1.19.16"}]}}
2022-03-08T10:40:56.058983500Z I0308 10:40:56.057591       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"kube-apiserver" "1.19.16"} {"operator" "4.6.52"}]
2022-03-08T10:40:56.058983500Z I0308 10:40:56.057620       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/kube-apiserver version "operator" changed from "" to "4.6.52"
2022-03-08T10:40:56.058983500Z I0308 10:40:56.057632       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/kube-apiserver version "kube-apiserver" changed from "" to "1.19.16"
2022-03-08T10:40:56.061432596Z I0308 10:40:56.061397       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:38:52Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:40:56.061693689Z I0308 10:40:56.061666       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready",status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"operator" "4.6.52"} {"kube-apiserver" "1.19.16"}]
2022-03-08T10:40:56.082916467Z I0308 10:40:56.075607       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
2022-03-08T10:40:56.781928066Z I0308 10:40:56.781824       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because static pod is pending
2022-03-08T10:40:56.978539810Z I0308 10:40:56.978465       1 request.go:645] Throttling request took 1.543628781s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:40:57.642941939Z I0308 10:40:57.635923       1 tlsconfig.go:178] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "admin-kubeconfig-signer" [] issuer="<self>" (2022-03-08 10:10:26 +0000 UTC to 2032-03-05 10:10:26 +0000 UTC (now=2022-03-08 10:40:57.635903191 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.635956       1 tlsconfig.go:178] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-signer" [] issuer="<self>" (2022-03-08 10:10:30 +0000 UTC to 2022-03-09 10:10:30 +0000 UTC (now=2022-03-08 10:40:57.635947338 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.635973       1 tlsconfig.go:178] loaded client CA [2/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-control-plane-signer" [] issuer="<self>" (2022-03-08 10:10:30 +0000 UTC to 2023-03-08 10:10:30 +0000 UTC (now=2022-03-08 10:40:57.635966011 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.635987       1 tlsconfig.go:178] loaded client CA [3/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-apiserver-to-kubelet-signer" [] issuer="<self>" (2022-03-08 10:10:31 +0000 UTC to 2023-03-08 10:10:31 +0000 UTC (now=2022-03-08 10:40:57.635980655 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.636004       1 tlsconfig.go:178] loaded client CA [4/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="<self>" (2022-03-08 10:10:27 +0000 UTC to 2032-03-05 10:10:27 +0000 UTC (now=2022-03-08 10:40:57.635997862 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.636022       1 tlsconfig.go:178] loaded client CA [5/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-csr-signer_@1646735926" [] issuer="kubelet-signer" (2022-03-08 10:38:45 +0000 UTC to 2022-03-09 10:10:30 +0000 UTC (now=2022-03-08 10:40:57.636014699 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.636038       1 tlsconfig.go:178] loaded client CA [6/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "openshift-kube-apiserver-operator_node-system-admin-signer@1646735931" [] issuer="<self>" (2022-03-08 10:38:50 +0000 UTC to 2023-03-08 10:38:51 +0000 UTC (now=2022-03-08 10:40:57.636030024 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.636054       1 tlsconfig.go:178] loaded client CA [7/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="<self>" (2022-03-08 10:10:28 +0000 UTC to 2022-03-09 10:10:28 +0000 UTC (now=2022-03-08 10:40:57.636046534 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.636330       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"]: "metrics.openshift-kube-apiserver-operator.svc" [serving] validServingFor=[metrics.openshift-kube-apiserver-operator.svc,metrics.openshift-kube-apiserver-operator.svc.cluster.local] issuer="openshift-service-serving-signer@1646735940" (2022-03-08 10:39:17 +0000 UTC to 2024-03-07 10:39:18 +0000 UTC (now=2022-03-08 10:40:57.636317332 +0000 UTC))
2022-03-08T10:40:57.642941939Z I0308 10:40:57.636590       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1646736002" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1646736001" (2022-03-08 09:40:01 +0000 UTC to 2023-03-08 09:40:01 +0000 UTC (now=2022-03-08 10:40:57.636578232 +0000 UTC))
2022-03-08T10:40:57.981453075Z I0308 10:40:57.980525       1 request.go:645] Throttling request took 1.943133571s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs
2022-03-08T10:40:59.178474051Z I0308 10:40:59.178376       1 request.go:645] Throttling request took 2.595866023s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:00.378487664Z I0308 10:41:00.378371       1 request.go:645] Throttling request took 1.991905581s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/services/apiserver
2022-03-08T10:41:01.381043581Z I0308 10:41:01.380935       1 request.go:645] Throttling request took 1.99709492s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:02.578741005Z I0308 10:41:02.578660       1 request.go:645] Throttling request took 1.795647759s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:03.182140960Z I0308 10:41:03.182087       1 installer_controller.go:331] "master-1.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because static pod is pending
2022-03-08T10:41:03.779767795Z I0308 10:41:03.778937       1 request.go:645] Throttling request took 1.795237376s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:04.978971767Z I0308 10:41:04.978361       1 request.go:645] Throttling request took 1.591899432s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:05.978536279Z I0308 10:41:05.978426       1 request.go:645] Throttling request took 1.595635188s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:07.178389563Z I0308 10:41:07.178326       1 request.go:645] Throttling request took 2.395512799s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:08.178413421Z I0308 10:41:08.178344       1 request.go:645] Throttling request took 2.194578795s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:08.409061264Z I0308 10:41:08.409019       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:41:08Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found","reason":"StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:39:05Z","message":"StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2","reason":"StaticPods_ZeroNodesActive","status":"False","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:41:08.423529867Z I0308 10:41:08.422999       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded changed from False to True ("StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found")
2022-03-08T10:41:09.178468193Z I0308 10:41:09.178376       1 request.go:645] Throttling request took 2.195031849s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dopenshift-kube-apiserver
2022-03-08T10:41:10.378487231Z I0308 10:41:10.378411       1 request.go:645] Throttling request took 2.39570514s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver
2022-03-08T10:41:11.181509210Z I0308 10:41:11.181452       1 installer_controller.go:318] "master-1.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:41:11.181509210Z  NodeName: (string) (len=49) "master-1.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:41:11.181509210Z  CurrentRevision: (int32) 2,
2022-03-08T10:41:11.181509210Z  TargetRevision: (int32) 0,
2022-03-08T10:41:11.181509210Z  LastFailedRevision: (int32) 0,
2022-03-08T10:41:11.181509210Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:41:11.181509210Z }
2022-03-08T10:41:11.181509210Z  because static pod is ready
2022-03-08T10:41:11.239530014Z I0308 10:41:11.239477       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:41:08Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found","reason":"StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:41:11Z","message":"StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:41:11.245263313Z I0308 10:41:11.245205       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeCurrentRevisionChanged' Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because static pod is ready
2022-03-08T10:41:11.254245774Z I0308 10:41:11.254180       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2",Available changed from False to True ("StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2")
2022-03-08T10:41:11.578503948Z I0308 10:41:11.578425       1 request.go:645] Throttling request took 1.991908512s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:12.778451956Z I0308 10:41:12.778396       1 request.go:645] Throttling request took 1.79643319s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/trusted-ca-bundle
2022-03-08T10:41:13.778464486Z I0308 10:41:13.778412       1 request.go:645] Throttling request took 2.395428542s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:14.778499529Z I0308 10:41:14.778434       1 request.go:645] Throttling request took 2.352932854s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:15.397910810Z I0308 10:41:15.397757       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
2022-03-08T10:41:15.781997997Z I0308 10:41:15.781933       1 installer_controller.go:350] node master-2.case03116926.lab.upshift.rdu2.redhat.com static pod not found and needs new revision 2
2022-03-08T10:41:15.782139868Z I0308 10:41:15.782120       1 installer_controller.go:359] "master-2.case03116926.lab.upshift.rdu2.redhat.com" moving to (v1.NodeStatus) {
2022-03-08T10:41:15.782139868Z  NodeName: (string) (len=49) "master-2.case03116926.lab.upshift.rdu2.redhat.com",
2022-03-08T10:41:15.782139868Z  CurrentRevision: (int32) 0,
2022-03-08T10:41:15.782139868Z  TargetRevision: (int32) 2,
2022-03-08T10:41:15.782139868Z  LastFailedRevision: (int32) 0,
2022-03-08T10:41:15.782139868Z  LastFailedRevisionErrors: ([]string) <nil>
2022-03-08T10:41:15.782139868Z }
2022-03-08T10:41:15.806064244Z I0308 10:41:15.805994       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeTargetRevisionChanged' Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-2.case03116926.lab.upshift.rdu2.redhat.com static pod not found
2022-03-08T10:41:15.978536320Z I0308 10:41:15.978435       1 request.go:645] Throttling request took 2.194805451s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:17.178743037Z I0308 10:41:17.178653       1 request.go:645] Throttling request took 2.196619786s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:18.378445313Z I0308 10:41:18.378386       1 request.go:645] Throttling request took 2.573051441s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:19.378467215Z I0308 10:41:19.378405       1 request.go:645] Throttling request took 2.794367276s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:20.378520498Z I0308 10:41:20.378436       1 request.go:645] Throttling request took 2.790815941s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dinstaller
2022-03-08T10:41:20.788399491Z I0308 10:41:20.788350       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'PodCreated' Created Pod/installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
2022-03-08T10:41:21.378611675Z I0308 10:41:21.378470       1 request.go:645] Throttling request took 2.392064785s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:22.578726570Z I0308 10:41:22.578492       1 request.go:645] Throttling request took 2.392826501s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/kube-apiserver-pod
2022-03-08T10:41:23.182315582Z I0308 10:41:23.182256       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because 
2022-03-08T10:41:23.778540908Z I0308 10:41:23.778473       1 request.go:645] Throttling request took 2.195884373s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:24.978887841Z I0308 10:41:24.978443       1 request.go:645] Throttling request took 2.387548949s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/client-ca
2022-03-08T10:41:25.979097365Z I0308 10:41:25.979039       1 request.go:645] Throttling request took 2.391393269s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:27.182034673Z I0308 10:41:27.178444       1 request.go:645] Throttling request took 2.193887019s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/kube-apiserver-server-ca
2022-03-08T10:41:27.782907691Z I0308 10:41:27.782804       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because 
2022-03-08T10:41:28.378523929Z I0308 10:41:28.378393       1 request.go:645] Throttling request took 2.194880897s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:29.578482837Z I0308 10:41:29.578386       1 request.go:645] Throttling request took 1.775922588s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs
2022-03-08T10:41:30.778642313Z I0308 10:41:30.778557       1 request.go:645] Throttling request took 1.794933045s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:31.978582063Z I0308 10:41:31.978401       1 request.go:645] Throttling request took 1.583558114s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:33.178544688Z I0308 10:41:33.178371       1 request.go:645] Throttling request took 2.366147615s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa
2022-03-08T10:41:34.382701695Z I0308 10:41:34.379089       1 request.go:645] Throttling request took 2.39247615s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:34.793874389Z I0308 10:41:34.788003       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because 
2022-03-08T10:41:35.578512284Z I0308 10:41:35.578445       1 request.go:645] Throttling request took 2.196122386s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:36.778420027Z I0308 10:41:36.778365       1 request.go:645] Throttling request took 2.161414528s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs
2022-03-08T10:41:37.778558042Z I0308 10:41:37.778459       1 request.go:645] Throttling request took 1.992160337s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps/kube-apiserver-pod
2022-03-08T10:41:38.781670413Z I0308 10:41:38.778965       1 request.go:645] Throttling request took 1.949049544s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs
2022-03-08T10:41:39.978597885Z I0308 10:41:39.978505       1 request.go:645] Throttling request took 2.206742525s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dinstaller
2022-03-08T10:41:41.178408179Z I0308 10:41:41.178332       1 request.go:645] Throttling request took 2.583773189s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:42.178441035Z I0308 10:41:42.178347       1 request.go:645] Throttling request took 2.392674224s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:43.178891644Z I0308 10:41:43.178741       1 request.go:645] Throttling request took 2.366332354s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:44.179013965Z I0308 10:41:44.178937       1 request.go:645] Throttling request took 2.025671541s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:44.217091021Z I0308 10:41:44.217033       1 status_controller.go:172] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2022-03-08T10:41:08Z","message":"StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found","reason":"StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2022-03-08T10:39:49Z","message":"NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2022-03-08T10:41:11Z","message":"StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2022-03-08T10:38:51Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
2022-03-08T10:41:44.234870511Z I0308 10:41:44.231593       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found" to "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found"
2022-03-08T10:41:44.382299462Z I0308 10:41:44.382243       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because static pod is pending
2022-03-08T10:41:45.378900613Z I0308 10:41:45.378800       1 request.go:645] Throttling request took 2.388761701s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:46.578491394Z I0308 10:41:46.578363       1 request.go:645] Throttling request took 2.371627549s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:47.582798105Z I0308 10:41:47.578988       1 request.go:645] Throttling request took 2.793967191s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dopenshift-kube-apiserver
2022-03-08T10:41:48.778529151Z I0308 10:41:48.778417       1 request.go:645] Throttling request took 2.794031358s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:49.778571744Z I0308 10:41:49.778428       1 request.go:645] Throttling request took 2.473698219s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/serviceaccounts/installer-sa
2022-03-08T10:41:50.982932024Z I0308 10:41:50.982849       1 request.go:645] Throttling request took 2.198321807s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:51.783948336Z I0308 10:41:51.782348       1 installer_controller.go:331] "master-2.case03116926.lab.upshift.rdu2.redhat.com" is in transition to 2, but has not made progress because static pod is pending
2022-03-08T10:41:52.178359726Z I0308 10:41:52.178297       1 request.go:645] Throttling request took 2.183097058s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/secrets/node-kubeconfigs
2022-03-08T10:41:53.178423483Z I0308 10:41:53.178352       1 request.go:645] Throttling request took 2.189596599s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:54.178736994Z I0308 10:41:54.178674       1 request.go:645] Throttling request took 2.593299919s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:55.389913295Z I0308 10:41:55.389257       1 request.go:645] Throttling request took 2.400158042s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:56.579302316Z I0308 10:41:56.578960       1 request.go:645] Throttling request took 2.397336112s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com
2022-03-08T10:41:56.926441037Z I0308 10:41:56.925892       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RevisionTriggered' new revision 3 triggered by "secret/localhost-recovery-client-token has changed"
2022-03-08T10:41:57.787963344Z I0308 10:41:57.781019       1 request.go:645] Throttling request took 2.370937882s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:41:58.979304817Z I0308 10:41:58.979226       1 request.go:645] Throttling request took 2.053328588s, request: POST:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/configmaps
2022-03-08T10:42:00.178486632Z I0308 10:42:00.178359       1 request.go:645] Throttling request took 2.182817948s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=apiserver%3Dtrue
2022-03-08T10:42:32.983306073Z I0308 10:42:32.983242       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"dea42a4a-e8ec-46a8-82da-66aa9cb987c6", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ConfigMapCreateFailed' Failed to create ConfigMap/revision-status-3 -n openshift-kube-apiserver: Timeout: request did not complete within requested timeout 34s
2022-03-08T10:42:37.626267640Z E0308 10:42:37.626152       1 leaderelection.go:321] error retrieving resource lock openshift-kube-apiserver-operator/kube-apiserver-operator-lock: Get "https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver-operator/configmaps/kube-apiserver-operator-lock?timeout=35s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2022-03-08T10:42:37.626267640Z I0308 10:42:37.626235       1 leaderelection.go:278] failed to renew lease openshift-kube-apiserver-operator/kube-apiserver-operator-lock: timed out waiting for the condition
2022-03-08T10:42:37.626314293Z E0308 10:42:37.626279       1 leaderelection.go:297] Failed to release lock: resource name may not be empty
2022-03-08T10:42:37.626364678Z W0308 10:42:37.626339       1 leaderelection.go:75] leader election lost
