Events

Time | Namespace | Component | RelatedObject | Reason | Message
--- | --- | --- | --- | --- | ---
10:47:40 | openshift-monitoring |  | thanos-querier-5c8f998999-vrvtc | Scheduled | Successfully assigned openshift-monitoring/thanos-querier-5c8f998999-vrvtc to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-etcd-operator |  | etcd-operator-859798c7f4-kb5lq | FailedScheduling | no nodes available to schedule pods
11:18:42 | openshift-marketplace |  | certified-operators-vgmxf | Scheduled | Successfully assigned openshift-marketplace/certified-operators-vgmxf to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:28:53 | openshift-marketplace |  | certified-operators-slfsz | Scheduled | Successfully assigned openshift-marketplace/certified-operators-slfsz to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-config-operator |  | openshift-config-operator-697f744f69-g6ck2 | FailedScheduling | no nodes available to schedule pods
10:14:09 | openshift-config-operator |  | openshift-config-operator-697f744f69-g6ck2 | FailedScheduling | no nodes available to schedule pods
10:14:09 | openshift-config-operator |  | openshift-config-operator-697f744f69-g6ck2 | FailedScheduling | no nodes available to schedule pods
12:04:10 | openshift-marketplace |  | certified-operators-vp4mv | Scheduled | Successfully assigned openshift-marketplace/certified-operators-vp4mv to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:25:43 | openshift-marketplace |  | certified-operators-wkgpr | Scheduled | Successfully assigned openshift-marketplace/certified-operators-wkgpr to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:47:30 | openshift-marketplace |  | certified-operators-p45rl | Scheduled | Successfully assigned openshift-marketplace/certified-operators-p45rl to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:14:17 | openshift-image-registry |  | cluster-image-registry-operator-98c4d4975-kxn6n | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-image-registry |  | cluster-image-registry-operator-98c4d4975-kxn6n | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-image-registry |  | cluster-image-registry-operator-98c4d4975-kxn6n | FailedScheduling | no nodes available to schedule pods
13:28:58 | openshift-marketplace |  | community-operators-4l9vk | Scheduled | Successfully assigned openshift-marketplace/community-operators-4l9vk to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:04:10 | openshift-marketplace |  | community-operators-9f85q | Scheduled | Successfully assigned openshift-marketplace/community-operators-9f85q to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:25:43 | openshift-marketplace |  | community-operators-bk9gr | Scheduled | Successfully assigned openshift-marketplace/community-operators-bk9gr to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:50:26 | openshift-marketplace |  | community-operators-cr2bj | Scheduled | Successfully assigned openshift-marketplace/community-operators-cr2bj to worker-1.case03116926.lab.upshift.rdu2.redhat.com
13:18:39 | openshift-marketplace |  | community-operators-gwd9c | Scheduled | Successfully assigned openshift-marketplace/community-operators-gwd9c to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:50:05 | openshift-marketplace |  | community-operators-hs279 | Scheduled | Successfully assigned openshift-marketplace/community-operators-hs279 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:50:29 | openshift-marketplace |  | community-operators-mljlz | Scheduled | Successfully assigned openshift-marketplace/community-operators-mljlz to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:46:25 | openshift-marketplace |  | community-operators-mpl78 | Scheduled | Successfully assigned openshift-marketplace/community-operators-mpl78 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:01:09 | openshift-image-registry |  | image-registry-6cb956b4c8-nfhv5 | Scheduled | Successfully assigned openshift-image-registry/image-registry-6cb956b4c8-nfhv5 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:01:10 | openshift-image-registry |  | image-registry-7958686d94-5n487 | Scheduled | Successfully assigned openshift-image-registry/image-registry-7958686d94-5n487 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-marketplace |  | community-operators-mprwd | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:13:48 | openshift-cluster-version |  | cluster-version-operator-66d6584679-59rx2 | FailedScheduling | skip schedule deleting pod: openshift-cluster-version/cluster-version-operator-66d6584679-59rx2
10:13:29 | openshift-cluster-version |  | cluster-version-operator-66d6584679-59rx2 | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-service-ca-operator |  | service-ca-operator-6fd6454f86-5bbh8 | FailedScheduling | no nodes available to schedule pods
10:13:58 | openshift-service-ca-operator |  | service-ca-operator-6fd6454f86-5bbh8 | FailedScheduling | no nodes available to schedule pods
10:13:57 | openshift-service-ca-operator |  | service-ca-operator-6fd6454f86-5bbh8 | FailedScheduling | no nodes available to schedule pods
10:13:29 | openshift-cluster-version |  | cluster-version-operator-66d6584679-59rx2 | FailedScheduling | no nodes available to schedule pods
10:46:43 | openshift-marketplace |  | community-operators-mprwd | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-marketplace |  | community-operators-mprwd | Scheduled | Successfully assigned openshift-marketplace/community-operators-mprwd to worker-2.case03116926.lab.upshift.rdu2.redhat.com
11:07:52 | openshift-marketplace |  | community-operators-p6njw | Scheduled | Successfully assigned openshift-marketplace/community-operators-p6njw to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:01:25 | openshift-apiserver |  | apiserver-5bdfbb9984-2hx9z | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match pod anti-affinity rules.
11:01:25 | openshift-apiserver |  | apiserver-5bdfbb9984-2hx9z | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match pod anti-affinity rules.
11:02:35 | openshift-apiserver |  | apiserver-5bdfbb9984-2hx9z | Scheduled | Successfully assigned openshift-apiserver/apiserver-5bdfbb9984-2hx9z to master-2.case03116926.lab.upshift.rdu2.redhat.com
11:40:17 | openshift-marketplace |  | community-operators-pqzhm | Scheduled | Successfully assigned openshift-marketplace/community-operators-pqzhm to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:35:44 | openshift-marketplace |  | community-operators-sl84r | Scheduled | Successfully assigned openshift-marketplace/community-operators-sl84r to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:43 | openshift-marketplace |  | certified-operators-p45rl | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:33:25 | openshift-cluster-version |  | cluster-version-operator-5d86d65f6b-4fm8l | FailedScheduling | no nodes available to schedule pods
10:13:48 | openshift-cluster-version |  | cluster-version-operator-5d86d65f6b-4fm8l | FailedScheduling | no nodes available to schedule pods
10:13:48 | openshift-cluster-version |  | cluster-version-operator-5d86d65f6b-4fm8l | FailedScheduling | no nodes available to schedule pods
11:28:53 | openshift-marketplace |  | community-operators-svx4n | Scheduled | Successfully assigned openshift-marketplace/community-operators-svx4n to worker-1.case03116926.lab.upshift.rdu2.redhat.com
13:07:21 | openshift-marketplace |  | community-operators-trkt6 | Scheduled | Successfully assigned openshift-marketplace/community-operators-trkt6 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:51:25 | openshift-console |  | console-7d6f969d46-7jdhw | Scheduled | Successfully assigned openshift-console/console-7d6f969d46-7jdhw to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-marketplace |  | certified-operators-p45rl | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
11:04:11 | openshift-apiserver |  | apiserver-5bdfbb9984-6rjfg | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match pod anti-affinity rules.
11:04:11 | openshift-apiserver |  | apiserver-5bdfbb9984-6rjfg | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match pod anti-affinity rules.
11:05:25 | openshift-apiserver |  | apiserver-5bdfbb9984-6rjfg | Scheduled | Successfully assigned openshift-apiserver/apiserver-5bdfbb9984-6rjfg to master-1.case03116926.lab.upshift.rdu2.redhat.com
13:14:45 | openshift-marketplace |  | certified-operators-nh9n8 | Scheduled | Successfully assigned openshift-marketplace/certified-operators-nh9n8 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:36:09 | openshift-marketplace |  | certified-operators-j54nl | Scheduled | Successfully assigned openshift-marketplace/certified-operators-j54nl to worker-1.case03116926.lab.upshift.rdu2.redhat.com
13:36:17 | openshift-marketplace |  | certified-operators-g44p8 | Scheduled | Successfully assigned openshift-marketplace/certified-operators-g44p8 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:48:15 | openshift-console |  | console-7d6f969d46-crrrm | Scheduled | Successfully assigned openshift-console/console-7d6f969d46-crrrm to master-0.case03116926.lab.upshift.rdu2.redhat.com
13:24:55 | openshift-marketplace |  | certified-operators-fdrn2 | Scheduled | Successfully assigned openshift-marketplace/certified-operators-fdrn2 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:07:52 | openshift-marketplace |  | certified-operators-cbbm2 | Scheduled | Successfully assigned openshift-marketplace/certified-operators-cbbm2 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:39:58 | openshift-marketplace |  | certified-operators-9zcz9 | Scheduled | Successfully assigned openshift-marketplace/certified-operators-9zcz9 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:15:33 | openshift-marketplace |  | certified-operators-8wkgf | Scheduled | Successfully assigned openshift-marketplace/certified-operators-8wkgf to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:50:26 | openshift-marketplace |  | certified-operators-498r2 | Scheduled | Successfully assigned openshift-marketplace/certified-operators-498r2 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:14:47 | openshift-marketplace |  | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Scheduled | Successfully assigned openshift-marketplace/773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:57:06 | openshift-marketplace |  | community-operators-vkbm4 | Scheduled | Successfully assigned openshift-marketplace/community-operators-vkbm4 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:02:48 | openshift-apiserver |  | apiserver-5bdfbb9984-k7b4h | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match pod anti-affinity rules.
11:02:48 | openshift-apiserver |  | apiserver-5bdfbb9984-k7b4h | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't match pod anti-affinity rules.
11:03:55 | openshift-apiserver |  | apiserver-5bdfbb9984-k7b4h | Scheduled | Successfully assigned openshift-apiserver/apiserver-5bdfbb9984-k7b4h to master-0.case03116926.lab.upshift.rdu2.redhat.com
11:18:35 | openshift-marketplace |  | community-operators-wmcq5 | Scheduled | Successfully assigned openshift-marketplace/community-operators-wmcq5 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:50:39 | openshift-marketplace |  | community-operators-xc9nn | Scheduled | Successfully assigned openshift-marketplace/community-operators-xc9nn to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:14:59 | openshift-marketplace |  | community-operators-zqhdc | Scheduled | Successfully assigned openshift-marketplace/community-operators-zqhdc to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:47:37 | openshift-console |  | console-7fdcf89666-fslmv | Scheduled | Successfully assigned openshift-console/console-7fdcf89666-fslmv to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-cluster-storage-operator |  | csi-snapshot-controller-operator-76dfcb9b88-lt78x | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-cluster-storage-operator |  | csi-snapshot-controller-operator-76dfcb9b88-lt78x | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-cluster-storage-operator |  | csi-snapshot-controller-operator-76dfcb9b88-lt78x | FailedScheduling | no nodes available to schedule pods
11:16:19 | openshift-marketplace |  | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Scheduled | Successfully assigned openshift-marketplace/d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:14:14 | openshift-marketplace |  | marketplace-operator-664c8865b-d9dx9 | FailedScheduling | no nodes available to schedule pods
10:14:14 | openshift-marketplace |  | marketplace-operator-664c8865b-d9dx9 | FailedScheduling | no nodes available to schedule pods
10:50:07 | openshift-marketplace |  | certified-operators-trz5k | Scheduled | Successfully assigned openshift-marketplace/certified-operators-trz5k to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-machine-config-operator |  | machine-config-operator-65cd958fc7-b5cm6 | FailedScheduling | no nodes available to schedule pods
10:14:12 | openshift-machine-config-operator |  | machine-config-operator-65cd958fc7-b5cm6 | FailedScheduling | no nodes available to schedule pods
10:14:12 | openshift-machine-config-operator |  | machine-config-operator-65cd958fc7-b5cm6 | FailedScheduling | no nodes available to schedule pods
10:47:37 | openshift-console |  | console-7fdcf89666-hh29k | Scheduled | Successfully assigned openshift-console/console-7fdcf89666-hh29k to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:29 | openshift-machine-config-operator |  | machine-config-daemon-8ttrt | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-daemon-8ttrt to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-machine-config-operator |  | machine-config-daemon-5qfjm | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-daemon-5qfjm to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-machine-config-operator |  | machine-config-daemon-2xgfc | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-daemon-2xgfc to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-marketplace |  | marketplace-operator-664c8865b-d9dx9 | FailedScheduling | no nodes available to schedule pods
10:14:10 | openshift-etcd-operator |  | etcd-operator-859798c7f4-kb5lq | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-cluster-storage-operator |  | cluster-storage-operator-bc56464-b7kxp | FailedScheduling | no nodes available to schedule pods
10:14:13 | openshift-cluster-storage-operator |  | cluster-storage-operator-bc56464-b7kxp | FailedScheduling | no nodes available to schedule pods
10:14:13 | openshift-cluster-storage-operator |  | cluster-storage-operator-bc56464-b7kxp | FailedScheduling | no nodes available to schedule pods
10:47:40 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:47:30 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:28 | openshift-sdn |  | sdn-9n262 | Scheduled | Successfully assigned openshift-sdn/sdn-9n262 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:46:29 | openshift-sdn |  | sdn-5vdnk | Scheduled | Successfully assigned openshift-sdn/sdn-5vdnk to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-sdn |  | sdn-2ff5l | Scheduled | Successfully assigned openshift-sdn/sdn-2ff5l to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:47:05 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:54 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:43 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:32 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:21 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:10 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:45:59 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:45:48 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:29 | openshift-sdn |  | ovs-dpkmc | Scheduled | Successfully assigned openshift-sdn/ovs-dpkmc to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:45:37 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:45:26 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:45:15 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:45:04 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:44:53 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:44:42 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:44:31 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:44:21 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:44:10 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:43:59 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:28 | openshift-sdn |  | ovs-cb7xs | Scheduled | Successfully assigned openshift-sdn/ovs-cb7xs to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:41:19 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | pod 9f24917b-03b1-4435-9da3-bb875c548f80 is in the cache, so can't be assumed
10:41:21 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:46:28 | openshift-sdn |  | ovs-8mdrj | Scheduled | Successfully assigned openshift-sdn/ovs-8mdrj to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:43:49 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:41:24 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:25 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
11:01:09 | openshift-image-registry |  | image-registry-7b5479fc47-m64ps | Scheduled | Successfully assigned openshift-image-registry/image-registry-7b5479fc47-m64ps to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:42:27 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
12:11:49 | openshift-marketplace |  | redhat-marketplace-27l28 | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-27l28 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:29 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
11:37:11 | openshift-marketplace |  | redhat-marketplace-2g7kf | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-2g7kf to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:01:00 | openshift-marketplace |  | redhat-marketplace-6xsjw | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-6xsjw to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-operator-lifecycle-manager |  | olm-operator-5fbfc8c78-h4nwt | FailedScheduling | no nodes available to schedule pods
10:14:10 | openshift-operator-lifecycle-manager |  | olm-operator-5fbfc8c78-h4nwt | FailedScheduling | no nodes available to schedule pods
10:14:10 | openshift-operator-lifecycle-manager |  | olm-operator-5fbfc8c78-h4nwt | FailedScheduling | no nodes available to schedule pods
10:41:38 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:49 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:27 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:38 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:25 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:35 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:49 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:59 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:10 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:20 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:31 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:42 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:53 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:04 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:15 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:26 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:37 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:48 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:59 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:46:10 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:46:21 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:46:32 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:43 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:54 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:05 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:16 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:30 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:40 | openshift-apiserver |  | apiserver-686bc75759-v9g2f | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
12:21:49 | openshift-marketplace |  | redhat-marketplace-7wcsb | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-7wcsb to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:38 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:46:28 | openshift-image-registry |  | node-ca-2qml7 | Scheduled | Successfully assigned openshift-image-registry/node-ca-2qml7 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
12:32:15 | openshift-marketplace |  | redhat-marketplace-98mxp | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-98mxp to worker-1.case03116926.lab.upshift.rdu2.redhat.com
13:31:54 | openshift-marketplace |  | redhat-marketplace-cfbxv | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-cfbxv to worker-1.case03116926.lab.upshift.rdu2.redhat.com
13:14:10 | openshift-marketplace |  | redhat-marketplace-cs46k | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-cs46k to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:33 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | pod 4967afcb-d839-439b-a851-8de17c654154 is in the cache, so can't be assumed
10:33:25 | openshift-operator-lifecycle-manager |  | catalog-operator-887559f54-g4cgp | FailedScheduling | no nodes available to schedule pods
10:14:11 | openshift-operator-lifecycle-manager |  | catalog-operator-887559f54-g4cgp | FailedScheduling | no nodes available to schedule pods
10:14:11 | openshift-operator-lifecycle-manager |  | catalog-operator-887559f54-g4cgp | FailedScheduling | no nodes available to schedule pods
10:47:40 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:30 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:05 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:54 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:43 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:32 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:21 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:14:07 | openshift-apiserver-operator |  | openshift-apiserver-operator-56c79799ff-ntg2n | FailedScheduling | no nodes available to schedule pods
10:14:07 | openshift-apiserver-operator |  | openshift-apiserver-operator-56c79799ff-ntg2n | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-apiserver-operator |  | openshift-apiserver-operator-56c79799ff-ntg2n | FailedScheduling | no nodes available to schedule pods
10:53:16 | openshift-marketplace |  | redhat-marketplace-dkmx8 | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-dkmx8 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:31 | openshift-cluster-samples-operator |  | cluster-samples-operator-6c4b6b887b-cq6nw | FailedScheduling | Binding rejected: plugin "DefaultBinder" failed to bind pod "openshift-cluster-samples-operator/cluster-samples-operator-6c4b6b887b-cq6nw": Operation cannot be fulfilled on pods/binding "cluster-samples-operator-6c4b6b887b-cq6nw": pod cluster-samples-operator-6c4b6b887b-cq6nw is already assigned to node "master-2.case03116926.lab.upshift.rdu2.redhat.com"
11:15:06 | openshift-marketplace |  | redhat-marketplace-kw79d | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-kw79d to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:53:33 | openshift-marketplace |  | redhat-marketplace-p8mr5 | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-p8mr5 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:26:52 | openshift-marketplace |  | redhat-marketplace-qs42c | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-qs42c to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:50:26 | openshift-marketplace |  | redhat-marketplace-r25h2 | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-r25h2 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:43:50 | openshift-image-registry |  | node-ca-gjw8n | Scheduled | Successfully assigned openshift-image-registry/node-ca-gjw8n to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:10 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:59 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:48 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:37 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:33:25 | openshift-machine-api |  | machine-api-operator-784c77b6d4-xfllc | FailedScheduling | no nodes available to schedule pods
10:46:28 | openshift-image-registry |  | node-ca-lg2rm | Scheduled | Successfully assigned openshift-image-registry/node-ca-lg2rm to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:45:26 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:15 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:14:19 | openshift-machine-api |  | machine-api-operator-784c77b6d4-xfllc | FailedScheduling | no nodes available to schedule pods
10:43:50 | openshift-image-registry |  | node-ca-n2hfh | Scheduled | Successfully assigned openshift-image-registry/node-ca-n2hfh to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:43:50 | openshift-image-registry |  | node-ca-tqqc8 | Scheduled | Successfully assigned openshift-image-registry/node-ca-tqqc8 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:46:29 | openshift-image-registry |  | node-ca-xzbkf | Scheduled | Successfully assigned openshift-image-registry/node-ca-xzbkf to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-ingress |  | router-default-5b76b97bbb-jp2g9 | FailedScheduling | 0/3 nodes are available: 3 node(s) didn't match node selector.
10:46:43 | openshift-ingress |  | router-default-5b76b97bbb-jp2g9 | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:45:04 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:47:40 | openshift-ingress |  | router-default-5b76b97bbb-jp2g9 | Scheduled | Successfully assigned openshift-ingress/router-default-5b76b97bbb-jp2g9 to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-ingress |  | router-default-5b76b97bbb-ltvlc | FailedScheduling | 0/3 nodes are available: 3 node(s) didn't match node selector.
10:46:43 | openshift-ingress |  | router-default-5b76b97bbb-ltvlc | FailedScheduling | 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-ingress |  | router-default-5b76b97bbb-ltvlc | Scheduled | Successfully assigned openshift-ingress/router-default-5b76b97bbb-ltvlc to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:14:15 | openshift-ingress-operator |  | ingress-operator-cc4bb6454-wqjrn | FailedScheduling | no nodes available to schedule pods
10:14:15 | openshift-ingress-operator |  | ingress-operator-cc4bb6454-wqjrn | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-ingress-operator |  | ingress-operator-cc4bb6454-wqjrn | FailedScheduling | no nodes available to schedule pods
12:53:52 | openshift-marketplace |  | redhat-marketplace-sj6bn | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-sj6bn to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:03:36 | openshift-marketplace |  | redhat-marketplace-vsvkh | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-vsvkh to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:43:21 | openshift-marketplace |  | redhat-marketplace-wp2vw | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-wp2vw to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-marketplace |  | redhat-marketplace-xm8bs | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:46:29 | openshift-cluster-node-tuning-operator |  | tuned-dqzh5 | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/tuned-dqzh5 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:14:19 | openshift-machine-api |  | machine-api-operator-784c77b6d4-xfllc | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-insights |  | insights-operator-59575c9c9d-8bw8w | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-insights |  | insights-operator-59575c9c9d-8bw8w | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-insights |  | insights-operator-59575c9c9d-8bw8w | FailedScheduling | no nodes available to schedule pods
10:43:25 | openshift-console |  | downloads-88d4c8dfb-w49d2 | Scheduled | Successfully assigned openshift-console/downloads-88d4c8dfb-w49d2 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-machine-api |  | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-machine-api |  | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | no nodes available to schedule pods
10:14:17 | openshift-machine-api |  | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | no nodes available to schedule pods
10:44:09 | openshift-console-operator |  | console-operator-68cc7b855f-nldb7 | Scheduled | Successfully assigned openshift-console-operator/console-operator-68cc7b855f-nldb7 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:43 | openshift-marketplace |  | redhat-marketplace-xm8bs | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-marketplace |  | redhat-marketplace-xm8bs | Scheduled | Successfully assigned openshift-marketplace/redhat-marketplace-xm8bs to worker-2.case03116926.lab.upshift.rdu2.redhat.com
12:32:18 | openshift-marketplace |  | redhat-operators-4brnp | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-4brnp to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:45:26 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
10:45:26 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
10:46:36 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
10:46:36 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
10:47:36 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
10:56:24 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
10:56:24 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
10:44:53 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:42 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:31 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:20 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:09 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:49 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:25 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:38 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:59 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:47 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:36 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:22 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:19 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:17 | openshift-oauth-apiserver |  | apiserver-b7585d69b-l9525 | FailedScheduling | Binding rejected: plugin "DefaultBinder" failed to bind pod "openshift-oauth-apiserver/apiserver-b7585d69b-l9525": Operation cannot be fulfilled on pods/binding "apiserver-b7585d69b-l9525": pod apiserver-b7585d69b-l9525 is already assigned to node "master-1.case03116926.lab.upshift.rdu2.redhat.com"
10:47:40 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:30 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:16 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:05 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:54 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:43 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:32 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:21 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:46:10 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:59 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:48 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:37 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:26 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:15 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:04 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:53 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:42 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:31 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:20 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:09 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:59 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:49 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:25 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:38 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:47 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:36 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:47:52 | openshift-authentication |  | oauth-openshift-667c95946d-9m5z4 | Scheduled | Successfully assigned openshift-authentication/oauth-openshift-667c95946d-9m5z4 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:41:22 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:19 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:17 | openshift-oauth-apiserver |  | apiserver-b7585d69b-f6jdj | FailedScheduling | Binding rejected: plugin "DefaultBinder" failed to bind pod "openshift-oauth-apiserver/apiserver-b7585d69b-f6jdj": Operation cannot be fulfilled on pods/binding "apiserver-b7585d69b-f6jdj": pod apiserver-b7585d69b-f6jdj is already assigned to node "master-0.case03116926.lab.upshift.rdu2.redhat.com"
10:47:49 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:48:02 | openshift-authentication |  | oauth-openshift-667c95946d-kk679 | Scheduled | Successfully assigned openshift-authentication/oauth-openshift-667c95946d-kk679 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:47:38 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:16 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:05 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:54 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:43 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:47:41 | openshift-authentication |  | oauth-openshift-765c977d67-9ztxx | Scheduled | Successfully assigned openshift-authentication/oauth-openshift-765c977d67-9ztxx to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:46:32 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 5 node(s) didn't match the requested hostname.
10:46:21 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:46:10 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:59 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:48 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:37 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:26 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:47:41 | openshift-authentication |  | oauth-openshift-765c977d67-lh7jg | Scheduled | Successfully assigned openshift-authentication/oauth-openshift-765c977d67-lh7jg to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:45:15 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:45:04 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:53 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:42 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:31 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:20 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:44:09 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:49 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:43:25 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:42:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:47 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:47:44 | openshift-authentication |  | oauth-openshift-d5d844698-kh5nv | Scheduled | Successfully assigned openshift-authentication/oauth-openshift-d5d844698-kh5nv to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:41:36 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:27 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:22 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:19 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match the requested hostname.
10:41:17 | openshift-oauth-apiserver |  | apiserver-b7585d69b-7p56k | FailedScheduling | Binding rejected: plugin "DefaultBinder" failed to bind pod "openshift-oauth-apiserver/apiserver-b7585d69b-7p56k": Operation cannot be fulfilled on pods/binding "apiserver-b7585d69b-7p56k": pod apiserver-b7585d69b-7p56k is already assigned to node "master-2.case03116926.lab.upshift.rdu2.redhat.com"
13:03:57 | openshift-marketplace |  | redhat-operators-6hkhj | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-6hkhj to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-cluster-node-tuning-operator |  | tuned-97fwx | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/tuned-97fwx to worker-2.case03116926.lab.upshift.rdu2.redhat.com
13:36:17 | openshift-marketplace |  | redhat-operators-7q9g9 | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-7q9g9 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:53:07 | openshift-marketplace |  | redhat-operators-8t585 | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-8t585 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:07:23 | openshift-marketplace |  | redhat-operators-9fsfr | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-9fsfr to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:42:22 | openshift-marketplace |  | redhat-operators-dft94 | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-dft94 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:50:01 | openshift-marketplace |  | redhat-operators-ds5xd | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-ds5xd to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-cluster-node-tuning-operator |  | tuned-2kbh9 | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/tuned-2kbh9 to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-network-operator |  | network-operator-79b76f75b6-shnrd | FailedScheduling | no nodes available to schedule pods
10:14:00 | openshift-network-operator |  | network-operator-79b76f75b6-shnrd | FailedScheduling | no nodes available to schedule pods
10:14:10 | openshift-authentication-operator |  | authentication-operator-6fcbcf79b6-5h2gz | FailedScheduling | no nodes available to schedule pods
10:14:10 | openshift-authentication-operator |  | authentication-operator-6fcbcf79b6-5h2gz | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-authentication-operator |  | authentication-operator-6fcbcf79b6-5h2gz | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-cluster-node-tuning-operator |  | cluster-node-tuning-operator-c56cc8888-p6tkq | FailedScheduling | no nodes available to schedule pods
10:14:02 | openshift-cluster-node-tuning-operator |  | cluster-node-tuning-operator-c56cc8888-p6tkq | FailedScheduling | no nodes available to schedule pods
10:14:02 | openshift-cluster-node-tuning-operator |  | cluster-node-tuning-operator-c56cc8888-p6tkq | FailedScheduling | no nodes available to schedule pods
11:00:43 | openshift-marketplace |  | redhat-operators-fllzs | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-fllzs to worker-1.case03116926.lab.upshift.rdu2.redhat.com
12:21:49 | openshift-marketplace |  | redhat-operators-grgl5 | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-grgl5 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-marketplace |  | redhat-operators-hr4dq | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:57:34 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
10:14:00 | openshift-network-operator |  | network-operator-79b76f75b6-shnrd | FailedScheduling | no nodes available to schedule pods
10:46:33 | openshift-marketplace |  | redhat-operators-hr4dq | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-marketplace |  | redhat-operators-hr4dq | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-hr4dq to worker-2.case03116926.lab.upshift.rdu2.redhat.com
13:14:45 | openshift-marketplace |  | redhat-operators-ljxld | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-ljxld to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:32:47 | openshift-marketplace |  | redhat-operators-tn8wx | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-tn8wx to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-cluster-machine-approver |  | machine-approver-58d468647f-z7j52 | FailedScheduling | no nodes available to schedule pods
10:57:34 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
10:58:34 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
11:09:03 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
11:09:03 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
11:10:13 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
11:10:13 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
11:11:13 | openshift-kube-apiserver | apiserver | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
10:48:48 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
10:48:48 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
10:49:58 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
10:46:29 | openshift-multus |  | network-metrics-daemon-xxx4f | Scheduled | Successfully assigned openshift-multus/network-metrics-daemon-xxx4f to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:49:58 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
10:50:58 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
11:02:23 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
11:02:23 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
11:03:33 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
11:03:33 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
11:04:33 | openshift-kube-apiserver | apiserver | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
10:52:01 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
10:52:02 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
10:53:11 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
10:53:11 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
10:54:11 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
11:05:40 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
11:05:40 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
11:06:50 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 1m10s finished
11:06:50 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationStoppedServing | Server has stopped listening
11:07:50 | openshift-kube-apiserver | apiserver | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | TerminationGracefulTerminationFinished | All pending requests processed
10:14:18 | openshift-kube-apiserver-operator |  | kube-apiserver-operator-c74696d8d-gsntz | FailedScheduling | no nodes available to schedule pods
10:14:18 | openshift-kube-apiserver-operator |  | kube-apiserver-operator-c74696d8d-gsntz | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-kube-apiserver-operator |  | kube-apiserver-operator-c74696d8d-gsntz | FailedScheduling | no nodes available to schedule pods
10:14:12 | openshift-cluster-machine-approver |  | machine-approver-58d468647f-z7j52 | FailedScheduling | no nodes available to schedule pods
10:14:12 | openshift-cluster-machine-approver |  | machine-approver-58d468647f-z7j52 | FailedScheduling | no nodes available to schedule pods
11:56:35 | openshift-marketplace |  | redhat-operators-wsvbq | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-wsvbq to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:11:15 | openshift-marketplace |  | redhat-operators-z4lwg | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-z4lwg to worker-1.case03116926.lab.upshift.rdu2.redhat.com
13:25:53 | openshift-marketplace |  | redhat-operators-zc6ts | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-zc6ts to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:22:21 | openshift-marketplace |  | redhat-operators-zrw9j | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-zrw9j to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:47:37 | openshift-monitoring |  | alertmanager-main-0 | Scheduled | Successfully assigned openshift-monitoring/alertmanager-main-0 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:14:10 | openshift-etcd-operator |  | etcd-operator-859798c7f4-kb5lq | FailedScheduling | no nodes available to schedule pods
10:47:37 | openshift-monitoring |  | alertmanager-main-1 | Scheduled | Successfully assigned openshift-monitoring/alertmanager-main-1 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:47:37 | openshift-monitoring |  | alertmanager-main-2 | Scheduled | Successfully assigned openshift-monitoring/alertmanager-main-2 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:14:14 | openshift-monitoring |  | cluster-monitoring-operator-565dd756d4-2st68 | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-kube-storage-version-migrator-operator |  | kube-storage-version-migrator-operator-5999bc745c-nqrdg | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-cloud-credential-operator |  | cloud-credential-operator-78479968fc-vnsj9 | FailedScheduling | no nodes available to schedule pods
10:14:12 | openshift-cloud-credential-operator |  | cloud-credential-operator-78479968fc-vnsj9 | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-dns-operator |  | dns-operator-6979ff5964-t5q7m | FailedScheduling | no nodes available to schedule pods
10:14:00 | openshift-dns-operator |  | dns-operator-6979ff5964-t5q7m | FailedScheduling | no nodes available to schedule pods
10:14:00 | openshift-dns-operator |  | dns-operator-6979ff5964-t5q7m | FailedScheduling | no nodes available to schedule pods
10:46:27 | openshift-dns |  | dns-default-zwxfh | Scheduled | Successfully assigned openshift-dns/dns-default-zwxfh to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-dns |  | dns-default-xbgq5 | Scheduled | Successfully assigned openshift-dns/dns-default-xbgq5 to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:29 | openshift-dns |  | dns-default-fcp4w | Scheduled | Successfully assigned openshift-dns/dns-default-fcp4w to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:14:12 | openshift-cloud-credential-operator |  | cloud-credential-operator-78479968fc-vnsj9 | FailedScheduling | no nodes available to schedule pods
10:14:14 | openshift-monitoring |  | cluster-monitoring-operator-565dd756d4-2st68 | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-monitoring |  | cluster-monitoring-operator-565dd756d4-2st68 | FailedScheduling | no nodes available to schedule pods
10:14:06 | openshift-kube-storage-version-migrator-operator |  | kube-storage-version-migrator-operator-5999bc745c-nqrdg | FailedScheduling | no nodes available to schedule pods
10:14:04 | openshift-kube-controller-manager-operator |  | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | FailedScheduling | no nodes available to schedule pods
10:14:04 | openshift-kube-controller-manager-operator |  | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | FailedScheduling | no nodes available to schedule pods
10:33:25 | openshift-kube-controller-manager-operator |  | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | FailedScheduling | no nodes available to schedule pods
10:14:06 | openshift-kube-storage-version-migrator-operator |  | kube-storage-version-migrator-operator-5999bc745c-nqrdg | FailedScheduling | no nodes available to schedule pods
10:47:30 | openshift-kube-storage-version-migrator |  | migrator-85c499d985-hznj9 | Scheduled | Successfully assigned openshift-kube-storage-version-migrator/migrator-85c499d985-hznj9 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:46:43 | openshift-kube-storage-version-migrator |  | migrator-85c499d985-hznj9 | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:46:28 | openshift-multus |  | network-metrics-daemon-pkjvx | Scheduled | Successfully assigned openshift-multus/network-metrics-daemon-pkjvx to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-multus |  | network-metrics-daemon-m86r5 | Scheduled | Successfully assigned openshift-multus/network-metrics-daemon-m86r5 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
11:01:27 | openshift-monitoring |  | prometheus-k8s-1 | Scheduled | Successfully assigned openshift-monitoring/prometheus-k8s-1 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-kube-storage-version-migrator |  | migrator-85c499d985-hznj9 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:47:41 | openshift-monitoring |  | grafana-c4597b685-hvrqk | Scheduled | Successfully assigned openshift-monitoring/grafana-c4597b685-hvrqk to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:41:03 | openshift-monitoring |  | kube-state-metrics-6c8b757584-2b2dg | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:46:43 | openshift-monitoring |  | kube-state-metrics-6c8b757584-2b2dg | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-monitoring |  | kube-state-metrics-6c8b757584-2b2dg | Scheduled | Successfully assigned openshift-monitoring/kube-state-metrics-6c8b757584-2b2dg to worker-2.case03116926.lab.upshift.rdu2.redhat.com
11:02:00 | openshift-controller-manager |  | controller-manager-fz2vw | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-fz2vw to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-multus |  | multus-vrcmx | Scheduled | Successfully assigned openshift-multus/multus-vrcmx to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:29 | openshift-multus |  | multus-t6jxt | Scheduled | Successfully assigned openshift-multus/multus-t6jxt to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:02:02 | openshift-controller-manager |  | controller-manager-ndhqq | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-ndhqq to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:46:29 | openshift-monitoring |  | node-exporter-bvmqv | Scheduled | Successfully assigned openshift-monitoring/node-exporter-bvmqv to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-monitoring |  | node-exporter-tg54f | Scheduled | Successfully assigned openshift-monitoring/node-exporter-tg54f to worker-0.case03116926.lab.upshift.rdu2.redhat.com
11:02:04 | openshift-controller-manager |  | controller-manager-pcvtn | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-pcvtn to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:46:28 | openshift-monitoring |  | node-exporter-x72zx | Scheduled | Successfully assigned openshift-monitoring/node-exporter-x72zx to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-kube-scheduler-operator |  | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | FailedScheduling | no nodes available to schedule pods
10:41:03 | openshift-monitoring |  | openshift-state-metrics-6d99649c49-4b6cb | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:46:43 | openshift-monitoring |  | openshift-state-metrics-6d99649c49-4b6cb | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-monitoring |  | openshift-state-metrics-6d99649c49-4b6cb | Scheduled | Successfully assigned openshift-monitoring/openshift-state-metrics-6d99649c49-4b6cb to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:43:25 | openshift-console |  | downloads-88d4c8dfb-5g9qb | Scheduled | Successfully assigned openshift-console/downloads-88d4c8dfb-5g9qb to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:14:04 | openshift-kube-scheduler-operator |  | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | FailedScheduling | no nodes available to schedule pods
10:44:14 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-bzs9k | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:44:14 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-bzs9k | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:46:43 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-bzs9k | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-bzs9k | Scheduled | Successfully assigned openshift-monitoring/prometheus-adapter-5fdd4bd886-bzs9k to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:44:14 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-nnfsr | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:44:14 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-nnfsr | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:46:43 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-nnfsr | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:38 | openshift-monitoring |  | prometheus-adapter-5fdd4bd886-nnfsr | Scheduled | Successfully assigned openshift-monitoring/prometheus-adapter-5fdd4bd886-nnfsr to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:47:47 | openshift-monitoring |  | prometheus-k8s-0 | Scheduled | Successfully assigned openshift-monitoring/prometheus-k8s-0 to worker-0.case03116926.lab.upshift.rdu2.redhat.com
11:01:57 | openshift-monitoring |  | prometheus-k8s-0 | Scheduled | Successfully assigned openshift-monitoring/prometheus-k8s-0 to worker-0.case03116926.lab.upshift.rdu2.redhat.com
10:47:48 | openshift-monitoring |  | prometheus-k8s-1 | Scheduled | Successfully assigned openshift-monitoring/prometheus-k8s-1 to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:14:04 | openshift-kube-scheduler-operator |  | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | FailedScheduling | no nodes available to schedule pods
10:46:27 | openshift-multus |  | multus-kr75l | Scheduled | Successfully assigned openshift-multus/multus-kr75l to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:14:06 | openshift-controller-manager-operator |  | openshift-controller-manager-operator-7cbb8486b4-2jgvl | FailedScheduling | no nodes available to schedule pods
10:44:12 | openshift-monitoring |  | prometheus-operator-79ffc97ccd-5h2pl | Scheduled | Successfully assigned openshift-monitoring/prometheus-operator-79ffc97ccd-5h2pl to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:14:06 | openshift-controller-manager-operator |  | openshift-controller-manager-operator-7cbb8486b4-2jgvl | FailedScheduling | no nodes available to schedule pods
10:41:03 | openshift-monitoring |  | telemeter-client-57c59fc45f-gsqrj | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:46:43 | openshift-monitoring |  | telemeter-client-57c59fc45f-gsqrj | FailedScheduling | 0/6 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:47:30 | openshift-monitoring |  | telemeter-client-57c59fc45f-gsqrj | Scheduled | Successfully assigned openshift-monitoring/telemeter-client-57c59fc45f-gsqrj to worker-2.case03116926.lab.upshift.rdu2.redhat.com
10:33:25 | openshift-controller-manager-operator |  | openshift-controller-manager-operator-7cbb8486b4-2jgvl | FailedScheduling | no nodes available to schedule pods
10:47:40 | openshift-monitoring |  | thanos-querier-5c8f998999-bwnht | Scheduled | Successfully assigned openshift-monitoring/thanos-querier-5c8f998999-bwnht to worker-0.case03116926.lab.upshift.rdu2.redhat.com
12:56:04 | openshift-marketplace |  | certified-operators-tk85r | Scheduled | Successfully assigned openshift-marketplace/certified-operators-tk85r to worker-1.case03116926.lab.upshift.rdu2.redhat.com
11:46:32 | openshift-marketplace |  | redhat-operators-dth47 | Scheduled | Successfully assigned openshift-marketplace/redhat-operators-dth47 to worker-1.case03116926.lab.upshift.rdu2.redhat.com
10:13:18 | openshift-cluster-version | openshift-cluster-version | version | LeaderElection | bootstrap-0.case03116926.lab.upshift.rdu2.redhat.com_79a248b7-d0aa-40b8-9b38-c145b0ecb3ef became leader
10:13:28 | openshift-cluster-version | deployment-controller | cluster-version-operator | ScalingReplicaSet | Scaled up replica set cluster-version-operator-66d6584679 to 1
10:13:29 | openshift-cluster-version | replicaset-controller | cluster-version-operator-66d6584679 | SuccessfulCreate | Created pod: cluster-version-operator-66d6584679-59rx2
10:13:38 | openshift-kube-controller-manager | cluster-policy-controller | cluster-policy-controller | LeaderElection | bootstrap-0.case03116926.lab.upshift.rdu2.redhat.com became leader
10:13:40 | openshift-cluster-version | openshift-cluster-version | version | RetrievePayload | retrieving payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:13:40 | openshift-cluster-version | openshift-cluster-version | version | VerifyPayload | verifying payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:13:40 | openshift-cluster-version | openshift-cluster-version | version | PayloadLoaded | payload loaded version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:13:48 | openshift-cluster-version | replicaset-controller | cluster-version-operator-66d6584679 | SuccessfulDelete | Deleted pod: cluster-version-operator-66d6584679-59rx2
10:13:48 | openshift-cluster-version | replicaset-controller | cluster-version-operator-5d86d65f6b | SuccessfulCreate | Created pod: cluster-version-operator-5d86d65f6b-4fm8l
10:13:48 | openshift-cluster-version | deployment-controller | cluster-version-operator | ScalingReplicaSet | Scaled down replica set cluster-version-operator-66d6584679 to 0
10:13:48 | openshift-cluster-version | deployment-controller | cluster-version-operator | ScalingReplicaSet | Scaled up replica set cluster-version-operator-5d86d65f6b to 1
10:13:57 | openshift-service-ca-operator | replicaset-controller | service-ca-operator-6fd6454f86 | SuccessfulCreate | Created pod: service-ca-operator-6fd6454f86-5bbh8
10:13:57 | openshift-service-ca-operator | deployment-controller | service-ca-operator | ScalingReplicaSet | Scaled up replica set service-ca-operator-6fd6454f86 to 1
10:14:00 | openshift-network-operator | deployment-controller | network-operator | ScalingReplicaSet | Scaled up replica set network-operator-79b76f75b6 to 1
10:14:00 | openshift-dns-operator | replicaset-controller | dns-operator-6979ff5964 | SuccessfulCreate | Created pod: dns-operator-6979ff5964-t5q7m
10:14:00 | openshift-dns-operator | deployment-controller | dns-operator | ScalingReplicaSet | Scaled up replica set dns-operator-6979ff5964 to 1
10:14:00 | openshift-network-operator | replicaset-controller | network-operator-79b76f75b6 | SuccessfulCreate | Created pod: network-operator-79b76f75b6-shnrd
10:14:02 | openshift-cluster-node-tuning-operator | replicaset-controller | cluster-node-tuning-operator-c56cc8888 | SuccessfulCreate | Created pod: cluster-node-tuning-operator-c56cc8888-p6tkq
10:14:02 | openshift-cluster-node-tuning-operator | deployment-controller | cluster-node-tuning-operator | ScalingReplicaSet | Scaled up replica set cluster-node-tuning-operator-c56cc8888 to 1
10:14:03 | openshift-marketplace | deployment-controller | marketplace-operator | ScalingReplicaSet | Scaled up replica set marketplace-operator-664c8865b to 1
10:14:04 | openshift-kube-scheduler-operator | deployment-controller | openshift-kube-scheduler-operator | ScalingReplicaSet | Scaled up replica set openshift-kube-scheduler-operator-85dfcb9bcd to 1
10:14:04 | openshift-kube-controller-manager-operator | deployment-controller | kube-controller-manager-operator | ScalingReplicaSet | Scaled up replica set kube-controller-manager-operator-7bb9b6d8b8 to 1
10:14:04 | openshift-kube-controller-manager-operator | replicaset-controller | kube-controller-manager-operator-7bb9b6d8b8 | SuccessfulCreate | Created pod: kube-controller-manager-operator-7bb9b6d8b8-h8hjn
10:14:04 | openshift-kube-scheduler-operator | replicaset-controller | openshift-kube-scheduler-operator-85dfcb9bcd | SuccessfulCreate | Created pod: openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k
10:14:06 | openshift-controller-manager-operator | deployment-controller | openshift-controller-manager-operator | ScalingReplicaSet | Scaled up replica set openshift-controller-manager-operator-7cbb8486b4 to 1
10:14:06 | openshift-kube-storage-version-migrator-operator | deployment-controller | kube-storage-version-migrator-operator | ScalingReplicaSet | Scaled up replica set kube-storage-version-migrator-operator-5999bc745c to 1
10:14:06 | openshift-controller-manager-operator | replicaset-controller | openshift-controller-manager-operator-7cbb8486b4 | SuccessfulCreate | Created pod: openshift-controller-manager-operator-7cbb8486b4-2jgvl
10:14:06 | openshift-kube-storage-version-migrator-operator | replicaset-controller | kube-storage-version-migrator-operator-5999bc745c | SuccessfulCreate | Created pod: kube-storage-version-migrator-operator-5999bc745c-nqrdg
10:14:07 | openshift-cloud-credential-operator | deployment-controller | cloud-credential-operator | ScalingReplicaSet | Scaled up replica set cloud-credential-operator-78479968fc to 1
10:14:07 | openshift-apiserver-operator | replicaset-controller | openshift-apiserver-operator-56c79799ff | SuccessfulCreate | Created pod: openshift-apiserver-operator-56c79799ff-ntg2n
10:14:07 | openshift-apiserver-operator | deployment-controller | openshift-apiserver-operator | ScalingReplicaSet | Scaled up replica set openshift-apiserver-operator-56c79799ff to 1
10:14:08 (x11) | openshift-marketplace | replicaset-controller | marketplace-operator-664c8865b | FailedCreate | Error creating: pods "marketplace-operator-664c8865b-" is forbidden: unable to validate against any security context constraint: []
10:14:09 | openshift-config-operator | replicaset-controller | openshift-config-operator-697f744f69 | SuccessfulCreate | Created pod: openshift-config-operator-697f744f69-g6ck2
10:14:09 | openshift-config-operator | deployment-controller | openshift-config-operator | ScalingReplicaSet | Scaled up replica set openshift-config-operator-697f744f69 to 1
10:14:09 (x10) | openshift-cloud-credential-operator | replicaset-controller | cloud-credential-operator-78479968fc | FailedCreate | Error creating: pods "cloud-credential-operator-78479968fc-" is forbidden: unable to validate against any security context constraint: []
10:14:10 | openshift-authentication-operator | deployment-controller | authentication-operator | ScalingReplicaSet | Scaled up replica set authentication-operator-6fcbcf79b6 to 1
10:14:10 | openshift-operator-lifecycle-manager | replicaset-controller | olm-operator-5fbfc8c78 | SuccessfulCreate | Created pod: olm-operator-5fbfc8c78-h4nwt
10:14:10 | openshift-etcd-operator | replicaset-controller | etcd-operator-859798c7f4 | SuccessfulCreate | Created pod: etcd-operator-859798c7f4-kb5lq
10:14:10 | openshift-etcd-operator | deployment-controller | etcd-operator | ScalingReplicaSet | Scaled up replica set etcd-operator-859798c7f4 to 1
10:14:10 | openshift-authentication-operator | replicaset-controller | authentication-operator-6fcbcf79b6 | SuccessfulCreate | Created pod: authentication-operator-6fcbcf79b6-5h2gz
10:14:10 | openshift-operator-lifecycle-manager | deployment-controller | olm-operator | ScalingReplicaSet | Scaled up replica set olm-operator-5fbfc8c78 to 1
10:14:11 | openshift-operator-lifecycle-manager | deployment-controller | catalog-operator | ScalingReplicaSet | Scaled up replica set catalog-operator-887559f54 to 1
10:14:11 | openshift-operator-lifecycle-manager | replicaset-controller | catalog-operator-887559f54 | SuccessfulCreate | Created pod: catalog-operator-887559f54-g4cgp
10:14:12 | openshift-machine-config-operator | replicaset-controller | machine-config-operator-65cd958fc7 | SuccessfulCreate | Created pod: machine-config-operator-65cd958fc7-b5cm6
10:14:12 | openshift-cluster-machine-approver | replicaset-controller | machine-approver-58d468647f | SuccessfulCreate | Created pod: machine-approver-58d468647f-z7j52
10:14:12 | openshift-cluster-machine-approver | deployment-controller | machine-approver | ScalingReplicaSet | Scaled up replica set machine-approver-58d468647f to 1
10:14:12 | openshift-cloud-credential-operator | replicaset-controller | cloud-credential-operator-78479968fc | SuccessfulCreate | Created pod: cloud-credential-operator-78479968fc-vnsj9
10:14:12 | openshift-machine-config-operator | deployment-controller | machine-config-operator | ScalingReplicaSet | Scaled up replica set machine-config-operator-65cd958fc7 to 1
10:14:13 | openshift-cluster-storage-operator | deployment-controller | cluster-storage-operator | ScalingReplicaSet | Scaled up replica set cluster-storage-operator-bc56464 to 1
10:14:13 | openshift-cluster-storage-operator | replicaset-controller | cluster-storage-operator-bc56464 | SuccessfulCreate | Created pod: cluster-storage-operator-bc56464-b7kxp
10:14:14 | openshift-marketplace | replicaset-controller | marketplace-operator-664c8865b | SuccessfulCreate | Created pod: marketplace-operator-664c8865b-d9dx9
10:14:14 | openshift-monitoring | replicaset-controller | cluster-monitoring-operator-565dd756d4 | SuccessfulCreate | Created pod: cluster-monitoring-operator-565dd756d4-2st68
10:14:14 | openshift-monitoring | deployment-controller | cluster-monitoring-operator | ScalingReplicaSet | Scaled up replica set cluster-monitoring-operator-565dd756d4 to 1
10:14:15 | openshift-ingress-operator | deployment-controller | ingress-operator | ScalingReplicaSet | Scaled up replica set ingress-operator-cc4bb6454 to 1
10:14:15 | openshift-ingress-operator | replicaset-controller | ingress-operator-cc4bb6454 | SuccessfulCreate | Created pod: ingress-operator-cc4bb6454-wqjrn
10:14:17 | openshift-cluster-storage-operator | deployment-controller | csi-snapshot-controller-operator | ScalingReplicaSet | Scaled up replica set csi-snapshot-controller-operator-76dfcb9b88 to 1
10:14:17 | openshift-insights | deployment-controller | insights-operator | ScalingReplicaSet | Scaled up replica set insights-operator-59575c9c9d to 1
10:14:17 | openshift-machine-api | replicaset-controller | cluster-autoscaler-operator-557599ccf8 | SuccessfulCreate | Created pod: cluster-autoscaler-operator-557599ccf8-lchg2
10:14:17 | openshift-machine-api | deployment-controller | cluster-autoscaler-operator | ScalingReplicaSet | Scaled up replica set cluster-autoscaler-operator-557599ccf8 to 1
10:14:17 | openshift-image-registry | deployment-controller | cluster-image-registry-operator | ScalingReplicaSet | Scaled up replica set cluster-image-registry-operator-98c4d4975 to 1
10:14:17 | openshift-image-registry | replicaset-controller | cluster-image-registry-operator-98c4d4975 | SuccessfulCreate | Created pod: cluster-image-registry-operator-98c4d4975-kxn6n
10:14:17 | openshift-cluster-storage-operator | replicaset-controller | csi-snapshot-controller-operator-76dfcb9b88 | SuccessfulCreate | Created pod: csi-snapshot-controller-operator-76dfcb9b88-lt78x
10:14:17 | openshift-insights | replicaset-controller | insights-operator-59575c9c9d | SuccessfulCreate | Created pod: insights-operator-59575c9c9d-8bw8w
10:14:18 | openshift-kube-apiserver-operator | deployment-controller | kube-apiserver-operator | ScalingReplicaSet | Scaled up replica set kube-apiserver-operator-c74696d8d to 1
10:14:18 | openshift-kube-apiserver-operator | replicaset-controller | kube-apiserver-operator-c74696d8d | SuccessfulCreate | Created pod: kube-apiserver-operator-c74696d8d-gsntz
10:14:19 | openshift-machine-api | replicaset-controller | machine-api-operator-784c77b6d4 | SuccessfulCreate | Created pod: machine-api-operator-784c77b6d4-xfllc
10:14:19 | openshift-machine-api | deployment-controller | machine-api-operator | ScalingReplicaSet | Scaled up replica set machine-api-operator-784c77b6d4 to 1
10:33:25 | openshift-cluster-version | openshift-cluster-version | version | VerifyPayload | verifying payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:33:25 | openshift-kube-controller-manager | cluster-policy-controller | cluster-policy-controller | LeaderElection | bootstrap-0.case03116926.lab.upshift.rdu2.redhat.com became leader
10:33:25 | openshift-cluster-version | openshift-cluster-version | version | RetrievePayload | retrieving payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:33:25 | openshift-cluster-version | openshift-cluster-version | version | LeaderElection | bootstrap-0.case03116926.lab.upshift.rdu2.redhat.com_e80a2918-adc0-4071-a64c-97b2a93135e3 became leader
10:33:26 | openshift-cluster-version | openshift-cluster-version | version | PayloadLoaded | payload loaded version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:35:28 | openshift-kube-controller-manager | cluster-policy-controller | cluster-policy-controller | LeaderElection | bootstrap-0.case03116926.lab.upshift.rdu2.redhat.com became leader
10:35:54 | openshift-cluster-version | openshift-cluster-version | version | PayloadLoaded | payload loaded version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:35:54 | openshift-cluster-version | openshift-cluster-version | version | RetrievePayload | retrieving payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:35:54 | openshift-cluster-version | openshift-cluster-version | version | LeaderElection | bootstrap-0.case03116926.lab.upshift.rdu2.redhat.com_2a9db8ea-ef8d-496c-b647-9d97994ced34 became leader
10:35:54 | openshift-cluster-version | openshift-cluster-version | version | VerifyPayload | verifying payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:37:18 (x2) | openshift-network-operator | default-scheduler | network-operator-79b76f75b6-shnrd | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-image-registry | default-scheduler | cluster-image-registry-operator-98c4d4975-kxn6n | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-apiserver-operator | default-scheduler | openshift-apiserver-operator-56c79799ff-ntg2n | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-cluster-storage-operator | default-scheduler | cluster-storage-operator-bc56464-b7kxp | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-cluster-machine-approver | default-scheduler | machine-approver-58d468647f-z7j52 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-marketplace | default-scheduler | marketplace-operator-664c8865b-d9dx9 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-insights | default-scheduler | insights-operator-59575c9c9d-8bw8w | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-machine-config-operator | default-scheduler | machine-config-operator-65cd958fc7-b5cm6 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-kube-scheduler-operator | default-scheduler | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-cloud-credential-operator | default-scheduler | cloud-credential-operator-78479968fc-vnsj9 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-kube-storage-version-migrator-operator | default-scheduler | kube-storage-version-migrator-operator-5999bc745c-nqrdg | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-cluster-storage-operator | default-scheduler | csi-snapshot-controller-operator-76dfcb9b88-lt78x | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-dns-operator | default-scheduler | dns-operator-6979ff5964-t5q7m | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-controller-manager-operator | default-scheduler | openshift-controller-manager-operator-7cbb8486b4-2jgvl | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-service-ca-operator | default-scheduler | service-ca-operator-6fd6454f86-5bbh8 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-machine-api | default-scheduler | machine-api-operator-784c77b6d4-xfllc | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-kube-apiserver-operator | default-scheduler | kube-apiserver-operator-c74696d8d-gsntz | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-cluster-node-tuning-operator | default-scheduler | cluster-node-tuning-operator-c56cc8888-p6tkq | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-authentication-operator | default-scheduler | authentication-operator-6fcbcf79b6-5h2gz | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-kube-controller-manager-operator | default-scheduler | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-operator-lifecycle-manager | default-scheduler | olm-operator-5fbfc8c78-h4nwt | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-config-operator | default-scheduler | openshift-config-operator-697f744f69-g6ck2 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-machine-api | default-scheduler | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-etcd-operator | default-scheduler | etcd-operator-859798c7f4-kb5lq | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-monitoring | default-scheduler | cluster-monitoring-operator-565dd756d4-2st68 | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-operator-lifecycle-manager | default-scheduler | catalog-operator-887559f54-g4cgp | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-cluster-version | default-scheduler | cluster-version-operator-5d86d65f6b-4fm8l | FailedScheduling | no nodes available to schedule pods
10:37:18 (x2) | openshift-ingress-operator | default-scheduler | ingress-operator-cc4bb6454-wqjrn | FailedScheduling | no nodes available to schedule pods
10:37:56 | openshift-machine-api | default-scheduler | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:37:57 | openshift-network-operator | default-scheduler | network-operator-79b76f75b6-shnrd | Scheduled | Successfully assigned openshift-network-operator/network-operator-79b76f75b6-shnrd to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:37:58 | openshift-network-operator | kubelet | network-operator-79b76f75b6-shnrd | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c293b27533b002d4d330ba3c56a91ac1932744ad46b7dc87f0d334eb7e0fabc7"
10:38:01 | openshift-network-operator | kubelet | network-operator-79b76f75b6-shnrd | Started | Started container network-operator
10:38:01 | openshift-network-operator | kubelet | network-operator-79b76f75b6-shnrd | Created | Created container network-operator
10:38:01 | openshift-network-operator | kubelet | network-operator-79b76f75b6-shnrd | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c293b27533b002d4d330ba3c56a91ac1932744ad46b7dc87f0d334eb7e0fabc7" in 2.839252914s
10:38:05 (x2) | openshift-machine-api | default-scheduler | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:07 | openshift-multus | daemonset-controller | multus | SuccessfulCreate | Created pod: multus-mdwnr
10:38:07 | openshift-multus | daemonset-controller | multus | SuccessfulCreate | Created pod: multus-67vwz
10:38:07 | openshift-multus | default-scheduler | network-metrics-daemon-2nbpw | Scheduled | Successfully assigned openshift-multus/network-metrics-daemon-2nbpw to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:07 | openshift-multus | default-scheduler | multus-mdwnr | Scheduled | Successfully assigned openshift-multus/multus-mdwnr to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:07 | openshift-multus | default-scheduler | network-metrics-daemon-djqpl | Scheduled | Successfully assigned openshift-multus/network-metrics-daemon-djqpl to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:07 | openshift-multus | daemonset-controller | network-metrics-daemon | SuccessfulCreate | Created pod: network-metrics-daemon-djqpl
10:38:07 | openshift-multus | daemonset-controller | network-metrics-daemon | SuccessfulCreate | Created pod: network-metrics-daemon-2nbpw
10:38:07 | openshift-multus | default-scheduler | multus-67vwz | Scheduled | Successfully assigned openshift-multus/multus-67vwz to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:08 | openshift-multus | kubelet | multus-mdwnr | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe"
10:38:09 | openshift-multus | kubelet | multus-67vwz | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe"
10:38:10 (x3) | openshift-cluster-storage-operator | default-scheduler | cluster-storage-operator-bc56464-b7kxp | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-image-registry | default-scheduler | cluster-image-registry-operator-98c4d4975-kxn6n | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-insights | default-scheduler | insights-operator-59575c9c9d-8bw8w | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-controller-manager-operator | default-scheduler | openshift-controller-manager-operator-7cbb8486b4-2jgvl | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-service-ca-operator | default-scheduler | service-ca-operator-6fd6454f86-5bbh8 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-cloud-credential-operator | default-scheduler | cloud-credential-operator-78479968fc-vnsj9 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-cluster-machine-approver | default-scheduler | machine-approver-58d468647f-z7j52 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-apiserver-operator | default-scheduler | openshift-apiserver-operator-56c79799ff-ntg2n | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-ingress-operator | default-scheduler | ingress-operator-cc4bb6454-wqjrn | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-dns-operator | default-scheduler | dns-operator-6979ff5964-t5q7m | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-cluster-version | default-scheduler | cluster-version-operator-5d86d65f6b-4fm8l | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-monitoring | default-scheduler | cluster-monitoring-operator-565dd756d4-2st68 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-operator-lifecycle-manager | default-scheduler | catalog-operator-887559f54-g4cgp | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-operator-lifecycle-manager | default-scheduler | olm-operator-5fbfc8c78-h4nwt | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 | openshift-sdn | daemonset-controller | sdn-controller | SuccessfulCreate | Created pod: sdn-controller-2pwxc
10:38:10 (x3) | openshift-cluster-storage-operator | default-scheduler | csi-snapshot-controller-operator-76dfcb9b88-lt78x | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-cluster-node-tuning-operator | default-scheduler | cluster-node-tuning-operator-c56cc8888-p6tkq | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-machine-config-operator | default-scheduler | machine-config-operator-65cd958fc7-b5cm6 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-etcd-operator | default-scheduler | etcd-operator-859798c7f4-kb5lq | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-machine-api | default-scheduler | machine-api-operator-784c77b6d4-xfllc | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-authentication-operator | default-scheduler | authentication-operator-6fcbcf79b6-5h2gz | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-kube-controller-manager-operator | default-scheduler | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:10 (x3) | openshift-kube-scheduler-operator | default-scheduler | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:11 | openshift-sdn | default-scheduler | sdn-controller-2pwxc | Scheduled | Successfully assigned openshift-sdn/sdn-controller-2pwxc to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:11 | openshift-sdn | default-scheduler | ovs-cc5d4 | Scheduled | Successfully assigned openshift-sdn/ovs-cc5d4 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:11 (x3) | openshift-marketplace | default-scheduler | marketplace-operator-664c8865b-d9dx9 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:11 | openshift-sdn | default-scheduler | sdn-b667g | Scheduled | Successfully assigned openshift-sdn/sdn-b667g to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:11 (x3) | openshift-kube-storage-version-migrator-operator | default-scheduler | kube-storage-version-migrator-operator-5999bc745c-nqrdg | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:11 | openshift-sdn | daemonset-controller | ovs | SuccessfulCreate | Created pod: ovs-cc5d4
10:38:11 | openshift-sdn | daemonset-controller | ovs | SuccessfulCreate | Created pod: ovs-jxg8z
10:38:11 | openshift-sdn | kubelet | ovs-jxg8z | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:11 | openshift-sdn | default-scheduler | ovs-jxg8z | Scheduled | Successfully assigned openshift-sdn/ovs-jxg8z to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:11 | openshift-sdn | default-scheduler | sdn-controller-65mtj | Scheduled | Successfully assigned openshift-sdn/sdn-controller-65mtj to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:11 | openshift-sdn | daemonset-controller | sdn-controller | SuccessfulCreate | Created pod: sdn-controller-65mtj
10:38:11 | openshift-sdn | default-scheduler | sdn-hjzkb | Scheduled | Successfully assigned openshift-sdn/sdn-hjzkb to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:11 | openshift-sdn | daemonset-controller | sdn | SuccessfulCreate | Created pod: sdn-hjzkb
10:38:11 | openshift-sdn | daemonset-controller | sdn | SuccessfulCreate | Created pod: sdn-b667g
10:38:11 | openshift-multus | kubelet | multus-mdwnr | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" in 2.958005763s
10:38:11 | openshift-multus | kubelet | multus-mdwnr | Created | Created container multus-binary-copy
10:38:11 | openshift-multus | kubelet | multus-mdwnr | Started | Started container multus-binary-copy
10:38:11 (x3) | openshift-config-operator | default-scheduler | openshift-config-operator-697f744f69-g6ck2 | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:11 (x3) | openshift-kube-apiserver-operator | default-scheduler | kube-apiserver-operator-c74696d8d-gsntz | FailedScheduling | 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:11 | openshift-sdn | kubelet | sdn-hjzkb | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:12 | openshift-multus | kubelet | multus-67vwz | Started | Started container multus-binary-copy
10:38:12 | openshift-sdn | kubelet | sdn-controller-65mtj | FailedMount | MountVolume.SetUp failed for volume "sdn-controller-token-f8nkk" : failed to sync secret cache: timed out waiting for the condition
10:38:12 | openshift-multus | kubelet | multus-mdwnr | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398"
10:38:12 | openshift-multus | kubelet | multus-67vwz | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" in 2.845175969s
10:38:12 | openshift-sdn | kubelet | sdn-controller-65mtj | FailedMount | MountVolume.SetUp failed for volume "env-overrides" : failed to sync configmap cache: timed out waiting for the condition
10:38:12 | openshift-multus | kubelet | multus-67vwz | Created | Created container multus-binary-copy
10:38:12 | openshift-sdn | kubelet | sdn-b667g | FailedMount | MountVolume.SetUp failed for volume "env-overrides" : failed to sync configmap cache: timed out waiting for the condition
10:38:12 | openshift-sdn | kubelet | sdn-controller-2pwxc | FailedMount | MountVolume.SetUp failed for volume "sdn-controller-token-f8nkk" : failed to sync secret cache: timed out waiting for the condition
10:38:12 | openshift-sdn | kubelet | ovs-cc5d4 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:12 | openshift-multus | kubelet | multus-67vwz | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398"
10:38:13 | openshift-sdn | kubelet | sdn-controller-2pwxc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:13 | openshift-sdn | kubelet | sdn-b667g | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:13 | openshift-sdn | kubelet | sdn-controller-65mtj | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:15 | openshift-sdn | daemonset-controller | sdn-controller | SuccessfulCreate | Created pod: sdn-controller-bzqsb
10:38:15 | openshift-multus | daemonset-controller | multus | SuccessfulCreate | Created pod: multus-8vk2z
10:38:15 | openshift-multus | daemonset-controller | network-metrics-daemon | SuccessfulCreate | Created pod: network-metrics-daemon-sw48w
10:38:15 | openshift-sdn | default-scheduler | ovs-d5br9 | Scheduled | Successfully assigned openshift-sdn/ovs-d5br9 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:15 | openshift-sdn | daemonset-controller | sdn | SuccessfulCreate | Created pod: sdn-cgjgd
10:38:15 | openshift-multus | default-scheduler | multus-8vk2z | Scheduled | Successfully assigned openshift-multus/multus-8vk2z to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:15 | openshift-multus | default-scheduler | network-metrics-daemon-sw48w | Scheduled | Successfully assigned openshift-multus/network-metrics-daemon-sw48w to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:15 | openshift-sdn | daemonset-controller | ovs | SuccessfulCreate | Created pod: ovs-d5br9
10:38:15 | openshift-sdn | default-scheduler | sdn-controller-bzqsb | Scheduled | Successfully assigned openshift-sdn/sdn-controller-bzqsb to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:15 | openshift-sdn | default-scheduler | sdn-cgjgd | Scheduled | Successfully assigned openshift-sdn/sdn-cgjgd to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:19 | openshift-sdn | kubelet | sdn-cgjgd | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:19 | openshift-multus | kubelet | multus-8vk2z | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe"
10:38:19 | openshift-sdn | kubelet | ovs-d5br9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:19 | openshift-sdn | kubelet | sdn-controller-bzqsb | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:38:21 | openshift-multus | kubelet | multus-67vwz | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398" in 9.061077879s
10:38:21 | openshift-apiserver-operator | default-scheduler | openshift-apiserver-operator-56c79799ff-ntg2n | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-sdn | kubelet | ovs-cc5d4 | Created | Created container openvswitch
10:38:21 | openshift-kube-scheduler-operator | default-scheduler | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-sdn | kubelet | sdn-controller-65mtj | Started | Started container sdn-controller
10:38:21 | openshift-operator-lifecycle-manager | default-scheduler | catalog-operator-887559f54-g4cgp | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-sdn | kubelet | sdn-b667g | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:38:21 | openshift-kube-controller-manager-operator | default-scheduler | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-sdn | kubelet | sdn-controller-65mtj | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.025868942s
10:38:21 | openshift-multus | kubelet | multus-mdwnr | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398" in 8.772198872s
10:38:21 | openshift-sdn | kubelet | sdn-controller-2pwxc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 7.778191179s
10:38:21 | openshift-cloud-credential-operator | default-scheduler | cloud-credential-operator-78479968fc-vnsj9 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-multus | kubelet | multus-mdwnr | Created | Created container cni-plugins
10:38:21 | openshift-multus | kubelet | multus-mdwnr | Started | Started container cni-plugins
10:38:21 | openshift-image-registry | default-scheduler | cluster-image-registry-operator-98c4d4975-kxn6n | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-sdn | kubelet | sdn-hjzkb | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:38:21 | openshift-sdn | kubelet | sdn-controller-65mtj | Created | Created container sdn-controller
10:38:21 | openshift-multus | kubelet | multus-67vwz | Started | Started container cni-plugins
10:38:21 | openshift-sdn | kubelet | ovs-cc5d4 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 9.025910687s
10:38:21 | openshift-sdn | kubelet | sdn-b667g | Started | Started container sdn
10:38:21 | openshift-sdn | kubelet | ovs-cc5d4 | Started | Started container openvswitch
10:38:21 | openshift-insights | default-scheduler | insights-operator-59575c9c9d-8bw8w | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-etcd-operator | default-scheduler | etcd-operator-859798c7f4-kb5lq | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-authentication-operator | default-scheduler | authentication-operator-6fcbcf79b6-5h2gz | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-cluster-storage-operator | default-scheduler | csi-snapshot-controller-operator-76dfcb9b88-lt78x | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-sdn | kubelet | sdn-hjzkb | Started | Started container sdn
10:38:21 | openshift-sdn | kubelet | sdn-hjzkb | Created | Created container sdn
10:38:21 | openshift-sdn | kubelet | sdn-hjzkb | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 9.09619009s
10:38:21 | openshift-sdn | kubelet | ovs-jxg8z | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 9.127374343s
10:38:21 | openshift-sdn | kubelet | ovs-jxg8z | Created | Created container openvswitch
10:38:21 | openshift-sdn | kubelet | ovs-jxg8z | Started | Started container openvswitch
10:38:21 | openshift-sdn | kubelet | sdn-b667g | Created | Created container sdn
10:38:21 | openshift-sdn | kubelet | sdn-b667g | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 7.779029504s
10:38:21 | openshift-multus | kubelet | multus-67vwz | Created | Created container cni-plugins
10:38:21 | openshift-cluster-node-tuning-operator | default-scheduler | cluster-node-tuning-operator-c56cc8888-p6tkq | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:21 | openshift-machine-config-operator | default-scheduler | machine-config-operator-65cd958fc7-b5cm6 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-cluster-machine-approver | default-scheduler | machine-approver-58d468647f-z7j52 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-kube-apiserver-operator | default-scheduler | kube-apiserver-operator-c74696d8d-gsntz | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-config-operator | default-scheduler | openshift-config-operator-697f744f69-g6ck2 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-controller-manager-operator | default-scheduler | openshift-controller-manager-operator-7cbb8486b4-2jgvl | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-kube-storage-version-migrator-operator | default-scheduler | kube-storage-version-migrator-operator-5999bc745c-nqrdg | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-multus | kubelet | multus-8vk2z | Started | Started container multus-binary-copy
10:38:22 | openshift-monitoring | default-scheduler | cluster-monitoring-operator-565dd756d4-2st68 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-multus | kubelet | multus-67vwz | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9"
10:38:22 | openshift-cluster-storage-operator | default-scheduler | cluster-storage-operator-bc56464-b7kxp | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-dns-operator | default-scheduler | dns-operator-6979ff5964-t5q7m | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-ingress-operator | default-scheduler | ingress-operator-cc4bb6454-wqjrn | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-multus | kubelet | multus-8vk2z | Created | Created container multus-binary-copy
10:38:22 | openshift-multus | kubelet | multus-8vk2z | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" in 3.147840795s
10:38:22 | openshift-cluster-version | default-scheduler | cluster-version-operator-5d86d65f6b-4fm8l | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-marketplace | default-scheduler | marketplace-operator-664c8865b-d9dx9 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-operator-lifecycle-manager | default-scheduler | olm-operator-5fbfc8c78-h4nwt | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:22 | openshift-multus | kubelet | multus-mdwnr | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9"
10:38:23 | openshift-multus | kubelet | multus-67vwz | Started | Started container routeoverride-cni
10:38:23 | openshift-multus | kubelet | multus-67vwz | Created | Created container routeoverride-cni
10:38:23 | openshift-multus | kubelet | multus-mdwnr | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9" in 1.252901217s
10:38:23 | openshift-sdn | kubelet | sdn-b667g | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 2.092735411s
10:38:23 | openshift-sdn | kubelet | sdn-hjzkb | Created | Created container kube-rbac-proxy
10:38:23 | openshift-sdn | kubelet | sdn-b667g | Started | Started container kube-rbac-proxy
10:38:23 | openshift-multus | kubelet | multus-mdwnr | Started | Started container routeoverride-cni
10:38:23 | openshift-multus | kubelet | multus-67vwz | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9" in 1.35785843s
10:38:23 | openshift-multus | kubelet | multus-mdwnr | Created | Created container routeoverride-cni
10:38:23 | openshift-multus | kubelet | multus-8vk2z | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398"
10:38:23 | openshift-sdn | kubelet | sdn-hjzkb | Started | Started container kube-rbac-proxy
10:38:23 | openshift-sdn | kubelet | sdn-hjzkb | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 1.719017944s
10:38:23 | openshift-sdn | kubelet | sdn-b667g | Created | Created container kube-rbac-proxy
10:38:24 | openshift-multus | kubelet | multus-67vwz | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754"
10:38:24 | openshift-multus | kubelet | multus-mdwnr | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754"
10:38:27 | openshift-multus | kubelet | multus-67vwz | Started | Started container whereabouts-cni
10:38:27 | openshift-multus | kubelet | multus-67vwz | Created | Created container whereabouts-cni
10:38:27 | openshift-sdn | kubelet | sdn-cgjgd | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.799051342s
10:38:27 | openshift-multus | kubelet | multus-67vwz | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" already present on machine
10:38:27 | openshift-sdn | kubelet | ovs-d5br9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.824960171s
10:38:27 | openshift-sdn | kubelet | sdn-controller-bzqsb | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.730383347s
10:38:27 | openshift-multus | kubelet | multus-67vwz | Started | Started container whereabouts-cni-bincopy
10:38:27 | openshift-multus | kubelet | multus-mdwnr | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" in 2.789218796s
10:38:27 | openshift-multus | kubelet | multus-mdwnr | Created | Created container whereabouts-cni-bincopy
10:38:27 | openshift-multus | kubelet | multus-67vwz | Created | Created container whereabouts-cni-bincopy
10:38:27 | openshift-multus | kubelet | multus-67vwz | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" in 2.716553972s
10:38:27 | openshift-multus | kubelet | multus-mdwnr | Started | Started container whereabouts-cni-bincopy
10:38:28 | openshift-multus | kubelet | multus-67vwz | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:38:28 | openshift-sdn | kubelet | ovs-d5br9 | Created | Created container openvswitch
10:38:28 | openshift-multus | kubelet | multus-mdwnr | Started | Started container whereabouts-cni
10:38:28 | openshift-multus | kubelet | multus-mdwnr | Created | Created container whereabouts-cni
10:38:28 | openshift-multus | kubelet | multus-mdwnr | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" already present on machine
10:38:28 | openshift-sdn | kubelet | sdn-controller-bzqsb | Started | Started container sdn-controller
10:38:28 | openshift-multus | kubelet | multus-67vwz | Started | Started container kube-multus
10:38:28 | openshift-sdn | kubelet | sdn-cgjgd | Created | Created container sdn
10:38:28 | openshift-multus | kubelet | multus-67vwz | Created | Created container kube-multus
10:38:28 | openshift-sdn | kubelet | sdn-controller-bzqsb | Created | Created container sdn-controller
10:38:28 | openshift-sdn | kubelet | sdn-cgjgd | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:38:28 | openshift-sdn | kubelet | sdn-cgjgd | Started | Started container sdn
10:38:28 | openshift-sdn | kubelet | ovs-d5br9 | Started | Started container openvswitch
10:38:29 | openshift-multus | kubelet | multus-mdwnr | Started | Started container kube-multus
10:38:29 (x12) | openshift-multus | kubelet | network-metrics-daemon-djqpl | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:38:29 | openshift-multus | kubelet | multus-mdwnr | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:38:29 | openshift-multus | kubelet | multus-mdwnr | Created | Created container kube-multus
10:38:30 | openshift-sdn | kubelet | sdn-cgjgd | Started | Started container kube-rbac-proxy
10:38:30 | openshift-sdn | kubelet | sdn-cgjgd | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 2.238408875s
10:38:30 | openshift-sdn | kubelet | sdn-cgjgd | Created | Created container kube-rbac-proxy
10:38:33 (x14) | openshift-multus | kubelet | network-metrics-daemon-2nbpw | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:38:34 | openshift-multus | kubelet | multus-8vk2z | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398" in 11.107784527s
10:38:35 | openshift-multus | kubelet | multus-8vk2z | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9"
10:38:35 | openshift-multus | kubelet | multus-8vk2z | Created | Created container cni-plugins
10:38:35 | openshift-multus | kubelet | multus-8vk2z | Started | Started container cni-plugins
10:38:36 | openshift-multus | kubelet | multus-8vk2z | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9" in 1.101968379s
10:38:37 | openshift-image-registry | default-scheduler | cluster-image-registry-operator-98c4d4975-kxn6n | Scheduled | Successfully assigned openshift-image-registry/cluster-image-registry-operator-98c4d4975-kxn6n to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-multus | kubelet | multus-8vk2z | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754"
10:38:37 | openshift-kube-storage-version-migrator-operator | default-scheduler | kube-storage-version-migrator-operator-5999bc745c-nqrdg | Scheduled | Successfully assigned openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-5999bc745c-nqrdg to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-cluster-node-tuning-operator | default-scheduler | cluster-node-tuning-operator-c56cc8888-p6tkq | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/cluster-node-tuning-operator-c56cc8888-p6tkq to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 (x2) | openshift-machine-api | default-scheduler | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:37 | openshift-multus | default-scheduler | multus-admission-controller-bkzjk | Scheduled | Successfully assigned openshift-multus/multus-admission-controller-bkzjk to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-multus | default-scheduler | multus-admission-controller-gsjzb | Scheduled | Successfully assigned openshift-multus/multus-admission-controller-gsjzb to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-multus | kubelet | multus-8vk2z | Started | Started container routeoverride-cni
10:38:37 | openshift-operator-lifecycle-manager | default-scheduler | olm-operator-5fbfc8c78-h4nwt | Scheduled | Successfully assigned openshift-operator-lifecycle-manager/olm-operator-5fbfc8c78-h4nwt to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-kube-controller-manager-operator | default-scheduler | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | Scheduled | Successfully assigned openshift-kube-controller-manager-operator/kube-controller-manager-operator-7bb9b6d8b8-h8hjn to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-cluster-machine-approver | default-scheduler | machine-approver-58d468647f-z7j52 | Scheduled | Successfully assigned openshift-cluster-machine-approver/machine-approver-58d468647f-z7j52 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-dns-operator | default-scheduler | dns-operator-6979ff5964-t5q7m | Scheduled | Successfully assigned openshift-dns-operator/dns-operator-6979ff5964-t5q7m to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-multus | daemonset-controller | multus-admission-controller | SuccessfulCreate | Created pod: multus-admission-controller-gsjzb
10:38:37 | openshift-multus | daemonset-controller | multus-admission-controller | SuccessfulCreate | Created pod: multus-admission-controller-bkzjk
10:38:37 | openshift-operator-lifecycle-manager | default-scheduler | catalog-operator-887559f54-g4cgp | Scheduled | Successfully assigned openshift-operator-lifecycle-manager/catalog-operator-887559f54-g4cgp to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-controller-manager-operator | default-scheduler | openshift-controller-manager-operator-7cbb8486b4-2jgvl | Scheduled | Successfully assigned openshift-controller-manager-operator/openshift-controller-manager-operator-7cbb8486b4-2jgvl to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-marketplace | default-scheduler | marketplace-operator-664c8865b-d9dx9 | Scheduled | Successfully assigned openshift-marketplace/marketplace-operator-664c8865b-d9dx9 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-multus | kubelet | multus-8vk2z | Created | Created container routeoverride-cni
10:38:37 | openshift-kube-scheduler-operator | default-scheduler | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | Scheduled | Successfully assigned openshift-kube-scheduler-operator/openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-apiserver-operator | default-scheduler | openshift-apiserver-operator-56c79799ff-ntg2n | Scheduled | Successfully assigned openshift-apiserver-operator/openshift-apiserver-operator-56c79799ff-ntg2n to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-monitoring | default-scheduler | cluster-monitoring-operator-565dd756d4-2st68 | Scheduled | Successfully assigned openshift-monitoring/cluster-monitoring-operator-565dd756d4-2st68 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-machine-config-operator | default-scheduler | machine-config-operator-65cd958fc7-b5cm6 | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-operator-65cd958fc7-b5cm6 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-authentication-operator | default-scheduler | authentication-operator-6fcbcf79b6-5h2gz | Scheduled | Successfully assigned openshift-authentication-operator/authentication-operator-6fcbcf79b6-5h2gz to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-cluster-storage-operator | default-scheduler | cluster-storage-operator-bc56464-b7kxp | Scheduled | Successfully assigned openshift-cluster-storage-operator/cluster-storage-operator-bc56464-b7kxp to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-etcd-operator | default-scheduler | etcd-operator-859798c7f4-kb5lq | Scheduled | Successfully assigned openshift-etcd-operator/etcd-operator-859798c7f4-kb5lq to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-cluster-version | default-scheduler | cluster-version-operator-5d86d65f6b-4fm8l | Scheduled | Successfully assigned openshift-cluster-version/cluster-version-operator-5d86d65f6b-4fm8l to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-cluster-storage-operator | default-scheduler | csi-snapshot-controller-operator-76dfcb9b88-lt78x | Scheduled | Successfully assigned openshift-cluster-storage-operator/csi-snapshot-controller-operator-76dfcb9b88-lt78x to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-cloud-credential-operator | default-scheduler | cloud-credential-operator-78479968fc-vnsj9 | Scheduled | Successfully assigned openshift-cloud-credential-operator/cloud-credential-operator-78479968fc-vnsj9 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-ingress-operator | default-scheduler | ingress-operator-cc4bb6454-wqjrn | Scheduled | Successfully assigned openshift-ingress-operator/ingress-operator-cc4bb6454-wqjrn to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-insights | default-scheduler | insights-operator-59575c9c9d-8bw8w | Scheduled | Successfully assigned openshift-insights/insights-operator-59575c9c9d-8bw8w to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 | openshift-config-operator | default-scheduler | openshift-config-operator-697f744f69-g6ck2 | Scheduled | Successfully assigned openshift-config-operator/openshift-config-operator-697f744f69-g6ck2 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:37 (x2) | openshift-service-ca-operator | default-scheduler | service-ca-operator-6fd6454f86-5bbh8 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:37 (x2) | openshift-machine-api | default-scheduler | machine-api-operator-784c77b6d4-xfllc | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
10:38:37 | openshift-kube-apiserver-operator | default-scheduler | kube-apiserver-operator-c74696d8d-gsntz | Scheduled | Successfully assigned openshift-kube-apiserver-operator/kube-apiserver-operator-c74696d8d-gsntz to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:38 | openshift-apiserver-operator | multus | openshift-apiserver-operator-56c79799ff-ntg2n | AddedInterface | Add eth0 [10.128.0.3/23]
10:38:38 | openshift-machine-config-operator | multus | machine-config-operator-65cd958fc7-b5cm6 | AddedInterface | Add eth0 [10.128.0.5/23]
10:38:38 | openshift-kube-controller-manager-operator | multus | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | AddedInterface | Add eth0 [10.130.0.3/23]
10:38:38 | openshift-kube-scheduler-operator | multus | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | AddedInterface | Add eth0 [10.130.0.5/23]
10:38:39 | openshift-etcd-operator | multus | etcd-operator-859798c7f4-kb5lq | AddedInterface | Add eth0 [10.130.0.4/23]
10:38:39 | openshift-cluster-node-tuning-operator | multus | cluster-node-tuning-operator-c56cc8888-p6tkq | AddedInterface | Add eth0 [10.128.0.2/23]
10:38:39 | openshift-kube-controller-manager-operator | kubelet | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a"
10:38:39 | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-operator-76dfcb9b88-lt78x | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b798a2aeb0d74b2644b613801feb06411c9f6444e4be462073b72df5bb35451"
10:38:39 | openshift-monitoring | multus | cluster-monitoring-operator-565dd756d4-2st68 | AddedInterface | Add eth0 [10.128.0.7/23]
10:38:39 | openshift-authentication-operator | kubelet | authentication-operator-6fcbcf79b6-5h2gz | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0e9ab39a035468cdc17b59fe61931ca6df3509099b6c2267338215499105524"
10:38:39 | openshift-etcd-operator | kubelet | etcd-operator-859798c7f4-kb5lq | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b"
10:38:39 | openshift-kube-storage-version-migrator-operator | multus | kube-storage-version-migrator-operator-5999bc745c-nqrdg | AddedInterface | Add eth0 [10.128.0.9/23]
10:38:39 | openshift-kube-scheduler-operator | kubelet | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369"
10:38:39 | openshift-cluster-storage-operator | multus | csi-snapshot-controller-operator-76dfcb9b88-lt78x | AddedInterface | Add eth0 [10.130.0.2/23]
10:38:39 | openshift-controller-manager-operator | multus | openshift-controller-manager-operator-7cbb8486b4-2jgvl | AddedInterface | Add eth0 [10.128.0.10/23]
10:38:39 | openshift-kube-apiserver-operator | multus | kube-apiserver-operator-c74696d8d-gsntz | AddedInterface | Add eth0 [10.128.0.6/23]
10:38:39 | openshift-insights | multus | insights-operator-59575c9c9d-8bw8w | AddedInterface | Add eth0 [10.128.0.4/23]
10:38:39 | openshift-authentication-operator | multus | authentication-operator-6fcbcf79b6-5h2gz | AddedInterface | Add eth0 [10.128.0.11/23]
10:38:39 | openshift-config-operator | multus | openshift-config-operator-697f744f69-g6ck2 | AddedInterface | Add eth0 [10.130.0.6/23]
10:38:39 | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7932986aa8485857de987ccfd248146e06d18cf52f65d28fa607f47cab34a49"
10:38:39 | openshift-insights | kubelet | insights-operator-59575c9c9d-8bw8w | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a380de2617755685ef62a43faefbe0041734ca0b213143bc45d29bfd0aec7a24"
10:38:39 | openshift-controller-manager-operator | kubelet | openshift-controller-manager-operator-7cbb8486b4-2jgvl | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f22313adee458c064e331dc1dbd0846895ed6ad3e1607f0b523d5a89ab34ed96"
10:38:39 | openshift-apiserver-operator | kubelet | openshift-apiserver-operator-56c79799ff-ntg2n | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46c56da38fc8caba5ee400ff9aea7dd30fa6cc23283517e912e0a054250b200c"
10:38:39 | openshift-cluster-storage-operator | multus | cluster-storage-operator-bc56464-b7kxp | AddedInterface | Add eth0 [10.128.0.8/23]
10:38:40 | openshift-cluster-storage-operator | kubelet | cluster-storage-operator-bc56464-b7kxp | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1c3255b55cd57639781c529c27f98ee3e8101c31265cbd6cc63896eee70d63c"
10:38:40 | openshift-kube-storage-version-migrator-operator | kubelet | kube-storage-version-migrator-operator-5999bc745c-nqrdg | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10ba9f188dfb4e2a087bce84c8b0efb93313cd194aa5d4542d478c1154210cf7"
10:38:40 | openshift-cluster-node-tuning-operator | kubelet | cluster-node-tuning-operator-c56cc8888-p6tkq | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2"
10:38:40 | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b41284d70a2778e5efd5c77dba208b3512ff412dc0d7ba1f487fcde6502eaf6"
10:38:40 | openshift-kube-apiserver-operator | kubelet | kube-apiserver-operator-c74696d8d-gsntz | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5"
10:38:41 | openshift-multus | kubelet | multus-8vk2z | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" in 3.354836358s
10:38:41 | openshift-machine-config-operator | daemonset-controller | machine-config-daemon | SuccessfulCreate | Created pod: machine-config-daemon-plfj2
10:38:41 | openshift-machine-config-operator | default-scheduler | machine-config-daemon-cfgj5 | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-daemon-cfgj5 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:41 | openshift-machine-config-operator | daemonset-controller | machine-config-daemon | SuccessfulCreate | Created pod: machine-config-daemon-cbhkk
10:38:41 | openshift-machine-config-operator | daemonset-controller | machine-config-daemon | SuccessfulCreate | Created pod: machine-config-daemon-cfgj5
10:38:41 | openshift-machine-config-operator | default-scheduler | machine-config-daemon-cbhkk | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-daemon-cbhkk to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:41 | openshift-multus | kubelet | multus-8vk2z | Started | Started container whereabouts-cni-bincopy
10:38:41 | openshift-multus | kubelet | multus-8vk2z | Created | Created container whereabouts-cni-bincopy
10:38:41 | openshift-multus | kubelet | multus-8vk2z | Created | Created container whereabouts-cni
10:38:41 | openshift-machine-config-operator | endpoint-controller | machine-config-daemon | FailedToUpdateEndpoint | Failed to update endpoint openshift-machine-config-operator/machine-config-daemon: Operation cannot be fulfilled on endpoints "machine-config-daemon": the object has been modified; please apply your changes to the latest version and try again
10:38:41 | openshift-multus | kubelet | multus-8vk2z | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" already present on machine
10:38:41 | openshift-machine-config-operator | default-scheduler | machine-config-daemon-plfj2 | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-daemon-plfj2 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:42 | openshift-multus | kubelet | multus-8vk2z | Started | Started container whereabouts-cni
10:38:42 | openshift-multus | kubelet | multus-8vk2z | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:38:42 | openshift-multus | kubelet | multus-8vk2z | Created | Created container kube-multus
10:38:43 | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-operator-76dfcb9b88-lt78x | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b798a2aeb0d74b2644b613801feb06411c9f6444e4be462073b72df5bb35451" in 3.929207299s
10:38:43 | openshift-kube-scheduler-operator | kubelet | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" in 4.455605803s
10:38:43 | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7932986aa8485857de987ccfd248146e06d18cf52f65d28fa607f47cab34a49" in 4.121177338s
10:38:43 | openshift-etcd-operator | kubelet | etcd-operator-859798c7f4-kb5lq | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" in 4.248975237s
10:38:43 | openshift-kube-controller-manager-operator | kubelet | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" in 4.462028277s
10:38:43 | openshift-multus | kubelet | multus-8vk2z | Started | Started container kube-multus
10:38:44 (x2) | openshift-config-operator | config-operator-status-controller-statussyncer_config-operator | openshift-config-operator | OperatorVersionChanged | clusteroperator/config-operator version "operator" changed from "" to "4.6.52"
10:38:44 | openshift-config-operator | config-operator-configoperatorcontroller | openshift-config-operator | ConfigOperatorStatusChanged | Operator conditions defaulted: [{OperatorAvailable True 2022-03-08 10:38:44 +0000 UTC AsExpected } {OperatorProgressing False 2022-03-08 10:38:44 +0000 UTC AsExpected } {OperatorUpgradeable True 2022-03-08 10:38:44 +0000 UTC AsExpected }]
10:38:44 | openshift-config-operator | config-operator-status-controller-statussyncer_config-operator | openshift-config-operator | OperatorStatusChanged | Status for clusteroperator/config-operator changed: Degraded set to Unknown (""),Progressing set to False (""),Available set to True (""),Upgradeable set to True (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "configs" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-operator"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:38:44 | openshift-config-operator | config-operator | config-operator-lock | LeaderElection | openshift-config-operator-697f744f69-g6ck2_95f0dc0a-18e9-423a-9f37-f13fce94e20f became leader
10:38:44 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator-lock | LeaderElection | kube-controller-manager-operator-7bb9b6d8b8-h8hjn_e22ff167-f449-4e23-bb30-1ecfcdee2fd9 became leader
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator | CABundleUpdateRequired | "csr-controller-signer-ca" in "openshift-kube-controller-manager-operator" requires a new cert
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-config-observer-configobserver | kube-controller-manager-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ + "extendedArguments": map[string]interface{}{ + "cluster-cidr": []interface{}{string("10.128.0.0/14")}, + "cluster-name": []interface{}{string("case03116926-lsn5h")}, + "feature-gates": []interface{}{ + string("APIPriorityAndFairness=true"), + string("RotateKubeletServerCertificate=true"), + string("SupportPodPidsLimit=true"), + string("NodeDisruptionExclusion=true"), + string("ServiceNodeExclusion=true"), + string("SCTPSupport=true"), + string("LegacyNodeRoleBehavior=false"), + }, + "service-cluster-ip-range": []interface{}{string("172.30.0.0/16")}, + }, }
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 1 triggered by "configmap \"kube-controller-manager-pod\" not found"
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-config-observer-configobserver | kube-controller-manager-operator | ObserveFeatureFlagsUpdated | Updated extendedArguments.feature-gates to APIPriorityAndFairness=true,RotateKubeletServerCertificate=true,SupportPodPidsLimit=true,NodeDisruptionExclusion=true,ServiceNodeExclusion=true,SCTPSupport=true,LegacyNodeRoleBehavior=false
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded changed from Unknown to False ("")
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | RevisionTriggered | new revision 1 triggered by "configmap \"etcd-pod\" not found"
10:38:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 1 triggered by "configmap \"kube-scheduler-pod\" not found"
10:38:45 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorVersionChanged | clusteroperator/kube-controller-manager version "raw-internal" changed from "" to "4.6.52"
10:38:45 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator-lock | LeaderElection | csi-snapshot-controller-operator-76dfcb9b88-lt78x_8858c97c-892c-4579-8efb-1aaeb1859c02 became leader
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorVersionChanged | clusteroperator/etcd version "raw-internal" changed from "" to "4.6.52"
10:38:45 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-loggingsyncer | csi-snapshot-controller-operator | OperatorLogLevelChange | Operator log level changed from "Debug" to "Normal"
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorVersionChanged | clusteroperator/etcd version "operator" changed from "" to "4.6.52"
10:38:45 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorStatusChanged | Status for clusteroperator/csi-snapshot-controller changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"" "namespaces" "" "openshift-cluster-storage-operator"} {"" "namespaces" "" "openshift-cluster-storage-operator"} {"operator.openshift.io" "csisnapshotcontrollers" "" "cluster"}]
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Upgradeable changed from Unknown to True ("")
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to True (""),status.relatedObjects changed from [{"operator.openshift.io" "kubecontrollermanagers" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-kube-controller-manager"} {"" "namespaces" "" "openshift-kube-controller-manager-operator"} {"" "nodes" "" ""} {"certificates.k8s.io" "certificatesigningrequests" "" ""}] to [{"operator.openshift.io" "kubecontrollermanagers" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-kube-controller-manager"} {"" "namespaces" "" "openshift-kube-controller-manager-operator"} {"certificates.k8s.io" "certificatesigningrequests" "" ""} {"" "nodes" "" ""}],status.versions changed from [] to [{"raw-internal" "4.6.52"}]
10:38:45 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorStatusChanged | Status for clusteroperator/csi-snapshot-controller changed: Degraded changed from Unknown to False ("")
10:38:45 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator | CustomResourceDefinitionCreated | Created CustomResourceDefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io because it was missing
10:38:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Upgradeable changed from Unknown to True ("")
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded set to False ("EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced"),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.versions changed from [] to [{"raw-internal" "4.6.52"} {"operator" "4.6.52"}]
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator | openshift-cluster-etcd-operator-lock | LeaderElection | etcd-operator-859798c7f4-kb5lq_dd16536e-bc48-4f74-ba3f-25580e5d284a became leader
10:38:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded changed from Unknown to False ("InstallerControllerDegraded: missing required resources: [configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0]"),Progressing changed from Unknown to False (""),Available changed from Unknown to False ("StaticPodsAvailable: 0 nodes are active; ")
10:38:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [{"operator.openshift.io" "kubeschedulers" "" "cluster"} {"config.openshift.io" "schedulers" "" ""} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-kube-scheduler-operator"} {"" "namespaces" "" "openshift-kube-scheduler"} {"controlplane.operator.openshift.io" "podnetworkconnectivitychecks" "openshift-kube-scheduler" ""}] to [{"operator.openshift.io" "kubeschedulers" "" "cluster"} {"config.openshift.io" "schedulers" "" ""} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-kube-scheduler"} {"" "namespaces" "" "openshift-kube-scheduler-operator"} {"controlplane.operator.openshift.io" "podnetworkconnectivitychecks" "openshift-kube-apiserver" ""}],status.versions changed from [] to [{"raw-internal" "4.6.52"}]
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]",Progressing changed from Unknown to False (""),Available changed from Unknown to False ("StaticPodsAvailable: 0 nodes are active; ")
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-nodecontroller | kube-controller-manager-operator | MasterNodeObserved | Observed new master node master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-nodecontroller | kube-controller-manager-operator | MasterNodeObserved | Observed new master node master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-nodecontroller | kube-controller-manager-operator | MasterNodeObserved | Observed new master node master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-lock | LeaderElection | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k_e20a6013-bbbe-418f-86f0-352af452cf98 became leader
10:38:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorVersionChanged | clusteroperator/kube-scheduler version "raw-internal" changed from "" to "4.6.52"
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-loggingsyncer | etcd-operator | OperatorLogLevelChange | Operator log level changed from "Debug" to "Normal"
10:38:45 | openshift-config-operator | config-operator-status-controller-statussyncer_config-operator | openshift-config-operator | OperatorStatusChanged | Status for clusteroperator/config-operator changed: Degraded changed from Unknown to False ("")
10:38:45 | openshift-config-operator | config-operator-status-controller-statussyncer_config-operator | openshift-config-operator | OperatorStatusChanged | Status for clusteroperator/config-operator changed: Degraded set to False (""),Progressing set to False (""),Available set to True (""),Upgradeable set to True (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "configs" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-operator"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-host-etcd-2-endpoints-controller2-hostendpoints2controller | etcd-operator | EndpointsUpdated | Updated endpoints/host-etcd-2 -n openshift-etcd because it changed: {"subsets":[{"addresses":[{"ip":"10.0.94.219","nodeName":"master-2.case03116926.lab.upshift.rdu2.redhat.com"},{"ip":"10.0.94.119","nodeName":"master-0.case03116926.lab.upshift.rdu2.redhat.com"},{"ip":"10.0.95.76","nodeName":"master-1.case03116926.lab.upshift.rdu2.redhat.com"}],"ports":[{"name":"etcd","port":2379,"protocol":"TCP"}]}]}
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-endpoints-controller-etcdendpointscontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/etcd-endpoints -n openshift-etcd: cause by changes in data.MTAuMC45NC4xMTk,data.MTAuMC45NC4yMTk,data.MTAuMC45NS43Ng
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-member-observer-controller-etcdmemberscontroller | etcd-operator | ReportEtcdMembersErrorUpdatingStatus | etcds.operator.openshift.io "cluster" not found
10:38:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values"
10:38:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [{"operator.openshift.io" "kubecontrollermanagers" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-kube-controller-manager"} {"" "namespaces" "" "openshift-kube-controller-manager-operator"} {"" "nodes" "" ""} {"certificates.k8s.io" "certificatesigningrequests" "" ""}] to [{"operator.openshift.io" "kubecontrollermanagers" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-kube-controller-manager"} {"" "namespaces" "" "openshift-kube-controller-manager-operator"} {"certificates.k8s.io" "certificatesigningrequests" "" ""} {"" "nodes" "" ""}],status.versions changed from [] to [{"raw-internal" "4.6.52"}]
10:38:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-serving-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-serving-metrics-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-serving-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-peer-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "" to "NodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)"
10:38:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-1 -n openshift-kube-controller-manager because it was missing
10:38:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/csr-controller-signer-ca -n openshift-kube-controller-manager-operator because it was missing
10:38:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator | TargetUpdateRequired | "csr-signer" in "openshift-kube-controller-manager-operator" requires a new target cert/key pair: missing notAfter
10:38:46 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator | CustomResourceDefinitionCreated | Created CustomResourceDefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io because it was missing
10:38:46 (x16) | openshift-multus | kubelet | network-metrics-daemon-sw48w | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:38:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/revision-status-1 -n openshift-etcd because it was missing
10:38:46 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/revision-status-1 -n openshift-kube-scheduler because it was missing
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Upgradeable changed from Unknown to True ("")
10:38:47 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Progressing message changed from "" to "NodeInstallerProgressing: 3 nodes are at revision 0",Available message changed from "StaticPodsAvailable: 0 nodes are active; " to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0"
10:38:47 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-backingresourcecontroller-staticresourcecontroller | openshift-kube-scheduler-operator | ServiceAccountCreated | Created ServiceAccount/installer-sa -n openshift-kube-scheduler because it was missing
10:38:47 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/config -n openshift-kube-scheduler because it was missing
10:38:47 (x11) | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | RequiredInstallerResourcesMissing | configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0
10:38:47 | openshift-service-ca-operator | default-scheduler | service-ca-operator-6fd6454f86-5bbh8 | Scheduled | Successfully assigned openshift-service-ca-operator/service-ca-operator-6fd6454f86-5bbh8 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:47 | openshift-machine-api | default-scheduler | machine-api-operator-784c77b6d4-xfllc | Scheduled | Successfully assigned openshift-machine-api/machine-api-operator-784c77b6d4-xfllc to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:47 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator | CustomResourceDefinitionCreated | Created CustomResourceDefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io because it was missing
10:38:47 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0]" to "InstallerControllerDegraded: missing required resources: [configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)"
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-ca-bundle -n openshift-etcd because it was missing
10:38:47 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator | SecretCreated | Created Secret/csr-signer -n openshift-kube-controller-manager-operator because it was missing
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-peer-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:47 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-nodecontroller | openshift-kube-scheduler-operator | MasterNodeObserved | Observed new master node master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:47 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-nodecontroller | openshift-kube-scheduler-operator | MasterNodeObserved | Observed new master node master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:47 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-nodecontroller | openshift-kube-scheduler-operator | MasterNodeObserved | Observed new master node master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:47 (x9) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMissing | no observedConfig
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-serving-metrics-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:47 | openshift-machine-api | default-scheduler | cluster-autoscaler-operator-557599ccf8-lchg2 | Scheduled | Successfully assigned openshift-machine-api/cluster-autoscaler-operator-557599ccf8-lchg2 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-serving-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/etcd-ca-bundle -n openshift-etcd-operator: cause by changes in data.ca-bundle.crt
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-serving-metrics-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:47 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-peer-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:38:48 | openshift-cluster-storage-operator | deployment-controller | csi-snapshot-controller | ScalingReplicaSet | Scaled up replica set csi-snapshot-controller-7cb4cb644d to 1
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator | DeploymentUpdated | Updated Deployment.apps/csi-snapshot-controller -n openshift-cluster-storage-operator because it changed
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorStatusChanged | Status for clusteroperator/csi-snapshot-controller changed: Progressing message changed from "Progressing: Waiting for Deployment to act on changes" to "Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods"
10:38:48 | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Started | Started container cluster-monitoring-operator
10:38:48 | openshift-cluster-node-tuning-operator | kubelet | cluster-node-tuning-operator-c56cc8888-p6tkq | Created | Created container cluster-node-tuning-operator
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-backingresourcecontroller-staticresourcecontroller | etcd-operator | ServiceAccountCreated | Created ServiceAccount/installer-sa -n openshift-etcd because it was missing
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-backingresourcecontroller-staticresourcecontroller | etcd-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:openshift-etcd-installer because it was missing
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-peer-client-ca -n openshift-etcd because it was missing
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcdstaticresources-staticresourcecontroller | etcd-operator | ServiceAccountCreated | Created ServiceAccount/etcd-sa -n openshift-etcd because it was missing
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-peer -n openshift-etcd because it was missing
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-serving -n openshift-etcd because it was missing
10:38:48 | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Created | Created container cluster-monitoring-operator
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-metrics-proxy-client-ca -n openshift-etcd because it was missing
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-backingresourcecontroller-staticresourcecontroller | kube-controller-manager-operator | ServiceAccountCreated | Created ServiceAccount/installer-sa -n openshift-kube-controller-manager because it was missing
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorStatusChanged | Status for clusteroperator/csi-snapshot-controller changed: Progressing changed from Unknown to True ("Progressing: Waiting for Deployment to act on changes"),Available changed from Unknown to False ("Available: Waiting for Deployment to deploy csi-snapshot-controller pods"),Upgradeable changed from Unknown to True ("")
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorStatusChanged | Status for clusteroperator/csi-snapshot-controller changed: status.versions changed from [] to [{"csi-snapshot-controller" "4.6.52"} {"operator" "4.6.52"}]
10:38:48 | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b41284d70a2778e5efd5c77dba208b3512ff412dc0d7ba1f487fcde6502eaf6" in 8.021561757s
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-election-lock-kube-controller-manager -n kube-system because it was missing
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:leader-election-lock-kube-controller-manager -n kube-system because it was missing
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorVersionChanged | clusteroperator/csi-snapshot-controller version "operator" changed from "" to "4.6.52"
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorVersionChanged | clusteroperator/csi-snapshot-controller version "csi-snapshot-controller" changed from "" to "4.6.52"
10:38:48 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-0,config-0,controller-manager-kubeconfig-0,kube-controller-cert-syncer-kubeconfig-0,kube-controller-manager-pod-0,recycler-config-0,service-ca-0,serviceaccount-ca-0, secrets: csr-signer,kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-0,service-account-private-key-0
10:38:48 | openshift-kube-apiserver-operator | kubelet | kube-apiserver-operator-c74696d8d-gsntz | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" in 8.198176963s
10:38:48 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator | DeploymentCreated | Created Deployment.apps/csi-snapshot-controller -n openshift-cluster-storage-operator because it was missing
10:38:48 | openshift-insights | kubelet | insights-operator-59575c9c9d-8bw8w | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a380de2617755685ef62a43faefbe0041734ca0b213143bc45d29bfd0aec7a24" in 8.322718356s
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-locking-kube-controller-manager -n kube-system because it was missing
10:38:48 | openshift-cluster-storage-operator | replicaset-controller | csi-snapshot-controller-7cb4cb644d | SuccessfulCreate | Created pod: csi-snapshot-controller-7cb4cb644d-wx58q
10:38:48 | openshift-cluster-storage-operator | default-scheduler | csi-snapshot-controller-7cb4cb644d-wx58q | Scheduled | Successfully assigned openshift-cluster-storage-operator/csi-snapshot-controller-7cb4cb644d-wx58q to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:48 | openshift-apiserver-operator | kubelet | openshift-apiserver-operator-56c79799ff-ntg2n | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46c56da38fc8caba5ee400ff9aea7dd30fa6cc23283517e912e0a054250b200c" in 8.810548696s
10:38:48 | openshift-cluster-node-tuning-operator | kubelet | cluster-node-tuning-operator-c56cc8888-p6tkq | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" in 7.926409306s
10:38:48 | openshift-cluster-storage-operator | kubelet | cluster-storage-operator-bc56464-b7kxp | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1c3255b55cd57639781c529c27f98ee3e8101c31265cbd6cc63896eee70d63c" in 8.036556462s
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-election-lock-cluster-policy-controller -n openshift-kube-controller-manager because it was missing
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:leader-election-lock-cluster-policy-controller -n openshift-kube-controller-manager because it was missing
10:38:48 | openshift-cluster-node-tuning-operator | kubelet | cluster-node-tuning-operator-c56cc8888-p6tkq | Started | Started container cluster-node-tuning-operator
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:sa-listing-configmaps -n openshift-kube-scheduler because it was missing
10:38:48 | openshift-controller-manager-operator | kubelet | openshift-controller-manager-operator-7cbb8486b4-2jgvl | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f22313adee458c064e331dc1dbd0846895ed6ad3e1607f0b523d5a89ab34ed96" in 8.464087311s
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:kube-scheduler:public-2 because it was missing
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-locking-kube-scheduler -n kube-system because it was missing
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-cert-syncer-kubeconfig -n openshift-kube-scheduler because it was missing
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | NamespaceUpdated | Updated Namespace/openshift-kube-scheduler because it changed
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-backingresourcecontroller-staticresourcecontroller | openshift-kube-scheduler-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:openshift-kube-scheduler-installer because it was missing
10:38:48 | openshift-kube-storage-version-migrator-operator | kubelet | kube-storage-version-migrator-operator-5999bc745c-nqrdg | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10ba9f188dfb4e2a087bce84c8b0efb93313cd194aa5d4542d478c1154210cf7" in 8.108148001s
10:38:48 | openshift-authentication-operator | kubelet | authentication-operator-6fcbcf79b6-5h2gz | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0e9ab39a035468cdc17b59fe61931ca6df3509099b6c2267338215499105524" in 8.354186352s
10:38:48 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:sa-listing-configmaps -n openshift-kube-scheduler because it was missing
10:38:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-cert-signer-controller-etcdcertsignercontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-serving-metrics -n openshift-etcd because it was missing
10:38:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig -n openshift-kube-controller-manager because it was missing
10:38:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca -n openshift-etcd because it was missing
10:38:49 | openshift-cluster-node-tuning-operator | daemonset-controller | tuned | SuccessfulCreate | Created pod: tuned-bsln7
10:38:49 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found" to "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found",Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1"),Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1"
10:38:49 | openshift-cluster-node-tuning-operator | daemonset-controller | tuned | SuccessfulCreate | Created pod: tuned-b8krr
10:38:49 | openshift-cluster-node-tuning-operator | daemonset-controller | tuned | SuccessfulCreate | Created pod: tuned-9v69g
10:38:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | SecretUpdated | Updated Secret/etcd-client -n openshift-etcd-operator because it changed
10:38:49 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config -n openshift-kube-controller-manager because it was missing
10:38:49 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "NodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found"
10:38:49 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-backingresourcecontroller-staticresourcecontroller | kube-controller-manager-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:openshift-kube-controller-manager-installer because it was missing
10:38:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcdstaticresources-staticresourcecontroller | etcd-operator | ServiceUpdated | Updated Service/etcd -n openshift-etcd because it changed
10:38:49 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "InstallerControllerDegraded: missing required resources: [configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found"
10:38:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | RequiredInstallerResourcesMissing | configmaps: etcd-metrics-proxy-serving-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0
10:38:49 (x15) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | RequiredInstallerResourcesMissing | configmaps: config-0,kube-scheduler-cert-syncer-kubeconfig-0,kube-scheduler-pod-0,scheduler-kubeconfig-0,serviceaccount-ca-0, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-0
10:38:49 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key -n openshift-kube-controller-manager because it was missing
10:38:49 | openshift-cluster-node-tuning-operator | kubelet | tuned-9v69g | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2"
10:38:49 | openshift-cluster-node-tuning-operator | default-scheduler | tuned-bsln7 | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/tuned-bsln7 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:49 (x7) | openshift-multus | kubelet | network-metrics-daemon-sw48w | FailedMount | MountVolume.SetUp failed for volume "metrics-certs" : secret "metrics-daemon-secret" not found
10:38:49 | openshift-cluster-node-tuning-operator | kubelet | tuned-b8krr | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2"
10:38:49 | openshift-cluster-node-tuning-operator | default-scheduler | tuned-b8krr | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/tuned-b8krr to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:49 | openshift-cluster-node-tuning-operator | default-scheduler | tuned-9v69g | Scheduled | Successfully assigned openshift-cluster-node-tuning-operator/tuned-9v69g to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-metrics-proxy-serving-ca -n openshift-etcd because it was missing
10:38:49 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca -n openshift-kube-scheduler because it was missing
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-status-controller-statussyncer_kube-storage-version-migrator | kube-storage-version-migrator-operator | OperatorStatusChanged | Status for clusteroperator/kube-storage-version-migrator changed: Progressing changed from True to False ("")
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-status-controller-statussyncer_kube-storage-version-migrator | kube-storage-version-migrator-operator | OperatorStatusChanged | Status for clusteroperator/kube-storage-version-migrator changed: Degraded changed from Unknown to False (""),Progressing changed from Unknown to True ("Progressing: deployment/migrator.openshift-kube-storage-version-migrator: observed generation is 0, desired generation is 1.\nProgressing: kubestorageversionmigrators/cluster: observed generation is 0, desired generation is 1."),Available changed from Unknown to False ("Available: deployment/migrator.openshift-kube-storage-version-migrator: no replicas are available")
10:38:50 | openshift-cluster-storage-operator | multus | csi-snapshot-controller-7cb4cb644d-wx58q | AddedInterface | Add eth0 [10.128.0.13/23]
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-workload-controller | kube-storage-version-migrator-operator | DeploymentCreated | Created Deployment.apps/migrator -n openshift-kube-storage-version-migrator because it was missing
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-status-controller-statussyncer_kube-storage-version-migrator | kube-storage-version-migrator-operator | OperatorStatusChanged | Status for clusteroperator/kube-storage-version-migrator changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "kubestorageversionmigrators" "" "cluster"} {"" "namespaces" "" "openshift-kube-storage-version-migrator"} {"" "namespaces" "" "openshift-kube-storage-version-migrator-operator"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:38:50 (x2) | openshift-cluster-storage-operator | cluster-storage-operator-status-controller-statussyncer_storage | cluster-storage-operator | OperatorStatusChanged | Status for clusteroperator/storage changed: Degraded changed from Unknown to False ("")
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-status-controller-statussyncer_kube-storage-version-migrator | kube-storage-version-migrator-operator | OperatorVersionChanged | clusteroperator/kube-storage-version-migrator version "operator" changed from "" to "4.6.52"
10:38:50 | openshift-cluster-storage-operator | cluster-storage-operator-status-controller-statussyncer_storage | cluster-storage-operator | OperatorStatusChanged | Status for clusteroperator/storage changed: Degraded set to False (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"" "namespaces" "" "openshift-cluster-storage-operator"} {"" "namespaces" "" "openshift-cluster-csi-drivers"} {"" "namespaces" "" "openshift-manila-csi-driver"} {"operator.openshift.io" "storages" "" "cluster"} {"operator.openshift.io" "clustercsidrivers" "" "ebs.csi.aws.com"} {"operator.openshift.io" "clustercsidrivers" "" "csi.ovirt.org"} {"operator.openshift.io" "clustercsidrivers" "" "manila.csi.openstack.org"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-lock | LeaderElection | kube-storage-version-migrator-operator-5999bc745c-nqrdg_a843da62-0797-446e-a7b5-9827a00980ff became leader
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-workload-controller | kube-storage-version-migrator-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/storage-version-migration-migrator because it was missing
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-workload-controller | kube-storage-version-migrator-operator | NamespaceCreated | Created Namespace/openshift-kube-storage-version-migrator because it was missing
10:38:50 | openshift-authentication-operator | cluster-authentication-operator | cluster-authentication-operator-lock | LeaderElection | authentication-operator-6fcbcf79b6-5h2gz_2ff4ece2-edb7-4628-b7dd-4a77a5b65c81 became leader
10:38:50 | openshift-cluster-storage-operator | cluster-storage-operator-status-controller-statussyncer_storage | cluster-storage-operator | OperatorStatusChanged | Status for clusteroperator/storage changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"" "namespaces" "" "openshift-cluster-storage-operator"} {"" "namespaces" "" "openshift-cluster-csi-drivers"} {"" "namespaces" "" "openshift-manila-csi-driver"} {"operator.openshift.io" "storages" "" "cluster"} {"operator.openshift.io" "clustercsidrivers" "" "ebs.csi.aws.com"} {"operator.openshift.io" "clustercsidrivers" "" "csi.ovirt.org"} {"operator.openshift.io" "clustercsidrivers" "" "manila.csi.openstack.org"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:38:50 (x2) | openshift-cluster-storage-operator | cluster-storage-operator-status-controller-statussyncer_storage | cluster-storage-operator | OperatorVersionChanged | clusteroperator/storage version "operator" changed from "" to "4.6.52"
10:38:50 | openshift-cluster-storage-operator | cluster-storage-operator-status-controller-statussyncer_storage | cluster-storage-operator | OperatorStatusChanged | Status for clusteroperator/storage changed: Progressing changed from Unknown to False (""),Available changed from Unknown to True ("DefaultStorageClassControllerAvailable: No default StorageClass for this platform")
10:38:50 | openshift-kube-storage-version-migrator | deployment-controller | migrator | ScalingReplicaSet | Scaled up replica set migrator-85c499d985 to 1
10:38:50 | openshift-kube-storage-version-migrator | replicaset-controller | migrator-85c499d985 | SuccessfulCreate | Created pod: migrator-85c499d985-hznj9
10:38:50 | openshift-cluster-storage-operator | cluster-storage-operator | cluster-storage-operator-lock | LeaderElection | cluster-storage-operator-bc56464-b7kxp_8b52a00e-70bd-4712-8dd0-003a5dfff082 became leader
10:38:50 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | ServiceCreated | Created Service/scheduler -n openshift-kube-scheduler because it was missing
10:38:50 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ServiceCreated | Created Service/kube-controller-manager -n openshift-kube-controller-manager because it was missing
10:38:50 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/revision-status-2 -n openshift-kube-scheduler because it was missing
10:38:50 | openshift-etcd-operator | openshift-cluster-etcd-operator-resource-sync-controller-resourcesynccontroller | etcd-operator | SecretCreated | Created Secret/etcd-client -n openshift-etcd because it was missing
10:38:50 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found" to "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found"
10:38:50 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-workload-controller | kube-storage-version-migrator-operator | ServiceAccountCreated | Created ServiceAccount/kube-storage-version-migrator-sa -n openshift-kube-storage-version-migrator because it was missing
10:38:50 | openshift-service-ca-operator | multus | service-ca-operator-6fd6454f86-5bbh8 | AddedInterface | Add eth0 [10.128.0.12/23]
10:38:51 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator-lock | LeaderElection | openshift-apiserver-operator-56c79799ff-ntg2n_e1ec0e9a-5d06-4cc5-b3c8-88ed4e03705e became leader
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Degraded changed from Unknown to False ("")
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "openshiftcontrollermanagers" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-controller-manager-operator"} {"" "namespaces" "" "openshift-controller-manager"}]
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator-config-observer-configobserver | openshift-controller-manager-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ + "build": map[string]interface{}{ + "buildDefaults": map[string]interface{}{"resources": map[string]interface{}{}}, + "imageTemplateFormat": map[string]interface{}{ + "format": string("quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17d408aaf7b8a00c66895a1e50b3c8529be76a342cfa84c478fa2af72576f67f"), + }, + }, + "deployer": map[string]interface{}{ + "imageTemplateFormat": map[string]interface{}{ + "format": string("quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31644151defbf36a8f70424531a0d83fa238a0fe2bea2fbcd1a1db527e1b78b2"), + }, + }, + "ingress": map[string]interface{}{"ingressIPNetworkCIDR": string("")}, }
10:38:51 (x2) | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorVersionChanged | clusteroperator/openshift-apiserver version "operator" changed from "" to "4.6.52"
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:tokenreview-openshift-controller-manager because it was missing
10:38:51 | openshift-monitoring | deployment-controller | kube-state-metrics | ScalingReplicaSet | Scaled up replica set kube-state-metrics-6c8b757584 to 1
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SignerUpdateRequired | "localhost-recovery-serving-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: missing notAfter
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "localhost-serving-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:openshift-controller-manager:ingress-to-route-controller because it was missing
10:38:51 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found"
10:38:51 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "loadbalancer-serving-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:tokenreview-openshift-controller-manager because it was missing
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "kube-apiserver-aggregator-client-ca" in "openshift-config-managed" requires a new cert
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:openshift-controller-manager:ingress-to-route-controller because it was missing
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:openshift-controller-manager because it was missing
10:38:51 (x4) | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "kube-control-plane-signer-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:38:51 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-1 -n openshift-kube-controller-manager: cause by changes in data.status
10:38:51 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-2 -n openshift-kube-controller-manager because it was missing
10:38:51 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config -n openshift-kube-controller-manager because it was missing
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SignerUpdateRequired | "node-system-admin-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: missing notAfter
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorVersionChanged | clusteroperator/kube-apiserver version "raw-internal" changed from "" to "4.6.52"
10:38:51 | openshift-monitoring | replicaset-controller | kube-state-metrics-6c8b757584 | SuccessfulCreate | Created pod: kube-state-metrics-6c8b757584-2b2dg
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:openshift-controller-manager because it was missing
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 1 triggered by "configmap \"kube-apiserver-pod\" not found"
10:38:51 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator-lock | LeaderElection | openshift-controller-manager-operator-7cbb8486b4-2jgvl_4bb3e4cd-39ed-4c00-99dc-f744687b9587 became leader
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator | kube-apiserver-operator-lock | LeaderElection | kube-apiserver-operator-c74696d8d-gsntz_cec9ace0-2e88-40db-9483-4564881eff69 became leader
10:38:51 (x2) | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "openshiftapiservers" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-apiserver-operator"} {"" "namespaces" "" "openshift-apiserver"} {"" "namespaces" "" "openshift-etcd-operator"} {"" "endpoints" "openshift-etcd" "host-etcd-2"} {"controlplane.operator.openshift.io" "podnetworkconnectivitychecks" "openshift-apiserver" ""} {"apiregistration.k8s.io" "apiservices" "" "v1.apps.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.authorization.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.build.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.image.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.oauth.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.project.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.quota.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.route.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.security.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.template.openshift.io"} {"apiregistration.k8s.io" "apiservices" "" "v1.user.openshift.io"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "service-network-serving-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:38:51 | openshift-cluster-node-tuning-operator | kubelet | tuned-bsln7 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" already present on machine
10:38:51 | openshift-monitoring | deployment-controller | prometheus-operator | ScalingReplicaSet | Scaled up replica set prometheus-operator-6db8fc66c7 to 1
10:38:51 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available changed from Unknown to False ("OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints")
10:38:51 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded changed from Unknown to False ("OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found")
10:38:51 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-1 -n openshift-kube-scheduler: cause by changes in data.status
10:38:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded set to False ("OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found"),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "authentications" "" "cluster"} {"config.openshift.io" "authentications" "" "cluster"} {"config.openshift.io" "infrastructures" "" "cluster"} {"config.openshift.io" "oauths" "" "cluster"} {"route.openshift.io" "routes" "openshift-authentication" "oauth-openshift"} {"" "services" "openshift-authentication" "oauth-openshift"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-authentication"} {"" "namespaces" "" "openshift-authentication-operator"} {"" "namespaces" "" "openshift-ingress"} {"" "namespaces" "" "openshift-oauth-apiserver"}]
10:38:51 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Upgradeable changed from Unknown to True ("")
10:38:51 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Available changed from Unknown to False ("APIServicesAvailable: endpoints \"api\" not found")
10:38:51 | openshift-cluster-storage-operator | cluster-storage-operator-status-controller-statussyncer_storage | cluster-storage-operator | OperatorStatusChanged | Status for clusteroperator/storage changed: Upgradeable changed from Unknown to True ("")
10:38:51 | openshift-monitoring | deployment-controller | openshift-state-metrics | ScalingReplicaSet | Scaled up replica set openshift-state-metrics-6d99649c49 to 1
10:38:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "authentications" "" "cluster"} {"config.openshift.io" "authentications" "" "cluster"} {"config.openshift.io" "infrastructures" "" "cluster"} {"config.openshift.io" "oauths" "" "cluster"} {"route.openshift.io" "routes" "openshift-authentication" "oauth-openshift"} {"" "services" "openshift-authentication" "oauth-openshift"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-authentication"} {"" "namespaces" "" "openshift-authentication-operator"} {"" "namespaces" "" "openshift-ingress"} {"" "namespaces" "" "openshift-oauth-apiserver"}]
10:38:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "kube-apiserver-to-kubelet-client-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:38:51 | openshift-monitoring | replicaset-controller | openshift-state-metrics-6d99649c49 | SuccessfulCreate | Created pod: openshift-state-metrics-6d99649c49-4b6cb
10:38:52 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to True (""),status.versions changed from [] to [{"raw-internal" "4.6.52"}]
10:38:52 | openshift-cluster-node-tuning-operator | kubelet | tuned-bsln7 | Created | Created container tuned
10:38:52 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:kube-scheduler-recovery because it was missing
10:38:52 | openshift-cluster-node-tuning-operator | kubelet | tuned-bsln7 | Started | Started container tuned
10:38:52 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleCreateFailed | Failed to create Role.rbac.authorization.k8s.io/system:openshift:leader-locking-openshift-controller-manager -n openshift-controller-manager: namespaces "openshift-controller-manager" not found
10:38:52 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | ServiceAccountCreated | Created ServiceAccount/openshift-kube-scheduler-sa -n openshift-kube-scheduler because it was missing
10:38:52 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/scheduler-kubeconfig -n openshift-kube-scheduler because it was missing
10:38:52 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:kube-controller-manager-recovery because it was missing
10:38:52 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config -n openshift-kube-controller-manager because it was missing
10:38:52 | openshift-service-ca-operator | kubelet | service-ca-operator-6fd6454f86-5bbh8 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf9539d6f39fdfc4049ed5c3f16ed3b9b97c11408b9b36fc7c2090d1cbddf178"
10:38:52 | openshift-monitoring | default-scheduler | node-exporter-kcxrx | Scheduled | Successfully assigned openshift-monitoring/node-exporter-kcxrx to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:52 | openshift-monitoring | default-scheduler | node-exporter-l5q6v | Scheduled | Successfully assigned openshift-monitoring/node-exporter-l5q6v to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:52 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from Unknown to False ("")
10:38:52 | openshift-monitoring | replicaset-controller | prometheus-operator-6db8fc66c7 | SuccessfulCreate | Created pod: prometheus-operator-6db8fc66c7-rgvbc
10:38:52 | openshift-monitoring | daemonset-controller | node-exporter | SuccessfulCreate | Created pod: node-exporter-l5q6v
10:38:52 | openshift-monitoring | default-scheduler | node-exporter-vrvq5 | Scheduled | Successfully assigned openshift-monitoring/node-exporter-vrvq5 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:52 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleBindingCreateFailed | Failed to create RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-locking-openshift-controller-manager -n openshift-controller-manager: namespaces "openshift-controller-manager" not found
10:38:52 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | NamespaceCreated | Created Namespace/openshift-controller-manager because it was missing
10:38:52 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:leader-locking-openshift-controller-manager -n kube-system because it was missing
10:38:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints"
10:38:52 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found"
10:38:52 (x11) | openshift-authentication-operator | oauth-apiserver-manage-oauth-api-controller-mangeoauthapicontroller | authentication-operator | PrereqNotReady | OAuth API servers are not ready. APIServicesAvailable condition is missing
10:38:52 | openshift-monitoring | daemonset-controller | node-exporter | SuccessfulCreate | Created pod: node-exporter-kcxrx
10:38:52 | openshift-monitoring | daemonset-controller | node-exporter | SuccessfulCreate | Created pod: node-exporter-vrvq5
10:38:52 | openshift-monitoring | default-scheduler | prometheus-operator-6db8fc66c7-rgvbc | Scheduled | Successfully assigned openshift-monitoring/prometheus-operator-6db8fc66c7-rgvbc to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:52 | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-7cb4cb644d-wx58q | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4a090f3ea32ac6d7f815bf1c7f58c589436bc80abf968fa9bac3f5e21e83744"
10:38:52 | openshift-cluster-node-tuning-operator | kubelet | tuned-b8krr | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" in 2.933575411s
10:38:52 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ServiceAccountCreated | Created ServiceAccount/kube-controller-manager-sa -n openshift-kube-controller-manager because it was missing
10:38:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:kube-controller-manager:gce-cloud-provider because it was missing
10:38:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-nodecontroller | kube-apiserver-operator | MasterNodeObserved | Observed new master node master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-nodecontroller | kube-apiserver-operator | MasterNodeObserved | Observed new master node master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:53 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/revision-status-2 -n openshift-etcd because it was missing
10:38:53 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-1 -n openshift-etcd: cause by changes in data.status
10:38:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObserveTLSSecurityProfile | cipherSuites changed to ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256" "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"]
10:38:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-1 -n openshift-kube-apiserver because it was missing
10:38:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-nodecontroller | kube-apiserver-operator | MasterNodeObserved | Observed new master node master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObserveTLSSecurityProfile | minTLSVersion changed to VersionTLS12
10:38:53 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Upgradeable changed from Unknown to True ("")
10:38:53 | openshift-cluster-node-tuning-operator | kubelet | tuned-9v69g | Created | Created container tuned
10:38:53 | openshift-cluster-node-tuning-operator | kubelet | tuned-b8krr | Created | Created container tuned
10:38:53 | openshift-cluster-node-tuning-operator | kubelet | tuned-b8krr | Started | Started container tuned
10:38:53 | openshift-cluster-node-tuning-operator | kubelet | tuned-9v69g | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" in 3.672694284s
10:38:53 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded changed from Unknown to False (""),Progressing changed from Unknown to False ("")
10:38:53 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:sa-creating-openshift-controller-manager -n openshift-infra because it was missing
10:38:53 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-locking-openshift-controller-manager -n kube-system because it was missing
10:38:53 | openshift-monitoring | deployment-controller | telemeter-client | ScalingReplicaSet | Scaled up replica set telemeter-client-57c59fc45f to 1
10:38:53 | openshift-monitoring | replicaset-controller | telemeter-client-57c59fc45f | SuccessfulCreate | Created pod: telemeter-client-57c59fc45f-gsqrj
10:38:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:kube-controller-manager:gce-cloud-provider because it was missing
10:38:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | ServiceAccountCreated | Created ServiceAccount/localhost-recovery-client -n openshift-kube-controller-manager because it was missing
10:38:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found"
10:38:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token -n openshift-kube-controller-manager because it was missing
10:38:54 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-pod -n openshift-kube-scheduler because it was missing
10:38:54 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ + "apiServerArguments": map[string]interface{}{ + "audit-policy-file": []interface{}{string("/var/run/configmaps/audit/secure-oauth-storage-default.yaml")}, + }, + "projectConfig": map[string]interface{}{"projectRequestMessage": string("")}, + "routingConfig": map[string]interface{}{"subdomain": string("apps.case03116926.lab.upshift.rdu2.redhat.com")}, + "servingInfo": map[string]interface{}{ + "cipherSuites": []interface{}{ + string("TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"), + string("TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"), + string("TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"), + string("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"), + string("TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"), + string("TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"), + }, + "minTLSVersion": string("VersionTLS12"), + }, + "storageConfig": map[string]interface{}{ + "urls": []interface{}{ + string("https://10.0.94.119:2379"), + string("https://10.0.94.219:2379"), + string("https://10.0.95.76:2379"), + }, + }, }
10:38:54 | openshift-cluster-storage-operator | snapshot-controller-leader/csi-snapshot-controller-7cb4cb644d-wx58q | snapshot-controller-leader | LeaderElection | csi-snapshot-controller-7cb4cb644d-wx58q became leader
10:38:54 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/csr-signer-ca -n openshift-kube-controller-manager-operator because it was missing
10:38:54 | openshift-service-ca-operator | kubelet | service-ca-operator-6fd6454f86-5bbh8 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf9539d6f39fdfc4049ed5c3f16ed3b9b97c11408b9b36fc7c2090d1cbddf178" in 2.440885593s
10:38:54 | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-7cb4cb644d-wx58q | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4a090f3ea32ac6d7f815bf1c7f58c589436bc80abf968fa9bac3f5e21e83744" in 2.53377309s
10:38:54 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: csr-signer,kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]",Progressing changed from Unknown to True ("NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1"),Available changed from Unknown to False ("StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1")
10:38:54 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | ObserveTLSSecurityProfile | minTLSVersion changed to VersionTLS12
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObservedConfigChanged | Writing updated section ("oauthAPIServer") of observed config: " map[string]interface{}(\n- \tnil,\n+ \t{\n+ \t\t\"apiServerArguments\": map[string]interface{}{\n+ \t\t\t\"audit-policy-file\": []interface{}{string(\"/var/run/configmaps/audit/secure-oauth-storage-default.yaml\")},\n+ \t\t\t\"cors-allowed-origins\": []interface{}{string(`//127\\.0\\.0\\.1(:|$)`), string(\"//localhost(:|$)\")},\n+ \t\t\t\"etcd-servers\": []interface{}{\n+ \t\t\t\tstring(\"https://10.0.94.119:2379\"),\n+ \t\t\t\tstring(\"https://10.0.94.219:2379\"),\n+ \t\t\t\tstring(\"https://10.0.95.76:2379\"),\n+ \t\t\t},\n+ \t\t\t\"tls-cipher-suites\": []interface{}{\n+ \t\t\t\tstring(\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\"),\n+ \t\t\t},\n+ \t\t\t\"tls-min-version\": string(\"VersionTLS12\"),\n+ \t\t},\n+ \t},\n )\n"
10:38:54 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | ObserveStorageUpdated | Updated storage urls to https://10.0.94.119:2379,https://10.0.94.219:2379,https://10.0.95.76:2379
10:38:54 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:sa-creating-openshift-controller-manager -n openshift-infra because it was missing
10:38:54 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ServiceAccountCreated | Created ServiceAccount/openshift-controller-manager-sa -n openshift-controller-manager because it was missing
10:38:54 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | ServiceAccountCreated | Created ServiceAccount/localhost-recovery-client -n openshift-kube-scheduler because it was missing
10:38:54 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ServiceCreated | Created Service/controller-manager -n openshift-controller-manager because it was missing
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveAPIServerArgumentsAudit | audit policy has been set to profile=Default
10:38:54 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/prometheus-k8s -n openshift-controller-manager because it was missing
10:38:54 | openshift-cluster-node-tuning-operator | kubelet | tuned-9v69g | Started | Started container tuned
10:38:54 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/prometheus-k8s -n openshift-controller-manager because it was missing
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveStorageUpdated | Updated storage urls to https://10.0.94.119:2379,https://10.0.94.219:2379,https://10.0.95.76:2379
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveTLSSecurityProfile | cipherSuites changed to ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256" "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"]
10:38:54 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config -n openshift-controller-manager because it was missing
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveTemplates | templates changed to map["error":"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/errors.html" "login":"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/login.html" "providerSelection":"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/providers.html"]
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveTokenConfig | accessTokenMaxAgeSeconds changed from %!d(float64=0) to %!d(float64=86400)
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveTLSSecurityProfile | minTLSVersion changed to VersionTLS12
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveTLSSecurityProfile | cipherSuites changed to ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256" "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"]
10:38:54 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "" to "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found"
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObservedConfigChanged | Writing updated section ("oauthServer") of observed config: " map[string]interface{}(\n- \tnil,\n+ \t{\n+ \t\t\"corsAllowedOrigins\": []interface{}{string(`//127\\.0\\.0\\.1(:|$)`), string(\"//localhost(:|$)\")},\n+ \t\t\"oauthConfig\": map[string]interface{}{\n+ \t\t\t\"assetPublicURL\": string(\"\"),\n+ \t\t\t\"loginURL\": string(\"https://api.case03116926.lab.upshift.rdu2.redhat.com:6443\"),\n+ \t\t\t\"templates\": map[string]interface{}{\n+ \t\t\t\t\"error\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/errors.html\"),\n+ \t\t\t\t\"login\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/login.html\"),\n+ \t\t\t\t\"providerSelection\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/providers.html\"),\n+ \t\t\t},\n+ \t\t\t\"tokenConfig\": map[string]interface{}{\n+ \t\t\t\t\"accessTokenMaxAgeSeconds\": float64(86400),\n+ \t\t\t\t\"authorizeTokenMaxAgeSeconds\": float64(300),\n+ \t\t\t},\n+ \t\t},\n+ \t\t\"servingInfo\": map[string]interface{}{\n+ \t\t\t\"cipherSuites\": []interface{}{\n+ \t\t\t\tstring(\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\"),\n+ \t\t\t\tstring(\"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\"),\n+ \t\t\t},\n+ \t\t\t\"minTLSVersion\": string(\"VersionTLS12\"),\n+ \t\t},\n+ \t\t\"volumesToMount\": map[string]interface{}{\"identityProviders\": string(\"{}\")},\n+ \t},\n )\n"
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveTLSSecurityProfile | minTLSVersion changed to VersionTLS12
10:38:54 (x4) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 2 triggered by "configmap \"kube-scheduler-pod\" not found"
10:38:54 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | ObserveTLSSecurityProfile | cipherSuites changed to ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256" "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"]
10:38:54 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | RoutingConfigSubdomainChanged | Domain changed from "" to "apps.case03116926.lab.upshift.rdu2.redhat.com"
10:38:54 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found"
10:38:54 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveAPIServerURL | loginURL changed from to https://api.case03116926.lab.upshift.rdu2.redhat.com:6443
10:38:54 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | ObserveAPIServerArgumentsAudit | audit policy has been set to profile=Default
10:38:55 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapCreated | Created configmap/openshift-service-ca-n openshift-controller-manager because it was missing
10:38:55 | openshift-controller-manager | default-scheduler | controller-manager-9tld6 | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-9tld6 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:55 | openshift-controller-manager | default-scheduler | controller-manager-fg5sh | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-fg5sh to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:38:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 2 triggered by "configmap \"kube-scheduler-pod-1\" not found"
10:38:55 | openshift-apiserver-operator | openshift-apiserver-operator-apiserverstaticresources-staticresourcecontroller | openshift-apiserver-operator | NamespaceCreated | Created Namespace/openshift-apiserver because it was missing
10:38:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-kubecontrollermanagerstaticresources-staticresourcecontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/localhost-recovery-client-token -n openshift-kube-scheduler because it was missing
10:38:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found\nTargetConfigControllerDegraded: \"serviceaccount/localhost-recovery-client\": serviceaccounts \"localhost-recovery-client\" not found" to "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found\nTargetConfigControllerDegraded: \"serviceaccount/localhost-recovery-client\": serviceaccounts \"localhost-recovery-client\" not found"
10:38:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found" to "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found\nTargetConfigControllerDegraded: \"serviceaccount/localhost-recovery-client\": serviceaccounts \"localhost-recovery-client\" not found"
10:38:55 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | DaemonSetCreated | Created DaemonSet.apps/controller-manager -n openshift-controller-manager because it was missing
10:38:55 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | ConfigMapCreateFailed | Failed to create ConfigMap/revision-status-1 -n openshift-apiserver: namespaces "openshift-apiserver" not found
10:38:55 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapCreated | Created configmap/openshift-global-ca-n openshift-controller-manager because it was missing
10:38:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/csr-controller-ca -n openshift-kube-controller-manager-operator because it was missing
10:38:55 | openshift-service-ca-operator | service-ca-operator-status-controller-statussyncer_service-ca | service-ca-operator | OperatorStatusChanged | Status for clusteroperator/service-ca changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "servicecas" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-service-ca-operator"} {"" "namespaces" "" "openshift-service-ca"}]
10:38:55 | openshift-service-ca-operator | service-ca-operator | service-ca-operator-lock | LeaderElection | service-ca-operator-6fd6454f86-5bbh8_74fa080d-edf8-407e-8c94-c4f6f7c450f0 became leader
10:38:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: csr-signer,kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: csr-signer,kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]"
10:38:55 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-9tld6
10:38:55 | openshift-controller-manager | default-scheduler | controller-manager-mzf7c | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-mzf7c to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:38:55 (x2) | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | RevisionTriggered | new revision 1 triggered by "configmap \"audit\" not found"
10:38:55 | openshift-apiserver-operator | openshift-apiserver-operator-resource-sync-controller-resourcesynccontroller | openshift-apiserver-operator | ConfigMapCreateFailed | Failed to create ConfigMap/etcd-serving-ca -n openshift-apiserver: namespaces "openshift-apiserver" not found
10:38:55 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "" to "RevisionControllerDegraded: namespaces \"openshift-apiserver\" not found"
10:38:55 | openshift-multus | daemonset-controller | multus-admission-controller | SuccessfulCreate | Created pod: multus-admission-controller-nskb9
10:38:55 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods)."
10:38:55 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-fg5sh
10:38:55 | openshift-multus | default-scheduler | multus-admission-controller-nskb9 | Scheduled | Successfully assigned openshift-multus/multus-admission-controller-nskb9 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:38:55 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-mzf7c
10:38:55 | openshift-apiserver-operator | openshift-apiserver-operator-apiserverstaticresources-staticresourcecontroller | openshift-apiserver-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:openshift-apiserver because it was missing
10:38:55 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-status-controller-statussyncer_csi-snapshot-controller | csi-snapshot-controller-operator | OperatorStatusChanged | Status for clusteroperator/csi-snapshot-controller changed: Progressing changed from True to False (""),Available changed from False to True ("")
10:38:56 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: aggregator-client,bound-service-account-signing-key,check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,external-loadbalancer-serving-certkey,internal-loadbalancer-serving-certkey,localhost-serving-cert-certkey,node-kubeconfigs,service-network-serving-certkey, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0
10:38:56 | openshift-kube-apiserver-operator | kube-apiserver-operator-boundsatokensignercontroller | kube-apiserver-operator | SecretCreated | Created Secret/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator because it was missing
10:38:56 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:leader-locking-openshift-controller-manager -n openshift-controller-manager because it was missing
10:38:56 | openshift-apiserver-operator | openshift-apiserver-operator-resource-sync-controller-resourcesynccontroller | openshift-apiserver-operator | SecretCreated | Created Secret/etcd-client -n openshift-apiserver because it was missing
10:38:56 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server"
10:38:56 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found" to "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)"
10:38:56 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: namespaces \"openshift-oauth-apiserver\" not found"
10:38:56 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:leader-locking-openshift-controller-manager -n openshift-controller-manager because it was missing
10:38:56 | openshift-service-ca-operator | service-ca-operator-status-controller-statussyncer_service-ca | service-ca-operator | OperatorStatusChanged | Status for clusteroperator/service-ca changed: Degraded changed from Unknown to False ("")
10:38:56 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-1 -n openshift-apiserver because it was missing
10:38:56 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | NamespaceCreated | Created Namespace/openshift-service-ca because it was missing
10:38:56 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "RevisionControllerDegraded: namespaces \"openshift-apiserver\" not found" to "RevisionControllerDegraded: namespaces \"openshift-apiserver\" not found\nResourceSyncControllerDegraded: namespaces \"openshift-apiserver\" not found"
10:38:56 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-pod-2 -n openshift-kube-scheduler because it was missing
10:38:57 | openshift-apiserver-operator | openshift-apiserver-operator-apiserverstaticresources-staticresourcecontroller | openshift-apiserver-operator | ServiceCreated | Created Service/api -n openshift-apiserver because it was missing
10:38:57 (x6) | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | FailedMount | MountVolume.SetUp failed for volume "proxy-tls" : secret "proxy-tls" not found
10:38:57 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-resource-sync-controller-resourcesynccontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/csr-controller-ca -n openshift-config-managed because it was missing
10:38:57 | openshift-authentication-operator | cluster-authentication-operator-resource-sync-controller-resourcesynccontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca -n openshift-oauth-apiserver because it was missing
10:38:57 | openshift-kube-apiserver-operator | kube-apiserver-operator-backingresourcecontroller-staticresourcecontroller | kube-apiserver-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:openshift-kube-apiserver-installer because it was missing
10:38:57 | openshift-kube-apiserver-operator | kube-apiserver-operator-backingresourcecontroller-staticresourcecontroller | kube-apiserver-operator | ServiceAccountCreated | Created ServiceAccount/installer-sa -n openshift-kube-apiserver because it was missing
10:38:57 | openshift-authentication-operator | oauth-apiserver-apiserverstaticresources-staticresourcecontroller | authentication-operator | NamespaceCreated | Created Namespace/openshift-oauth-apiserver because it was missing
10:38:57 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: namespaces \"openshift-oauth-apiserver\" not found"
10:38:57 | openshift-authentication-operator | cluster-authentication-operator-openshiftauthenticationstaticresources-staticresourcecontroller | authentication-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:openshift-authentication because it was missing
10:38:57 | openshift-apiserver-operator | openshift-apiserver-operator-resource-sync-controller-resourcesynccontroller | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca -n openshift-apiserver because it was missing
10:38:57 | openshift-authentication-operator | cluster-authentication-operator-openshiftauthenticationstaticresources-staticresourcecontroller | authentication-operator | NamespaceCreated | Created Namespace/openshift-authentication because it was missing
10:38:57 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "RevisionControllerDegraded: namespaces \"openshift-apiserver\" not found\nResourceSyncControllerDegraded: namespaces \"openshift-apiserver\" not found" to "RevisionControllerDegraded: configmaps \"audit\" not found\nResourceSyncControllerDegraded: namespaces \"openshift-apiserver\" not found"
10:38:57 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/config-2 -n openshift-kube-scheduler because it was missing
10:38:57 (x2) | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | ConfigMapCreateFailed | Failed to create ConfigMap/revision-status-1 -n openshift-oauth-apiserver: namespaces "openshift-oauth-apiserver" not found
10:38:57 (x6) | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | FailedMount | MountVolume.SetUp failed for volume "proxy-tls" : secret "proxy-tls" not found
10:38:57 (x6) | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | FailedMount | MountVolume.SetUp failed for volume "proxy-tls" : secret "proxy-tls" not found
10:38:57 (x3) | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | RevisionTriggered | new revision 1 triggered by "configmap \"audit\" not found"
10:38:57 | openshift-authentication-operator | oauth-apiserver-apiserverstaticresources-staticresourcecontroller | authentication-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:oauth-apiserver because it was missing
10:38:57 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-2 -n openshift-apiserver because it was missing
10:38:58 | openshift-authentication-operator | cluster-authentication-operator-openshiftauthenticationstaticresources-staticresourcecontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/v4-0-config-system-trusted-ca-bundle -n openshift-authentication because it was missing
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: The master nodes not ready: node \"master-2.case03116926.lab.upshift.rdu2.redhat.com\" not ready since 2022-03-08 10:38:15 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready"
10:38:58 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "RevisionControllerDegraded: configmaps \"audit\" not found\nResourceSyncControllerDegraded: namespaces \"openshift-apiserver\" not found" to "RevisionControllerDegraded: configmaps \"audit\" not found"
10:38:58 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-1 -n openshift-apiserver: cause by changes in data.status
10:38:58 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-controller-manager: cause by changes in data.config.yaml,data.openshift-controller-manager.openshift-global-ca.configmap
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-control-plane-signer-ca -n openshift-kube-apiserver-operator because it was missing
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "kube-controller-manager-client-cert-key" in "openshift-config-managed" requires a new target cert/key pair: missing notAfter
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-aggregator-client-ca -n openshift-config-managed because it was missing
10:38:58 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-2 -n openshift-kube-scheduler because it was missing
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "aggregator-client" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-to-kubelet-client-ca -n openshift-kube-apiserver-operator because it was missing
10:38:58 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-9tld6
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "kubelet-client" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/service-network-serving-ca -n openshift-kube-apiserver-operator because it was missing
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "service-network-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/loadbalancer-serving-ca -n openshift-kube-apiserver-operator because it was missing
10:38:58 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "external-loadbalancer-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:38:58 (x12) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: csr-signer,kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1
10:38:58 | openshift-apiserver-operator | openshift-apiserver-operator-apiserverstaticresources-staticresourcecontroller | openshift-apiserver-operator | ServiceAccountCreated | Created ServiceAccount/openshift-apiserver-sa -n openshift-apiserver because it was missing
10:38:58 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-fg5sh
10:38:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | SecretCreated | Created Secret/csr-signer -n openshift-kube-controller-manager because it was missing
10:38:58 | openshift-authentication-operator | cluster-authentication-operator-resource-sync-controller-resourcesynccontroller | authentication-operator | SecretCreated | Created Secret/etcd-client -n openshift-oauth-apiserver because it was missing
10:38:58 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-mzf7c
10:38:58 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "RevisionControllerDegraded: configmaps \"audit\" not found\nResourceSyncControllerDegraded: namespaces \"openshift-apiserver\" not found" to "RevisionControllerDegraded: configmaps \"audit\" not found",Available message changed from "APIServicesAvailable: endpoints \"api\" not found" to "APIServicesAvailable: PreconditionNotReady"
10:38:58 | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/revision-status-1 -n openshift-oauth-apiserver because it was missing
10:38:58 (x2) | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing changed from Unknown to True ("Progressing: daemonset/controller-manager: observed generation is 1, desired generation is 2.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\nProgressing: openshiftcontrollermanagers.operator.openshift.io/cluster: observed generation is 0, desired generation is 2."),Available changed from Unknown to False ("Available: no daemon pods available on any node.")
10:38:59 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/localhost-serving-ca -n openshift-kube-apiserver-operator because it was missing
10:38:59 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "localhost-serving-cert-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:38:59 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreateFailed | Failed to create ConfigMap/loadbalancer-serving-ca -n openshift-kube-apiserver-operator: configmaps "loadbalancer-serving-ca" already exists
10:38:59 | openshift-apiserver-operator | openshift-apiserver-operator-apiserverstaticresources-staticresourcecontroller | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/trusted-ca-bundle -n openshift-apiserver because it was missing
10:38:59 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Available message changed from "APIServicesAvailable: endpoints \"api\" not found" to "APIServicesAvailable: PreconditionNotReady"
10:38:59 | openshift-apiserver-operator | openshift-apiserver-operator-apiserverstaticresources-staticresourcecontroller | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/audit -n openshift-apiserver because it was missing
10:38:59 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | RevisionTriggered | new revision 2 triggered by "configmap \"audit-1\" not found"
10:38:59 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:controller:service-ca because it was missing
10:38:59 (x3) | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | RevisionTriggered | new revision 2 triggered by "configmap \"audit\" not found"
10:38:59 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "Progressing: daemonset/controller-manager: observed generation is 1, desired generation is 2.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\nProgressing: openshiftcontrollermanagers.operator.openshift.io/cluster: observed generation is 0, desired generation is 2." to "",Available message changed from "Available: no daemon pods available on any node." to ""
10:38:59 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1
10:38:59 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:controller:service-ca because it was missing
10:38:59 | openshift-authentication-operator | cluster-authentication-operator-openshiftauthenticationstaticresources-staticresourcecontroller | authentication-operator | SecretCreated | Created Secret/v4-0-config-system-ocp-branding-template -n openshift-authentication because it was missing
10:38:59 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | RoleCreated | Created Role.rbac.authorization.k8s.io/system:openshift:controller:service-ca -n openshift-service-ca because it was missing
10:38:59 (x3) | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreateFailed | Failed to create ConfigMap/kube-control-plane-signer-ca -n openshift-kube-apiserver-operator: configmaps "kube-control-plane-signer-ca" already exists
10:38:59 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: namespaces \"openshift-oauth-apiserver\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found"
10:38:59 | openshift-authentication-operator | oauth-apiserver-apiserverstaticresources-staticresourcecontroller | authentication-operator | ServiceCreated | Created Service/api -n openshift-oauth-apiserver because it was missing
10:38:59 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: csr-signer,kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]"
10:38:59 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:controller:service-ca -n openshift-service-ca because it was missing
10:39:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "localhost-recovery-serving-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:39:00 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:00 | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/revision-status-2 -n openshift-oauth-apiserver because it was missing
10:39:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/node-system-admin-signer -n openshift-kube-apiserver-operator because it was missing
10:39:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | CABundleUpdateRequired | "node-system-admin-ca" in "openshift-kube-apiserver-operator" requires a new cert
10:39:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-boundsatokensignercontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/bound-sa-token-signing-certs -n openshift-kube-apiserver because it was missing
10:39:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing changed from Unknown to False ("")
10:39:00 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/scheduler-kubeconfig-2 -n openshift-kube-scheduler because it was missing
10:39:00 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/audit-2 -n openshift-apiserver because it was missing
10:39:00 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | ServiceAccountCreated | Created ServiceAccount/service-ca -n openshift-service-ca because it was missing
10:39:00 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-env-var-controller | etcd-operator | EnvVarControllerUpdatingStatus | Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
10:39:00 | openshift-apiserver-operator | openshift-apiserver-operator-revisioncontroller | openshift-apiserver-operator | RevisionCreate | Revision 1 created because configmap "audit-1" not found
10:39:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-serving-signer -n openshift-kube-apiserver-operator because it was missing
10:39:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found\nTargetConfigControllerDegraded: \"serviceaccount/localhost-recovery-client\": serviceaccounts \"localhost-recovery-client\" not found" to "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found"
10:39:01 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kubelet-serving-ca -n openshift-kube-apiserver because it was missing
10:39:01 (x2) | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "RevisionControllerDegraded: configmaps \"audit\" not found" to ""
10:39:01 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | SecretCreated | Created Secret/signing-key -n openshift-service-ca because it was missing
10:39:01 (x9) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | RequiredInstallerResourcesMissing | configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1
10:39:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-cert-syncer-kubeconfig-2 -n openshift-kube-scheduler because it was missing
10:39:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "InstallerControllerDegraded: missing required resources: secrets: kube-scheduler-client-cert-key\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready",Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
10:39:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-scheduler-pod\" not found" to "InstallerControllerDegraded: missing required resources: [configmaps: config-1,kube-scheduler-cert-syncer-kubeconfig-1,kube-scheduler-pod-1,scheduler-kubeconfig-1,serviceaccount-ca-1, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-1]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:39:01 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | RequiredInstallerResourcesMissing | secrets: kube-scheduler-client-cert-key
10:39:01 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca -n openshift-kube-controller-manager because it was missing
10:39:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionCreate | Revision 1 created because configmap "kube-scheduler-pod-1" not found
10:39:01 | openshift-authentication-operator | oauth-apiserver-apiserverstaticresources-staticresourcecontroller | authentication-operator | ServiceAccountCreated | Created ServiceAccount/oauth-apiserver-sa -n openshift-oauth-apiserver because it was missing
10:39:01 | openshift-authentication-operator | cluster-authentication-operator-openshiftauthenticationstaticresources-staticresourcecontroller | authentication-operator | ServiceAccountCreated | Created ServiceAccount/oauth-openshift -n openshift-authentication because it was missing
10:39:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/localhost-recovery-client-token-2 -n openshift-kube-scheduler because it was missing
10:39:01 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ServiceCreated | Created Service/apiserver -n openshift-kube-apiserver because it was missing
10:39:01 | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-1 -n openshift-oauth-apiserver: cause by changes in data.status
10:39:02 (x2) | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | RevisionTriggered | new revision 2 triggered by "configmap \"audit\" not found"
10:39:02 | openshift-service-ca-operator | service-ca-operator-resource-sync-controller-resourcesynccontroller | service-ca-operator | ConfigMapCreated | Created ConfigMap/service-ca -n openshift-config-managed because it was missing
10:39:02 | openshift-controller-manager | default-scheduler | controller-manager-9ktwp | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-9ktwp to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:02 (x26) | openshift-authentication-operator | cluster-authentication-operator | authentication-operator | OAuthServiceCheck | failed to get oauth service: service "oauth-openshift" not found
10:39:02 (x23) | openshift-authentication-operator | cluster-authentication-operator | authentication-operator | OAuthServiceEndpointsCheck | failed to get oauth service endpoints: endpoints "oauth-openshift" not found
10:39:02 | openshift-service-ca-operator | service-ca-operator-status-controller-statussyncer_service-ca | service-ca-operator | OperatorStatusChanged | Status for clusteroperator/service-ca changed: Progressing changed from Unknown to True ("Progressing: \nProgressing: service-ca does not have available replicas"),Available changed from Unknown to True ("")
10:39:02 | openshift-authentication-operator | oauth-apiserver-apiserverstaticresources-staticresourcecontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/audit -n openshift-oauth-apiserver because it was missing
10:39:02 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | ConfigMapCreated | Created ConfigMap/signing-cabundle -n openshift-service-ca because it was missing
10:39:02 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-9ktwp
10:39:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]"
10:39:02 | openshift-service-ca-operator | service-ca-operator | service-ca-operator | DeploymentCreated | Created Deployment.apps/service-ca -n openshift-service-ca because it was missing
10:39:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-resource-sync-controller-resourcesynccontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/aggregator-client-ca -n openshift-kube-controller-manager because it was missing
10:39:03 (x5) | openshift-controller-manager | kubelet | controller-manager-fg5sh | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "serving-cert" not found
10:39:03 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-9ktwp
10:39:03 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-resource-sync-controller-resourcesynccontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca -n openshift-kube-controller-manager because it was missing
10:39:03 | openshift-authentication-operator | cluster-authentication-operator-openshiftauthenticationstaticresources-staticresourcecontroller | authentication-operator | ServiceCreated | Created Service/oauth-openshift -n openshift-authentication because it was missing
10:39:03 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Failed to get oauth-openshift enpoints\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:03 | openshift-kube-apiserver-operator | kube-apiserver-operator-boundsatokensignercontroller | kube-apiserver-operator | SecretCreated | Created Secret/bound-service-account-signing-key -n openshift-kube-apiserver because it was missing
10:39:03 (x5) | openshift-controller-manager | kubelet | controller-manager-9tld6 | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:03 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nEnvVarControllerDegraded: at least three nodes are required to have a valid configuration"
10:39:03 (x6) | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | FailedMount | MountVolume.SetUp failed for volume "machine-api-operator-tls" : secret "machine-api-operator-tls" not found
10:39:03 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "" to "Progressing: daemonset/controller-manager: observed generation is 3, desired generation is 4.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\nProgressing: daemonset/controller-manager: updated number scheduled is 1, desired number scheduled is 3",Available message changed from "" to "Available: no daemon pods available on any node."
10:39:03 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-config-observer-configobserver | kube-controller-manager-operator | ObserveServiceCAConfigMap | observed change in config
10:39:03 (x3) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1
10:39:03 (x5) | openshift-controller-manager | kubelet | controller-manager-mzf7c | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "serving-cert" not found
10:39:03 (x5) | openshift-controller-manager | kubelet | controller-manager-mzf7c | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:03 (x5) | openshift-controller-manager | kubelet | controller-manager-9tld6 | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "serving-cert" not found
10:39:03 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-config-observer-configobserver | kube-controller-manager-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ "extendedArguments": map[string]interface{}{"cluster-cidr": []interface{}{string("10.128.0.0/14")}, "cluster-name": []interface{}{string("case03116926-lsn5h")}, "feature-gates": []interface{}{string("APIPriorityAndFairness=true"), string("RotateKubeletServerCertificate=true"), string("SupportPodPidsLimit=true"), string("NodeDisruptionExclusion=true"), string("ServiceNodeExclusion=true"), string("SCTPSupport=true"), string("LegacyNodeRoleBehavior=false")}, "service-cluster-ip-range": []interface{}{string("172.30.0.0/16")}}, + "serviceServingCert": map[string]interface{}{ + "certFile": string("/etc/kubernetes/static-pod-resources/configmaps/service-ca/ca-bundle.crt"), + }, }
10:39:03 | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | RevisionTriggered | new revision 2 triggered by "configmap \"audit-1\" not found"
10:39:03 (x5) | openshift-controller-manager | kubelet | controller-manager-fg5sh | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-cert-syncer-kubeconfig -n openshift-kube-apiserver because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:controller:kube-apiserver-check-endpoints-auth-delegator because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:controller:check-endpoints-crd-reader because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:controller:kube-apiserver-check-endpoints-node-reader because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/kube-controller-manager-client-cert-key -n openshift-config-managed because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:controller:check-endpoints-node-reader because it was missing
10:39:04 | openshift-controller-manager | default-scheduler | controller-manager-ktjz4 | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-ktjz4 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:controller:kube-apiserver-check-endpoints-crd-reader because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/aggregator-client -n openshift-kube-apiserver because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/kubelet-client -n openshift-kube-apiserver because it was missing
10:39:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleCreated | Created ClusterRole.rbac.authorization.k8s.io/system:openshift:controller:check-endpoints because it was missing
10:39:04 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-ktjz4
10:39:04 | openshift-controller-manager | default-scheduler | controller-manager-t54r4 | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-t54r4 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:04 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:04 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nRevisionControllerDegraded: configmaps \"audit\" not found\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:04 | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | RevisionCreate | Revision 1 created because configmap "audit-1" not found
10:39:04 | openshift-authentication-operator | oauth-apiserver-revisioncontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/audit-2 -n openshift-oauth-apiserver because it was missing
10:39:04 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-t54r4
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/node-system-admin-ca -n openshift-kube-apiserver-operator because it was missing
10:39:05 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | RequiredInstallerResourcesMissing | configmaps: etcd-scripts,restore-etcd-pod, configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/external-loadbalancer-serving-certkey -n openshift-kube-apiserver because it was missing
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-serving-cert-certkey -n openshift-kube-apiserver because it was missing
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/service-network-serving-certkey -n openshift-kube-apiserver because it was missing
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0
10:39:05 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nOAuthServiceDegraded: Unable to get oauth server service: service \"oauth-openshift\" not found\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nSystemServiceCAConfigDegraded: Config \"\" has no service CA data\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "node-system-admin-client" in "openshift-kube-apiserver-operator" requires a new target cert/key pair: missing notAfter
10:39:05 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nEnvVarControllerDegraded: at least three nodes are required to have a valid configuration" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nEnvVarControllerDegraded: at least three nodes are required to have a valid configuration\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded"
10:39:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready" to "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: aggregator-client,bound-service-account-signing-key,check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,external-loadbalancer-serving-certkey,internal-loadbalancer-serving-certkey,localhost-serving-cert-certkey,node-kubeconfigs,service-network-serving-certkey, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]",Progressing message changed from "" to "NodeInstallerProgressing: 3 nodes are at revision 0",Available changed from Unknown to False ("StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0")
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kubelet-serving-ca -n openshift-config-managed because it was missing
10:39:06 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: service \"oauth-openshift\" not found\nSystemServiceCAConfigDegraded: Config \"\" has no service CA data\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host\nSystemServiceCAConfigDegraded: Config \"\" has no service CA data\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:06 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods)." to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host"
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionCreateFailed | Failed to create revision 1: configmaps "kube-apiserver-pod" not found
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/localhost-recovery-serving-ca -n openshift-kube-apiserver-operator because it was missing
10:39:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig -n openshift-kube-controller-manager because it was missing
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "localhost-recovery-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:39:06 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host\nSystemServiceCAConfigDegraded: Config \"\" has no service CA data\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host\nSystemServiceCAConfigDegraded: Config \"v4-0-config-system-service-ca\" has no service CA data\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/check-endpoints-kubeconfig -n openshift-kube-apiserver because it was missing
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:controller:kube-apiserver-check-endpoints -n kube-system because it was missing
10:39:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | RoleBindingCreated | Created RoleBinding.rbac.authorization.k8s.io/system:openshift:controller:check-endpoints -n openshift-kube-apiserver because it was missing
10:39:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-resource-sync-controller-resourcesynccontroller | kube-controller-manager-operator | SecretCreated | Created Secret/kube-controller-manager-client-cert-key -n openshift-kube-controller-manager because it was missing
10:39:07 (x4) | openshift-etcd-operator | openshift-cluster-etcd-operator-nodecontroller | etcd-operator | MasterNodeObserved | Observed new master node master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:07 (x4) | openshift-etcd-operator | openshift-cluster-etcd-operator-nodecontroller | etcd-operator | MasterNodeObserved | Observed new master node master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:07 (x4) | openshift-etcd-operator | openshift-cluster-etcd-operator-nodecontroller | etcd-operator | MasterNodeObserved | Observed new master node master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:08 (x7) | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | FailedMount | MountVolume.SetUp failed for volume "metrics-tls" : secret "metrics-tls" not found
10:39:08 (x7) | openshift-cluster-version | kubelet | cluster-version-operator-5d86d65f6b-4fm8l | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "cluster-version-operator-serving-cert" not found
10:39:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: aggregator-client,bound-service-account-signing-key,check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,external-loadbalancer-serving-certkey,internal-loadbalancer-serving-certkey,localhost-serving-cert-certkey,node-kubeconfigs,service-network-serving-certkey, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]" to "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]"
10:39:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: aggregator-client-ca,client-ca, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0
10:39:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/node-system-admin-client -n openshift-kube-apiserver-operator because it was missing
10:39:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ClusterRoleBindingCreated | Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:kube-apiserver-recovery because it was missing
10:39:08 (x7) | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | FailedMount | MountVolume.SetUp failed for volume "cloud-credential-operator-serving-cert" : secret "cloud-credential-operator-serving-cert" not found
10:39:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/control-plane-node-kubeconfig -n openshift-kube-apiserver because it was missing
10:39:08 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nEnvVarControllerDegraded: at least three nodes are required to have a valid configuration\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nEnvVarControllerDegraded: at least three nodes are required to have a valid configuration\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:08 | openshift-etcd-operator | openshift-cluster-etcd-operator-target-config-controller-targetconfigcontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/config -n openshift-etcd because it was missing
10:39:08 | openshift-etcd-operator | openshift-cluster-etcd-operator-script-controller-scriptcontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-scripts -n openshift-etcd because it was missing
10:39:08 (x7) | openshift-image-registry | kubelet | cluster-image-registry-operator-98c4d4975-kxn6n | FailedMount | MountVolume.SetUp failed for volume "image-registry-operator-tls" : secret "image-registry-operator-tls" not found
10:39:08 (x4) | openshift-controller-manager | kubelet | controller-manager-ktjz4 | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "serving-cert" not found
10:39:08 (x7) | openshift-operator-lifecycle-manager | kubelet | catalog-operator-887559f54-g4cgp | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "catalog-operator-serving-cert" not found
10:39:08 (x6) | openshift-monitoring | kubelet | node-exporter-kcxrx | FailedMount | MountVolume.SetUp failed for volume "node-exporter-tls" : secret "node-exporter-tls" not found
10:39:08 (x6) | openshift-monitoring | kubelet | node-exporter-l5q6v | FailedMount | MountVolume.SetUp failed for volume "node-exporter-tls" : secret "node-exporter-tls" not found
10:39:08 (x6) | openshift-monitoring | kubelet | node-exporter-vrvq5 | FailedMount | MountVolume.SetUp failed for volume "node-exporter-tls" : secret "node-exporter-tls" not found
10:39:08 (x4) | openshift-controller-manager | kubelet | controller-manager-t54r4 | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "serving-cert" not found
10:39:08 (x7) | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | FailedMount | MountVolume.SetUp failed for volume "metrics-tls" : secret "metrics-tls" not found
10:39:08 (x7) | openshift-multus | kubelet | multus-admission-controller-gsjzb | FailedMount | MountVolume.SetUp failed for volume "webhook-certs" : secret "multus-admission-controller-secret" not found
10:39:08 (x7) | openshift-multus | kubelet | multus-admission-controller-bkzjk | FailedMount | MountVolume.SetUp failed for volume "webhook-certs" : secret "multus-admission-controller-secret" not found
10:39:09 (x7) | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | FailedMount | MountVolume.SetUp failed for volume "machine-approver-tls" : secret "machine-approver-tls" not found
10:39:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "internal-loadbalancer-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:39:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-serving-certkey -n openshift-kube-apiserver because it was missing
10:39:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "check-endpoints-client-cert-key" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:39:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/bound-sa-token-signing-certs -n openshift-config-managed because it was missing
10:39:09 (x19) | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | RevisionTriggered | new revision 2 triggered by "configmap \"etcd-pod\" not found"
10:39:09 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nEnvVarControllerDegraded: at least three nodes are required to have a valid configuration\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:09 | openshift-etcd-operator | openshift-cluster-etcd-operator-target-config-controller-targetconfigcontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-pod -n openshift-etcd because it was missing
10:39:09 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | RevisionTriggered | new revision 2 triggered by "configmap \"etcd-pod-1\" not found"
10:39:09 (x7) | openshift-operator-lifecycle-manager | kubelet | olm-operator-5fbfc8c78-h4nwt | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "olm-operator-serving-cert" not found
10:39:09 | openshift-service-ca-operator | service-ca-operator-status-controller-statussyncer_service-ca | service-ca-operator | OperatorVersionChanged | clusteroperator/service-ca version "operator" changed from "" to "4.6.52"
10:39:09 | openshift-service-ca-operator | service-ca-operator-status-controller-statussyncer_service-ca | service-ca-operator | OperatorStatusChanged | Status for clusteroperator/service-ca changed: status.versions changed from [] to [{"operator" "4.6.52"}]
10:39:09 (x7) | openshift-marketplace | kubelet | marketplace-operator-664c8865b-d9dx9 | FailedMount | MountVolume.SetUp failed for volume "marketplace-operator-metrics" : secret "marketplace-operator-metrics" not found
10:39:09 | openshift-service-ca-operator | service-ca-operator-status-controller-statussyncer_service-ca | service-ca-operator | OperatorStatusChanged | Status for clusteroperator/service-ca changed: Progressing changed from True to False ("Progressing: All service-ca-operator deployments updated")
10:39:10 (x5) | openshift-controller-manager | kubelet | controller-manager-9ktwp | FailedMount | MountVolume.SetUp failed for volume "serving-cert" : secret "serving-cert" not found
10:39:10 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host\nSystemServiceCAConfigDegraded: Config \"v4-0-config-system-service-ca\" has no service CA data\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ServiceAccountCreated | Created ServiceAccount/localhost-recovery-client -n openshift-kube-apiserver because it was missing
10:39:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca -n openshift-kube-apiserver because it was missing
10:39:10 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod -n openshift-kube-controller-manager because it was missing
10:39:10 (x17) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 2 triggered by "configmap \"kube-controller-manager-pod\" not found"
10:39:10 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-pod-2 -n openshift-etcd because it was missing
10:39:10 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | RequiredInstallerResourcesMissing | configmaps: restore-etcd-pod, configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1
10:39:10 (x5) | openshift-controller-manager | kubelet | controller-manager-9ktwp | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "kube-scheduler-client-cert-key" in "openshift-config-managed" requires a new target cert/key pair: missing notAfter
10:39:11 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca-2 -n openshift-etcd because it was missing
10:39:11 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/sa-token-signing-certs -n openshift-kube-apiserver because it was missing
10:39:11 (x8) | openshift-multus | kubelet | network-metrics-daemon-djqpl | FailedMount | MountVolume.SetUp failed for volume "metrics-certs" : secret "metrics-daemon-secret" not found
10:39:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | TargetUpdateRequired | "control-plane-node-admin-client-cert-key" in "openshift-kube-apiserver" requires a new target cert/key pair: missing notAfter
10:39:11 | openshift-etcd-operator | openshift-cluster-etcd-operator-target-config-controller-targetconfigcontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/restore-etcd-pod -n openshift-etcd because it was missing
10:39:11 (x6) | openshift-multus | kubelet | multus-admission-controller-nskb9 | FailedMount | MountVolume.SetUp failed for volume "webhook-certs" : secret "multus-admission-controller-secret" not found
10:39:11 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/config-2 -n openshift-etcd because it was missing
10:39:11 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 2 triggered by "configmap \"kube-controller-manager-pod-1\" not found"
10:39:11 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "Progressing: daemonset/controller-manager: observed generation is 3, desired generation is 4.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\nProgressing: daemonset/controller-manager: updated number scheduled is 1, desired number scheduled is 3" to "",Available message changed from "Available: no daemon pods available on any node." to ""
10:39:11 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:11 (x8) | openshift-multus | kubelet | network-metrics-daemon-2nbpw | FailedMount | MountVolume.SetUp failed for volume "metrics-certs" : secret "metrics-daemon-secret" not found
10:39:11 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: no route to host" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
10:39:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/check-endpoints-client-cert-key -n openshift-kube-apiserver because it was missing
10:39:12 (x4) | openshift-etcd-operator | openshift-cluster-etcd-operator-host-etcd-2-endpoints-controller2-hostendpoints2controller | etcd-operator | HostEndpoints2ErrorUpdatingStatus | Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
10:39:12 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]"
10:39:12 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-controller-manager: cause by changes in data.openshift-controller-manager.serving-cert.secret
10:39:12 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-t54r4
10:39:12 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-ktjz4
10:39:12 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/internal-loadbalancer-serving-certkey -n openshift-kube-apiserver because it was missing
10:39:12 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-peer-client-ca-2 -n openshift-etcd because it was missing
10:39:12 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-client-token -n openshift-kube-apiserver because it was missing
10:39:12 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-7lhx9
10:39:12 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-2 -n openshift-kube-controller-manager because it was missing
10:39:12 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused"
10:39:12 (x3) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: localhost-recovery-client-token-1,service-account-private-key-1
10:39:12 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: kube-controller-manager-client-cert-key, secrets: localhost-recovery-client-token-1,service-account-private-key-1]" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: localhost-recovery-client-token-1,service-account-private-key-1]"
10:39:12 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:12 | openshift-controller-manager | default-scheduler | controller-manager-7lhx9 | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-7lhx9 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:12 (x4) | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-endpoints-controller-etcdendpointscontroller | etcd-operator | EtcdEndpointsErrorUpdatingStatus | Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
10:39:13 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | SecretCreated | Created Secret/etcd-client -n openshift-kube-apiserver because it was missing
10:39:13 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:39:13 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-resource-sync-controller-resourcesynccontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/kube-scheduler-client-cert-key -n openshift-kube-scheduler because it was missing
10:39:13 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Created | Created container machine-config-daemon
10:39:13 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:39:13 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Started | Started container machine-config-daemon
10:39:13 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/kube-scheduler-client-cert-key -n openshift-config-managed because it was missing
10:39:13 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd"
10:39:13 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-metrics-proxy-serving-ca-2 -n openshift-etcd because it was missing
10:39:13 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-2 -n openshift-kube-controller-manager because it was missing
10:39:13 | openshift-kube-apiserver-operator | kube-apiserver-operator-kubeapiserverstaticresources-staticresourcecontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-audit-policies -n openshift-kube-apiserver because it was missing
10:39:13 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:39:14 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Created | Created container machine-config-daemon
10:39:14 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-2 -n openshift-kube-controller-manager because it was missing
10:39:14 | openshift-kube-apiserver-operator | kube-apiserver-operator-cert-rotation-controller | kube-apiserver-operator | SecretCreated | Created Secret/control-plane-node-admin-client-cert-key -n openshift-kube-apiserver because it was missing
10:39:14 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-metrics-proxy-client-ca-2 -n openshift-etcd because it was missing
10:39:14 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready",Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1"),Available message changed from "StaticPodsAvailable: 0 nodes are active; " to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1"
10:39:14 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd"
10:39:14 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Started | Started container machine-config-daemon
10:39:14 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Created | Created container machine-config-daemon
10:39:14 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd"
10:39:14 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Started | Started container machine-config-daemon
10:39:15 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" in 1.727193712s
10:39:15 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:15 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" in 1.883622339s
10:39:15 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-peer-2 -n openshift-etcd because it was missing
10:39:15 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Started | Started container oauth-proxy
10:39:15 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-kube-controller-manager: cause by changes in data.config.yaml
10:39:15 | openshift-machine-config-operator | kubelet | machine-config-daemon-plfj2 | Created | Created container oauth-proxy
10:39:16 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:16 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-2 -n openshift-kube-controller-manager because it was missing
10:39:16 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Created | Created container oauth-proxy
10:39:16 | openshift-machine-config-operator | kubelet | machine-config-daemon-cfgj5 | Started | Started container oauth-proxy
10:39:16 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Started | Started container oauth-proxy
10:39:16 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Created | Created container oauth-proxy
10:39:16 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-serving-2 -n openshift-etcd because it was missing
10:39:16 | openshift-machine-config-operator | kubelet | machine-config-daemon-cbhkk | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" in 2.080073757s
10:39:16 (x5) | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | RequiredInstallerResourcesMissing | configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1
10:39:17 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:17 | openshift-machine-config-operator | default-scheduler | machine-config-controller-7df85d4c4-lsgx2 | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-controller-7df85d4c4-lsgx2 to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:17 | openshift-machine-config-operator | replicaset-controller | machine-config-controller-7df85d4c4 | SuccessfulCreate | Created pod: machine-config-controller-7df85d4c4-lsgx2
10:39:17 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-serving-metrics-2 -n openshift-etcd because it was missing
10:39:17 | openshift-machine-config-operator | deployment-controller | machine-config-controller | ScalingReplicaSet | Scaled up replica set machine-config-controller-7df85d4c4 to 1
10:39:17 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | RevisionCreate | Revision 1 created because configmap "etcd-pod-1" not found
10:39:18 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/aggregator-client-ca -n openshift-kube-apiserver because it was missing
10:39:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/cluster-policy-controller-config -n openshift-kube-controller-manager: cause by changes in data.config.yaml
10:39:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-2 -n openshift-kube-controller-manager because it was missing
10:39:18 (x4) | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | BackOff | Back-off restarting failed container
10:39:19 (x7) | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | FailedMount | MountVolume.SetUp failed for volume "cert" : secret "cluster-autoscaler-operator-cert" not found
10:39:19 | openshift-machine-config-operator | multus | machine-config-controller-7df85d4c4-lsgx2 | AddedInterface | Add eth0 [10.129.0.3/23]
10:39:19 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nInstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:19 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
10:39:19 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-2.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:39:20 (x6) | openshift-controller-manager | kubelet | controller-manager-t54r4 | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:20 (x6) | openshift-controller-manager | kubelet | controller-manager-ktjz4 | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:20 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-2 -n openshift-kube-controller-manager because it was missing
10:39:21 | openshift-machine-api | multus | machine-api-operator-784c77b6d4-xfllc | AddedInterface | Add eth0 [10.130.0.7/23]
10:39:21 | openshift-machine-config-operator | daemonset-controller | machine-config-server | SuccessfulCreate | Created pod: machine-config-server-j9fh9
10:39:21 | openshift-machine-config-operator | daemonset-controller | machine-config-server | SuccessfulCreate | Created pod: machine-config-server-szk5p
10:39:21 | openshift-machine-config-operator | default-scheduler | machine-config-server-szk5p | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-server-szk5p to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:21 | openshift-machine-config-operator | daemonset-controller | machine-config-server | SuccessfulCreate | Created pod: machine-config-server-mmdzw
10:39:21 | openshift-machine-config-operator | default-scheduler | machine-config-server-mmdzw | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-server-mmdzw to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:21 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 3 triggered by "secret/serving-cert has changed"
10:39:21 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | PodCreated | Created Pod/installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:39:21 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-2 -n openshift-kube-controller-manager because it was missing
10:39:21 | openshift-machine-config-operator | default-scheduler | machine-config-server-j9fh9 | Scheduled | Successfully assigned openshift-machine-config-operator/machine-config-server-j9fh9 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:22 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-pod-3 -n openshift-kube-scheduler because it was missing
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-mmdzw | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-mmdzw | Started | Started container machine-config-server
10:39:22 | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Created | Created container kube-rbac-proxy
10:39:22 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-2 -n openshift-kube-controller-manager because it was missing
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-szk5p | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-szk5p | Created | Created container machine-config-server
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-szk5p | Started | Started container machine-config-server
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-j9fh9 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-j9fh9 | Created | Created container machine-config-server
10:39:22 | openshift-multus | multus | network-metrics-daemon-sw48w | AddedInterface | Add eth0 [10.129.0.4/23]
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-mmdzw | Created | Created container machine-config-server
10:39:22 | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed187e9c9e7585720b3b407ceeeb827d2d4d38fad8843cd914677eb3ab53c13b"
10:39:22 | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Started | Started container kube-rbac-proxy
10:39:22 | openshift-machine-config-operator | kubelet | machine-config-server-j9fh9 | Started | Started container machine-config-server
10:39:22 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/revision-status-3 -n openshift-kube-scheduler because it was missing
10:39:22 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-2 -n openshift-kube-scheduler: cause by changes in data.status
10:39:22 | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:23 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/config-3 -n openshift-kube-scheduler because it was missing
10:39:23 | openshift-multus | kubelet | network-metrics-daemon-sw48w | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1"
10:39:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-2 -n openshift-kube-controller-manager because it was missing
10:39:23 | openshift-etcd | multus | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.5/23]
10:39:23 (x7) | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | FailedMount | MountVolume.SetUp failed for volume "prometheus-operator-tls" : secret "prometheus-operator-tls" not found
10:39:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/csr-controller-ca -n openshift-kube-controller-manager-operator: cause by changes in data.ca-bundle.crt
10:39:24 | openshift-monitoring | kubelet | node-exporter-kcxrx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708"
10:39:24 | openshift-monitoring | kubelet | node-exporter-l5q6v | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708"
10:39:24 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-2 -n openshift-kube-controller-manager because it was missing
10:39:24 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-3 -n openshift-kube-scheduler because it was missing
10:39:24 | openshift-controller-manager | default-scheduler | controller-manager-pxstf | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-pxstf to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:24 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-pxstf
10:39:24 | openshift-controller-manager | default-scheduler | controller-manager-q5w7b | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-q5w7b to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:24 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-q5w7b
10:39:25 | openshift-multus | kubelet | network-metrics-daemon-sw48w | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1" in 2.005244668s
10:39:25 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708"
10:39:25 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/scheduler-kubeconfig-3 -n openshift-kube-scheduler because it was missing
10:39:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-resource-sync-controller-resourcesynccontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/csr-controller-ca -n openshift-config-managed: cause by changes in data.ca-bundle.crt
10:39:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 1 created because configmap "kube-controller-manager-pod-1" not found
10:39:25 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,check-endpoints-kubeconfig,client-ca,control-plane-node-kubeconfig, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]"
10:39:25 | openshift-etcd | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b"
10:39:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 3 triggered by "configmap/config has changed,configmap/cluster-policy-controller-config has changed"
10:39:25 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-cert-syncer-kubeconfig-3 -n openshift-kube-scheduler because it was missing
10:39:25 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/kube-scheduler-pod -n openshift-kube-scheduler: cause by changes in data.pod.yaml
10:39:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: localhost-recovery-client-token-1,service-account-private-key-1]" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: configmaps: client-ca",Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
10:39:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-2 -n openshift-kube-controller-manager because it was missing
10:39:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: configmaps \"kube-controller-manager-pod\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: localhost-recovery-client-token-1,service-account-private-key-1]" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: cluster-policy-controller-config-1,config-1,controller-manager-kubeconfig-1,kube-controller-cert-syncer-kubeconfig-1,kube-controller-manager-pod-1,recycler-config-1,service-ca-1,serviceaccount-ca-1, secrets: localhost-recovery-client-token-1,service-account-private-key-1]"
10:39:26 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/kubelet-serving-ca -n openshift-config-managed: cause by changes in data.ca-bundle.crt
10:39:26 | openshift-monitoring | kubelet | node-exporter-l5q6v | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:39:26 | openshift-monitoring | kubelet | node-exporter-l5q6v | Started | Started container init-textfile
10:39:26 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-3 -n openshift-kube-controller-manager because it was missing
10:39:26 | openshift-monitoring | kubelet | node-exporter-l5q6v | Created | Created container init-textfile
10:39:26 | openshift-monitoring | kubelet | node-exporter-l5q6v | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" in 1.164368177s
10:39:26 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" in 1.512881377s
10:39:26 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/kubelet-serving-ca -n openshift-kube-apiserver: cause by changes in data.ca-bundle.crt
10:39:26 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/serving-cert-3 -n openshift-kube-scheduler because it was missing
10:39:27 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Created | Created container init-textfile
10:39:27 | openshift-monitoring | kubelet | node-exporter-l5q6v | Started | Started container kube-rbac-proxy
10:39:27 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 3 triggered by "secret/serving-cert has changed,configmap/kube-scheduler-pod has changed"
10:39:27 | openshift-etcd | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" in 2.175636078s
10:39:27 | openshift-monitoring | kubelet | node-exporter-l5q6v | Started | Started container node-exporter
10:39:27 | openshift-monitoring | kubelet | node-exporter-l5q6v | Created | Created container kube-rbac-proxy
10:39:27 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Started | Started container init-textfile
10:39:27 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/kube-scheduler-pod-3 -n openshift-kube-scheduler: cause by changes in data.pod.yaml
10:39:27 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:39:27 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionCreate | Revision 2 created because secret/serving-cert has changed
10:39:27 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/localhost-recovery-client-token-3 -n openshift-kube-scheduler because it was missing
10:39:27 | openshift-monitoring | kubelet | node-exporter-l5q6v | Created | Created container node-exporter
10:39:27 | openshift-monitoring | kubelet | node-exporter-l5q6v | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: secrets: kube-scheduler-client-cert-key\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:39:28 | openshift-etcd | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:39:28 | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed187e9c9e7585720b3b407ceeeb827d2d4d38fad8843cd914677eb3ab53c13b" in 5.699310179s
10:39:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 3" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:39:28 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Started | Started container kube-rbac-proxy
10:39:28 | openshift-etcd | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:39:28 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-2 -n openshift-kube-controller-manager: cause by changes in data.status
10:39:28 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Created | Created container kube-rbac-proxy
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" in 3.384553013s
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Created | Created container init-textfile
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Started | Started container init-textfile
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Created | Created container node-exporter
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Started | Started container node-exporter
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Created | Created container kube-rbac-proxy
10:39:28 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 3 because node master-0.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:39:28 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Started | Started container node-exporter
10:39:28 (x6) | openshift-controller-manager | kubelet | controller-manager-7lhx9 | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:28 | openshift-monitoring | kubelet | node-exporter-vrvq5 | Created | Created container node-exporter
10:39:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 3",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 3"
10:39:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 3"
10:39:28 | openshift-monitoring | kubelet | node-exporter-kcxrx | Started | Started container kube-rbac-proxy
10:39:29 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: aggregator-client-ca,client-ca, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: check-endpoints-client-cert-key,control-plane-node-admin-client-cert-key,internal-loadbalancer-serving-certkey,node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]"
10:39:29 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88fbce0b30a1d0786fbdfeb0f93431d4703f1a0ec974f7e9426859dee8619c9b"
10:39:29 | openshift-multus | multus | multus-admission-controller-nskb9 | AddedInterface | Add eth0 [10.129.0.6/23]
10:39:29 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-3 -n openshift-kube-controller-manager because it was missing
10:39:30 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:39:30 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-3 -n openshift-kube-controller-manager because it was missing
10:39:30 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:30 (x20) | openshift-etcd-operator | openshift-cluster-etcd-operator-staticpodstatecontroller | etcd-operator | MissingVersion | no image found for operand pod
10:39:31 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Created | Created container multus-admission-controller
10:39:31 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:31 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Started | Started container multus-admission-controller
10:39:31 (x5) | openshift-controller-manager | kubelet | controller-manager-pxstf | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:31 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88fbce0b30a1d0786fbdfeb0f93431d4703f1a0ec974f7e9426859dee8619c9b" in 1.675049145s
10:39:31 | openshift-etcd | static-pod-installer | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 2
10:39:31 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-3 -n openshift-kube-controller-manager because it was missing
10:39:32 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorVersionChanged | clusteroperator/etcd version "etcd" changed from "" to "4.6.52"
10:39:32 | openshift-kube-scheduler | multus | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.8/23]
10:39:32 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "" to "Progressing: daemonset/controller-manager: observed generation is 5, desired generation is 6.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1",Available message changed from "" to "Available: no daemon pods available on any node."
10:39:32 (x4) | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Created | Created container kube-rbac-proxy
10:39:32 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb"
10:39:32 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-pxstf
10:39:32 (x4) | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Started | Started container kube-rbac-proxy
10:39:32 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-7lhx9
10:39:32 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Created | Created container kube-rbac-proxy
10:39:32 | openshift-multus | kubelet | multus-admission-controller-nskb9 | Started | Started container kube-rbac-proxy
10:39:32 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-q5w7b
10:39:32 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: status.versions changed from [{"raw-internal" "4.6.52"} {"operator" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"operator" "4.6.52"} {"etcd" "4.6.52"}]
10:39:32 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-3 -n openshift-kube-controller-manager because it was missing
10:39:32 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pods \"etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:32 (x4) | openshift-monitoring | kubelet | cluster-monitoring-operator-565dd756d4-2st68 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:33 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/kube-controller-manager-pod -n openshift-kube-controller-manager: cause by changes in data.pod.yaml
10:39:33 | openshift-kube-scheduler | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:39:34 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-3 -n openshift-kube-controller-manager because it was missing
10:39:34 | openshift-controller-manager | default-scheduler | controller-manager-mpm5t | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-mpm5t to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:34 | openshift-kube-scheduler | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:39:34 | openshift-kube-scheduler | static-pod-installer | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 3
10:39:34 | openshift-kube-scheduler | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:39:35 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-3 -n openshift-kube-controller-manager because it was missing
10:39:35 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-3 -n openshift-kube-controller-manager because it was missing
10:39:35 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:35 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:39:35 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-ensure-env-vars
10:39:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe"
10:39:35 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObserveFeatureFlagsUpdated | Updated apiServerArguments.feature-gates to APIPriorityAndFairness=true,RotateKubeletServerCertificate=true,SupportPodPidsLimit=true,NodeDisruptionExclusion=true,ServiceNodeExclusion=true,SCTPSupport=true,LegacyNodeRoleBehavior=false
10:39:35 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObserveStorageUpdated | Updated storage urls to https://10.0.94.119:2379,https://10.0.94.219:2379,https://10.0.95.76:2379,https://localhost:2379
10:39:35 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" in 2.867155581s
10:39:35 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-ensure-env-vars
10:39:35 (x14) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-staticpodstatecontroller | openshift-kube-scheduler-operator | MissingVersion | no image found for operand pod
10:39:36 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:39:36 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-resources-copy
10:39:36 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-3 -n openshift-kube-controller-manager because it was missing
10:39:36 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-resources-copy
10:39:37 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:39:37 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-3 -n openshift-kube-controller-manager because it was missing
10:39:37 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcdctl
10:39:37 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorVersionChanged | clusteroperator/kube-scheduler version "operator" changed from "" to "4.6.52"
10:39:37 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:39:37 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorVersionChanged | clusteroperator/kube-scheduler version "kube-scheduler" changed from "" to "1.19.16"
10:39:37 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcdctl
10:39:37 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-metrics
10:39:37 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-metrics
10:39:37 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"operator" "4.6.52"} {"kube-scheduler" "1.19.16"}]
10:39:37 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"kube-scheduler" "1.19.16"} {"operator" "4.6.52"}]
10:39:38 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ + "admission": map[string]interface{}{ + "pluginConfig": map[string]interface{}{ + "network.openshift.io/ExternalIPRanger": map[string]interface{}{ + "configuration": map[string]interface{}{ + "allowIngressIP": bool(false), + "apiVersion": string("network.openshift.io/v1"), + "kind": string("ExternalIPRangerAdmissionConfig"), + }, + }, + "network.openshift.io/RestrictedEndpointsAdmission": map[string]interface{}{ + "configuration": map[string]interface{}{ + "apiVersion": string("network.openshift.io/v1"), + "kind": string("RestrictedEndpointsAdmissionConfig"), + "restrictedCIDRs": []interface{}{string("10.128.0.0/14"), string("172.30.0.0/16")}, + }, + }, + }, + }, + "apiServerArguments": map[string]interface{}{ + "audit-policy-file": []interface{}{ + string("/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-audit-policies/default.yaml"), + }, + "etcd-servers": []interface{}{ + string("https://10.0.94.119:2379"), + string("https://10.0.94.219:2379"), + string("https://10.0.95.76:2379"), + string("https://localhost:2379"), + }, + "feature-gates": []interface{}{ + string("APIPriorityAndFairness=true"), + string("RotateKubeletServerCertificate=true"), + string("SupportPodPidsLimit=true"), + string("NodeDisruptionExclusion=true"), + string("ServiceNodeExclusion=true"), + string("SCTPSupport=true"), + string("LegacyNodeRoleBehavior=false"), + }, + }, + "corsAllowedOrigins": []interface{}{string(`//127\.0\.0\.1(:|$)`), string("//localhost(:|$)")}, + "servicesSubnet": string("172.30.0.0/16"), + "servingInfo": map[string]interface{}{ + "bindAddress": string("0.0.0.0:6443"), + "bindNetwork": string("tcp4"), + "cipherSuites": []interface{}{ + string("TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"), + string("TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"), + string("TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"), + string("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"), + string("TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"), + string("TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"), + }, + "minTLSVersion": string("VersionTLS12"), + "namedCertificates": []interface{}{ + map[string]interface{}{ + "certFile": string("/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt"), + "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key"), + }, + map[string]interface{}{ + "certFile": string("/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt"), + "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"), + }, + map[string]interface{}{ + "certFile": string("/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt"), + "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key"), + }, + map[string]interface{}{ + "certFile": string("/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt"), + "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key"), + }, + map[string]interface{}{ + "certFile": string("/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt"), + "keyFile": string("/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key"), + }, + }, + }, }
10:39:38 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-3 -n openshift-kube-controller-manager because it was missing
10:39:38 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObserveAPIServerArgumentsAudit | audit policy has been set to profile=Default
10:39:39 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-3 -n openshift-kube-controller-manager because it was missing
10:39:39 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 2 created because configmap/config has changed,configmap/cluster-policy-controller-config has changed
10:39:39 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 3",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 3"
10:39:39 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 4 triggered by "configmap/kube-controller-manager-pod has changed"
10:39:40 (x6) | openshift-controller-manager | kubelet | controller-manager-q5w7b | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:40 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-4 -n openshift-kube-controller-manager because it was missing
10:39:41 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:41 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Created | Created container kube-rbac-proxy
10:39:41 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-3 -n openshift-kube-controller-manager: cause by changes in data.status
10:39:41 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:37 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:41 | openshift-cluster-version | kubelet | cluster-version-operator-5d86d65f6b-4fm8l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:39:42 | openshift-multus | multus | multus-admission-controller-bkzjk | AddedInterface | Add eth0 [10.128.0.15/23]
10:39:42 | openshift-controller-manager | default-scheduler | controller-manager-nv4gt | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-nv4gt to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:42 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Started | Started container kube-rbac-proxy
10:39:42 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f821afd6941c9070e6cfc5e3e6169d6b16168ad9fca89a5e0265a863f867ea9e"
10:39:43 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88fbce0b30a1d0786fbdfeb0f93431d4703f1a0ec974f7e9426859dee8619c9b"
10:39:43 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:39:43 | openshift-marketplace | multus | marketplace-operator-664c8865b-d9dx9 | AddedInterface | Add eth0 [10.128.0.17/23]
10:39:43 | openshift-ingress-operator | multus | ingress-operator-cc4bb6454-wqjrn | AddedInterface | Add eth0 [10.128.0.16/23]
10:39:43 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-4 -n openshift-kube-controller-manager because it was missing
10:39:43 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:39:43 | openshift-operator-lifecycle-manager | multus | olm-operator-5fbfc8c78-h4nwt | AddedInterface | Add eth0 [10.128.0.18/23]
10:39:43 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" in 7.589925206s
10:39:43 | openshift-dns-operator | multus | dns-operator-6979ff5964-t5q7m | AddedInterface | Add eth0 [10.128.0.14/23]
10:39:44 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-4 -n openshift-kube-controller-manager because it was missing
10:39:44 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82de2aba4a537a4feeeb610784d934404be43d0e72c44bf9418f1999d2b4c92b"
10:39:44 | openshift-operator-lifecycle-manager | kubelet | olm-operator-5fbfc8c78-h4nwt | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082"
10:39:44 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:44 | openshift-cloud-credential-operator | multus | cloud-credential-operator-78479968fc-vnsj9 | AddedInterface | Add eth0 [10.130.0.9/23]
10:39:44 | openshift-controller-manager | default-scheduler | controller-manager-wf9nk | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-wf9nk to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:44 | openshift-multus | multus | multus-admission-controller-gsjzb | AddedInterface | Add eth0 [10.130.0.11/23]
10:39:44 | openshift-marketplace | kubelet | marketplace-operator-664c8865b-d9dx9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ef555704335b67df71a9bbb9b0b08fd28cf1e97abe1a78848b828e087a632ab"
10:39:44 (x11) | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: client-ca, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0
10:39:44 | openshift-operator-lifecycle-manager | multus | catalog-operator-887559f54-g4cgp | AddedInterface | Add eth0 [10.130.0.12/23]
10:39:45 | openshift-image-registry | kubelet | cluster-image-registry-operator-98c4d4975-kxn6n | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf64307a7341e67a8a6cea2649cfa1734be025fd1ec515d9f739f9cc9038764c"
10:39:45 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88fbce0b30a1d0786fbdfeb0f93431d4703f1a0ec974f7e9426859dee8619c9b"
10:39:45 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:39:45 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:39:45 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:39:45 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:39:45 | openshift-image-registry | multus | cluster-image-registry-operator-98c4d4975-kxn6n | AddedInterface | Add eth0 [10.130.0.10/23]
10:39:45 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Created | Created container kube-rbac-proxy
10:39:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-4 -n openshift-kube-controller-manager because it was missing
10:39:45 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f821afd6941c9070e6cfc5e3e6169d6b16168ad9fca89a5e0265a863f867ea9e" in 3.452037241s
10:39:45 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Created | Created container multus-admission-controller
10:39:45 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Started | Started container multus-admission-controller
10:39:45 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:45 | openshift-operator-lifecycle-manager | kubelet | catalog-operator-887559f54-g4cgp | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082"
10:39:45 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ecd3287693f1fb5c4fc5a0c93bd8885850a2935ac2dd6ee3dedbacefafb6242"
10:39:45 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:37 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:37 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:45 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Started | Started container kube-rbac-proxy
10:39:45 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Created | Created container machine-approver-controller
10:39:45 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:de73c710774cc4e1e291afcaf2e80647b16efd8fc6565c6407684c30f48c6df5"
10:39:45 | openshift-cluster-version | kubelet | cluster-version-operator-5d86d65f6b-4fm8l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6" in 3.817468773s
10:39:45 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88fbce0b30a1d0786fbdfeb0f93431d4703f1a0ec974f7e9426859dee8619c9b" in 2.428630793s
10:39:46 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Started | Started container kube-rbac-proxy
10:39:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:39:46 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-2 -n openshift-kube-apiserver because it was missing
10:39:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
10:39:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
10:39:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:39:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:39:46 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 2 triggered by "configmap \"kube-apiserver-pod\" not found"
10:39:46 | openshift-cluster-machine-approver | kubelet | machine-approver-58d468647f-z7j52 | Started | Started container machine-approver-controller
10:39:46 | openshift-multus | kubelet | multus-admission-controller-bkzjk | Created | Created container kube-rbac-proxy
10:39:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-4 -n openshift-kube-controller-manager because it was missing
10:39:47 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-1 -n openshift-kube-apiserver: cause by changes in data.status
10:39:48 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:de73c710774cc4e1e291afcaf2e80647b16efd8fc6565c6407684c30f48c6df5" in 3.586468673s
10:39:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-4 -n openshift-kube-controller-manager because it was missing
10:39:48 | openshift-marketplace | kubelet | marketplace-operator-664c8865b-d9dx9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ef555704335b67df71a9bbb9b0b08fd28cf1e97abe1a78848b828e087a632ab" in 4.281424907s
10:39:48 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Created | Created container ingress-operator
10:39:48 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82de2aba4a537a4feeeb610784d934404be43d0e72c44bf9418f1999d2b4c92b" in 4.258942727s
10:39:49 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Created | Created container dns-operator
10:39:49 | openshift-marketplace | kubelet | marketplace-operator-664c8865b-d9dx9 | Created | Created container marketplace-operator
10:39:49 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88fbce0b30a1d0786fbdfeb0f93431d4703f1a0ec974f7e9426859dee8619c9b" in 4.119418903s
10:39:49 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-0,config-0,etcd-serving-ca-0,kube-apiserver-audit-policies-0,kube-apiserver-cert-syncer-kubeconfig-0,kube-apiserver-pod-0,kubelet-serving-ca-0,sa-token-signing-certs-0, secrets: node-kubeconfigs, secrets: etcd-client-0,kubelet-client-0,localhost-recovery-client-token-0,localhost-recovery-serving-certkey-0]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]",Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1"),Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1"
10:39:49 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Started | Started container ingress-operator
10:39:49 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Started | Started container dns-operator
10:39:49 | openshift-marketplace | kubelet | marketplace-operator-664c8865b-d9dx9 | Started | Started container marketplace-operator
10:39:50 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-4 -n openshift-kube-controller-manager because it was missing
10:39:51 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Created | Created container multus-admission-controller
10:39:51 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Started | Started container multus-admission-controller
10:39:51 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "Progressing: daemonset/controller-manager: observed generation is 5, desired generation is 6.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1" to "",Available message changed from "Available: no daemon pods available on any node." to ""
10:39:51 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Created | Created container kube-rbac-proxy
10:39:51 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Started | Started container kube-rbac-proxy
10:39:51 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Started | Started container kube-rbac-proxy
10:39:51 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Created | Created container kube-rbac-proxy
10:39:51 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:51 | openshift-dns | default-scheduler | dns-default-ks2fv | Scheduled | Successfully assigned openshift-dns/dns-default-ks2fv to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:51 | openshift-operator-lifecycle-manager | kubelet | olm-operator-5fbfc8c78-h4nwt | Started | Started container olm-operator
10:39:51 | openshift-ingress-operator | kubelet | ingress-operator-cc4bb6454-wqjrn | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:51 | openshift-dns | default-scheduler | dns-default-bgt8d | Scheduled | Successfully assigned openshift-dns/dns-default-bgt8d to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:39:51 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-4 -n openshift-kube-controller-manager because it was missing
10:39:51 | openshift-operator-lifecycle-manager | kubelet | olm-operator-5fbfc8c78-h4nwt | Created | Created container olm-operator
10:39:51 | openshift-operator-lifecycle-manager | kubelet | olm-operator-5fbfc8c78-h4nwt | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082" in 6.670213769s
10:39:51 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Started | Started container kube-rbac-proxy
10:39:51 | openshift-dns | default-scheduler | dns-default-z5jj6 | Scheduled | Successfully assigned openshift-dns/dns-default-z5jj6 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:51 | openshift-image-registry | image-registry-operator | openshift-master-controllers | LeaderElection | cluster-image-registry-operator-98c4d4975-kxn6n_631acae8-5d4b-45a1-9cb8-d53b632b724a became leader
10:39:51 | openshift-dns | daemonset-controller | dns-default | SuccessfulCreate | Created pod: dns-default-z5jj6
10:39:51 | openshift-dns-operator | kubelet | dns-operator-6979ff5964-t5q7m | Created | Created container kube-rbac-proxy
10:39:51 | openshift-operator-lifecycle-manager | kubelet | catalog-operator-887559f54-g4cgp | Created | Created container catalog-operator
10:39:51 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ecd3287693f1fb5c4fc5a0c93bd8885850a2935ac2dd6ee3dedbacefafb6242" in 6.036925061s
10:39:51 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Created | Created container cloud-credential-operator
10:39:51 | openshift-cloud-credential-operator | kubelet | cloud-credential-operator-78479968fc-vnsj9 | Started | Started container cloud-credential-operator
10:39:51 | openshift-operator-lifecycle-manager | kubelet | catalog-operator-887559f54-g4cgp | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082" in 6.072032582s
10:39:51 | openshift-multus | kubelet | multus-admission-controller-gsjzb | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:51 | openshift-dns | daemonset-controller | dns-default | SuccessfulCreate | Created pod: dns-default-ks2fv
10:39:51 | openshift-operator-lifecycle-manager | kubelet | catalog-operator-887559f54-g4cgp | Started | Started container catalog-operator
10:39:51 | openshift-image-registry | kubelet | cluster-image-registry-operator-98c4d4975-kxn6n | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf64307a7341e67a8a6cea2649cfa1734be025fd1ec515d9f739f9cc9038764c" in 5.575088167s
10:39:51 | openshift-dns | daemonset-controller | dns-default | SuccessfulCreate | Created pod: dns-default-bgt8d
10:39:51 | openshift-dns | kubelet | dns-default-ks2fv | FailedMount | MountVolume.SetUp failed for volume "metrics-tls" : secret "dns-default-metrics-tls" not found
10:39:52 (x150) | openshift-kube-apiserver-operator | kube-apiserver-operator-target-config-controller-targetconfigcontroller | kube-apiserver-operator | ConfigMissing | no observedConfig
10:39:52 (x3) | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | (combined from similar events): Deleted pod: controller-manager-mpm5t
10:39:52 | openshift-ingress-operator | certificate_controller | router-ca | CreatedWildcardCACert | Created a default wildcard CA certificate
10:39:52 (x6) | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | DaemonSetUpdated | Updated DaemonSet.apps/controller-manager -n openshift-controller-manager because it changed
10:39:52 | openshift-dns | kubelet | dns-default-bgt8d | FailedMount | MountVolume.SetUp failed for volume "metrics-tls" : secret "dns-default-metrics-tls" not found
10:39:52 | openshift-dns | kubelet | dns-default-z5jj6 | FailedMount | MountVolume.SetUp failed for volume "metrics-tls" : secret "dns-default-metrics-tls" not found
10:39:52 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-4 -n openshift-kube-controller-manager because it was missing
10:39:53 (x2) | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd
10:39:53 (x2) | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd
10:39:53 | openshift-machine-api | multus | cluster-autoscaler-operator-557599ccf8-lchg2 | AddedInterface | Add eth0 [10.130.0.13/23]
10:39:53 | openshift-ingress | replicaset-controller | router-default-5b76b97bbb | SuccessfulCreate | Created pod: router-default-5b76b97bbb-ltvlc
10:39:53 | openshift-operator-lifecycle-manager | operator-lifecycle-manager | packageserver | RequirementsUnknown | requirements not yet checked
10:39:53 (x2) | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:39:53 | openshift-ingress-operator | ingress_controller | default | Admitted | ingresscontroller passed validation
10:39:53 | openshift-ingress | deployment-controller | router-default | ScalingReplicaSet | Scaled up replica set router-default-5b76b97bbb to 2
10:39:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-4 -n openshift-kube-controller-manager because it was missing
10:39:53 | openshift-ingress | replicaset-controller | router-default-5b76b97bbb | SuccessfulCreate | Created pod: router-default-5b76b97bbb-jp2g9
10:39:54 | openshift-dns | kubelet | dns-default-ks2fv | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf"
10:39:54 | openshift-dns | multus | dns-default-bgt8d | AddedInterface | Add eth0 [10.129.0.7/23]
10:39:54 | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c41fd0616e68c5a87537186bc935c5d846d2beaef2c8b2d9343ceec4b6263be"
10:39:54 | openshift-authentication-operator | cluster-authentication-operator-resource-sync-controller-resourcesynccontroller | authentication-operator | SecretCreated | Created Secret/v4-0-config-system-router-certs -n openshift-authentication because it was missing
10:39:54 | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Started | Started container kube-rbac-proxy
10:39:54 | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:54 | openshift-config-managed | certificate_controller | default-ingress-cert | PublishedRouterCA | Published "default-ingress-cert" in "openshift-config-managed"
10:39:54 | openshift-operator-lifecycle-manager | operator-lifecycle-manager | packageserver | AllRequirementsMet | all requirements found, attempting install
10:39:54 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-4 -n openshift-kube-controller-manager because it was missing
10:39:54 | openshift-dns | multus | dns-default-z5jj6 | AddedInterface | Add eth0 [10.130.0.14/23]
10:39:54 | openshift-ingress-operator | certificate_controller | default | CreatedDefaultCertificate | Created default wildcard certificate "router-certs-default"
10:39:54 | openshift-config-managed | certificate_publisher_controller | router-certs | PublishedRouterCertificates | Published router certificates
10:39:54 | openshift-dns | kubelet | dns-default-bgt8d | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf"
10:39:54 | openshift-dns | multus | dns-default-ks2fv | AddedInterface | Add eth0 [10.128.0.19/23]
10:39:54 | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Created | Created container kube-rbac-proxy
10:39:54 | openshift-dns | kubelet | dns-default-z5jj6 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf"
10:39:55 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:37 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: unknown reason\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is terminated: Error: ootstrap\", peerURLs=[https://10.0.89.186:2380}, clientURLs=[https://10.0.89.186:2379]\nStaticPodsDegraded: member \"https://10.0.94.219:2380\" not found in member list, check operator logs for possible scaling problems\nStaticPodsDegraded: #### sleeping...\nStaticPodsDegraded: #### attempt 3\nStaticPodsDegraded: member={name=\"etcd-bootstrap\", peerURLs=[https://10.0.89.186:2380}, clientURLs=[https://10.0.89.186:2379]\nStaticPodsDegraded: #### attempt 4\nStaticPodsDegraded: #### attempt 5\nStaticPodsDegraded: #### attempt 6\nStaticPodsDegraded: #### attempt 7\nStaticPodsDegraded: #### attempt 8\nStaticPodsDegraded: #### attempt 9\nStaticPodsDegraded: timed out\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:55 | openshift-operator-lifecycle-manager | replicaset-controller | packageserver-6886676455 | SuccessfulCreate | Created pod: packageserver-6886676455-bnt4p
10:39:55 | openshift-operator-lifecycle-manager | replicaset-controller | packageserver-6886676455 | SuccessfulCreate | Created pod: packageserver-6886676455-gjjmp
10:39:55 | openshift-operator-lifecycle-manager | deployment-controller | packageserver | ScalingReplicaSet | Scaled up replica set packageserver-6886676455 to 2
10:39:55 | openshift-operator-lifecycle-manager | default-scheduler | packageserver-6886676455-gjjmp | Scheduled | Successfully assigned openshift-operator-lifecycle-manager/packageserver-6886676455-gjjmp to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:39:55 | openshift-operator-lifecycle-manager | default-scheduler | packageserver-6886676455-bnt4p | Scheduled | Successfully assigned openshift-operator-lifecycle-manager/packageserver-6886676455-bnt4p to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:39:55 (x7) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: client-ca
10:39:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 3 created because configmap/kube-controller-manager-pod has changed
10:39:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 3" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 4",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 3" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 4"
10:39:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-3 -n openshift-kube-scheduler: cause by changes in data.status
10:39:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/revision-status-4 -n openshift-kube-scheduler because it was missing
10:39:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/serviceaccount-ca -n openshift-kube-scheduler: cause by changes in data.ca-bundle.crt
10:39:55 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 4 triggered by "configmap/serviceaccount-ca has changed"
10:39:55 | openshift-kube-apiserver-operator | kube-apiserver-operator-target-config-controller-targetconfigcontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/config -n openshift-kube-apiserver because it was missing
10:39:55 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nRouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \"v4-0-config-system-router-certs\" not found\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-4 -n openshift-kube-controller-manager because it was missing
10:39:56 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
10:39:56 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found"
10:39:56 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-pod-4 -n openshift-kube-scheduler because it was missing
10:39:56 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/config-4 -n openshift-kube-scheduler because it was missing
10:39:56 (x4) | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
10:39:56 | openshift-dns | kubelet | dns-default-ks2fv | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf" in 1.993395731s
10:39:56 | openshift-operator-lifecycle-manager | multus | packageserver-6886676455-bnt4p | AddedInterface | Add eth0 [10.128.0.20/23]
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Started | Started container dns
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Created | Created container dns
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf" in 2.329021108s
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Created | Created container kube-rbac-proxy
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d"
10:39:57 | openshift-operator-lifecycle-manager | kubelet | packageserver-6886676455-gjjmp | Created | Created container packageserver
10:39:57 | openshift-operator-lifecycle-manager | kubelet | packageserver-6886676455-gjjmp | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082" already present on machine
10:39:57 | openshift-kube-apiserver-operator | kube-apiserver-operator-target-config-controller-targetconfigcontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-pod -n openshift-kube-apiserver because it was missing
10:39:57 | openshift-dns | kubelet | dns-default-bgt8d | Started | Started container kube-rbac-proxy
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Started | Started container kube-rbac-proxy
10:39:57 | openshift-dns | kubelet | dns-default-ks2fv | Created | Created container dns
10:39:57 | openshift-dns | kubelet | dns-default-ks2fv | Started | Started container dns
10:39:57 | openshift-dns | kubelet | dns-default-ks2fv | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:57 | openshift-dns | kubelet | dns-default-ks2fv | Created | Created container kube-rbac-proxy
10:39:57 | openshift-dns | kubelet | dns-default-ks2fv | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d"
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Created | Created container kube-rbac-proxy
10:39:57 | openshift-operator-lifecycle-manager | kubelet | packageserver-6886676455-bnt4p | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082" already present on machine
10:39:57 | openshift-operator-lifecycle-manager | kubelet | packageserver-6886676455-bnt4p | Created | Created container packageserver
10:39:57 | openshift-operator-lifecycle-manager | kubelet | packageserver-6886676455-bnt4p | Started | Started container packageserver
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Created | Created container dns
10:39:57 | openshift-operator-lifecycle-manager | multus | packageserver-6886676455-gjjmp | AddedInterface | Add eth0 [10.130.0.15/23]
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Started | Started container dns
10:39:57 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-4 -n openshift-kube-scheduler because it was missing
10:39:57 | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c41fd0616e68c5a87537186bc935c5d846d2beaef2c8b2d9343ceec4b6263be" in 2.671054517s
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d"
10:39:57 | openshift-operator-lifecycle-manager | kubelet | packageserver-6886676455-gjjmp | Started | Started container packageserver
10:39:57 | openshift-dns | kubelet | dns-default-ks2fv | Started | Started container kube-rbac-proxy
10:39:57 | openshift-dns | kubelet | dns-default-z5jj6 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf" in 2.686079646s
10:39:58 (x69) | openshift-authentication-operator | oauth-apiserver-workload-controller | authentication-operator | PrereqNotReady | clusteroperator/kube-apiserver is not Available
10:39:58 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveRouterSecret | namedCertificates changed to []interface {}{map[string]interface {}{"certFile":"/var/config/system/secrets/v4-0-config-system-router-certs/apps.case03116926.lab.upshift.rdu2.redhat.com", "keyFile":"/var/config/system/secrets/v4-0-config-system-router-certs/apps.case03116926.lab.upshift.rdu2.redhat.com", "names":[]interface {}{"*.apps.case03116926.lab.upshift.rdu2.redhat.com"}}}
10:39:58 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/scheduler-kubeconfig-4 -n openshift-kube-scheduler because it was missing
10:39:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 5 triggered by "configmap/serviceaccount-ca has changed"
10:39:58 (x2) | openshift-operator-lifecycle-manager | operator-lifecycle-manager | packageserver | InstallSucceeded | waiting for install components to report healthy
10:39:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: unknown reason\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is terminated: Error: ootstrap\", peerURLs=[https://10.0.89.186:2380}, clientURLs=[https://10.0.89.186:2379]\nStaticPodsDegraded: member \"https://10.0.94.219:2380\" not found in member list, check operator logs for possible scaling problems\nStaticPodsDegraded: #### sleeping...\nStaticPodsDegraded: #### attempt 3\nStaticPodsDegraded: member={name=\"etcd-bootstrap\", peerURLs=[https://10.0.89.186:2380}, clientURLs=[https://10.0.89.186:2379]\nStaticPodsDegraded: #### attempt 4\nStaticPodsDegraded: #### attempt 5\nStaticPodsDegraded: #### attempt 6\nStaticPodsDegraded: #### attempt 7\nStaticPodsDegraded: #### attempt 8\nStaticPodsDegraded: #### attempt 9\nStaticPodsDegraded: timed out\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:39:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/serviceaccount-ca -n openshift-kube-controller-manager: cause by changes in data.ca-bundle.crt
10:39:58 | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c651f0af4ba650af214a59e3f974b72caeda32535ce116617ea588e3fdb762e0"
10:39:58 (x6) | openshift-controller-manager | kubelet | controller-manager-nv4gt | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:39:58 | openshift-monitoring | multus | prometheus-operator-6db8fc66c7-rgvbc | AddedInterface | Add eth0 [10.130.0.16/23]
10:39:58 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthServerConfigObservationDegraded: secret \"v4-0-config-system-router-certs\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
10:39:58 | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObservedConfigChanged | Writing updated section ("oauthServer") of observed config: " map[string]interface{}{\n \t\"corsAllowedOrigins\": []interface{}{string(`//127\\.0\\.0\\.1(:|$)`), string(\"//localhost(:|$)\")},\n \t\"oauthConfig\": map[string]interface{}{\"assetPublicURL\": string(\"\"), \"loginURL\": string(\"https://api.case03116926.lab.upshift.rdu2.redhat.com:6443\"), \"templates\": map[string]interface{}{\"error\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/errors.html\"), \"login\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/login.html\"), \"providerSelection\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/providers.html\")}, \"tokenConfig\": map[string]interface{}{\"accessTokenMaxAgeSeconds\": float64(86400), \"authorizeTokenMaxAgeSeconds\": float64(300)}},\n \t\"servingInfo\": map[string]interface{}{\n \t\t\"cipherSuites\": []interface{}{string(\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\"), string(\"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\"), string(\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"), string(\"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\"), string(\"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\"), string(\"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\")},\n \t\t\"minTLSVersion\": string(\"VersionTLS12\"),\n+ \t\t\"namedCertificates\": []interface{}{\n+ \t\t\tmap[string]interface{}{\n+ \t\t\t\t\"certFile\": string(\"/var/config/system/secrets/v4-0-config-system-router-certs/apps.case03116926.lab.upshift.rdu2.redhat.com\"),\n+ \t\t\t\t\"keyFile\": string(\"/var/config/system/secrets/v4-0-config-system-router-certs/apps.case03116926.lab.upshift.rdu2.redhat.com\"),\n+ \t\t\t\t\"names\": []interface{}{string(\"*.apps.case03116926.lab.upshift.rdu2.redhat.com\")},\n+ \t\t\t},\n+ \t\t},\n \t},\n \t\"volumesToMount\": map[string]interface{}{\"identityProviders\": string(\"{}\")},\n }\n"
10:39:59 | openshift-dns | kubelet | dns-default-ks2fv | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" in 2.300072306s
10:39:59 | openshift-dns | kubelet | dns-default-ks2fv | Created | Created container dns-node-resolver
10:39:59 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-cert-syncer-kubeconfig-4 -n openshift-kube-scheduler because it was missing
10:39:59 | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Unhealthy | Liveness probe failed: Get "https://10.130.0.6:8443/healthz": dial tcp 10.130.0.6:8443: connect: connection refused
10:39:59 | openshift-dns | kubelet | dns-default-ks2fv | Started | Started container dns-node-resolver
10:39:59 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/serving-cert-4 -n openshift-kube-scheduler because it was missing
10:39:59 (x2) | openshift-operator-lifecycle-manager | operator-lifecycle-manager | packageserver | InstallWaiting | APIServices not installed
10:39:59 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-5 -n openshift-kube-controller-manager because it was missing
10:39:59 (x22) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SATokenSignerControllerStuck | unexpected addresses: 10.0.89.186
10:39:59 | openshift-kube-apiserver-operator | kube-apiserver-operator-target-config-controller-targetconfigcontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/client-ca -n openshift-kube-apiserver because it was missing
10:40:00 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-4 -n openshift-kube-controller-manager: cause by changes in data.status
10:40:00 | openshift-dns | kubelet | dns-default-bgt8d | Started | Started container dns-node-resolver
10:40:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
10:40:00 (x20) | openshift-kube-apiserver-operator | kube-apiserver-operator-staticpodstatecontroller | kube-apiserver-operator | MissingVersion | no image found for operand pod
10:40:00 | openshift-dns | kubelet | dns-default-bgt8d | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" in 2.661760588s
10:40:00 (x6) | openshift-controller-manager | kubelet | controller-manager-wf9nk | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:40:00 | openshift-dns | kubelet | dns-default-bgt8d | Created | Created container dns-node-resolver
10:40:00 | openshift-dns | kubelet | dns-default-z5jj6 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" in 3.008481646s
10:40:00 (x4) | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Unhealthy | Readiness probe failed: dial tcp 10.0.94.219:2380: connect: connection refused
10:40:01 | openshift-authentication-operator | cluster-authentication-operator | cluster-authentication-operator-lock | LeaderElection | authentication-operator-6fcbcf79b6-5h2gz_b6c8ca0e-47ff-4466-98c3-77bac1018604 became leader
10:40:01 | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Started | Started container prometheus-operator
10:40:01 | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Created | Created container prometheus-operator
10:40:01 | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c651f0af4ba650af214a59e3f974b72caeda32535ce116617ea588e3fdb762e0" in 2.621586105s
10:40:01 | openshift-etcd-operator | openshift-cluster-etcd-operator | openshift-cluster-etcd-operator-lock | LeaderElection | etcd-operator-859798c7f4-kb5lq_02b0f5d9-9c2c-41fc-ad32-5e15a246e928 became leader
10:40:01 | openshift-dns | kubelet | dns-default-z5jj6 | Started | Started container dns-node-resolver
10:40:01 | openshift-dns | kubelet | dns-default-z5jj6 | Created | Created container dns-node-resolver
10:40:01 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-5 -n openshift-kube-controller-manager because it was missing
10:40:02 (x4) | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | (combined from similar events): Created pod: controller-manager-vc55x
10:40:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-5 -n openshift-kube-controller-manager because it was missing
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "BootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-member-observer-controller-etcdmemberscontroller | etcd-operator | ReportEtcdMembersErrorUpdatingStatus | etcds.operator.openshift.io "cluster" not found
10:40:02 | openshift-kube-apiserver-operator | kube-apiserver-operator | kube-apiserver-operator-lock | LeaderElection | kube-apiserver-operator-c74696d8d-gsntz_12b45cd8-f925-45de-b00e-7572ff5be42c became leader
10:40:02 (x2) | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "BootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:02 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "InstallerControllerDegraded: missing required resources: [configmaps: config-3,kube-scheduler-cert-syncer-kubeconfig-3,kube-scheduler-pod-3,scheduler-kubeconfig-3,serviceaccount-ca-3, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-3]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:40:02 | openshift-config-operator | config-operator | config-operator-lock | LeaderElection | openshift-config-operator-697f744f69-g6ck2_a04c4ba9-7b4e-46e7-a8a5-78a9aab6154d became leader
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:40:02 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [configmaps: config-3,kube-scheduler-cert-syncer-kubeconfig-3,kube-scheduler-pod-3,scheduler-kubeconfig-3,serviceaccount-ca-3, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-3]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "InstallerControllerDegraded: missing required resources: [secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-3]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.219:2379 https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.89.186:2379]: context deadline exceeded\nStaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready"
10:40:02 (x17) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-staticpodstatecontroller | kube-controller-manager-operator | MissingVersion | no image found for operand pod
10:40:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-loggingsyncer | etcd-operator | OperatorLogLevelChange | Operator log level changed from "Debug" to "Normal"
10:40:02 | openshift-controller-manager | default-scheduler | controller-manager-vc55x | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-vc55x to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:40:02 (x5) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | RequiredInstallerResourcesMissing | configmaps: config-3,kube-scheduler-cert-syncer-kubeconfig-3,kube-scheduler-pod-3,scheduler-kubeconfig-3,serviceaccount-ca-3, secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-3
10:40:02 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | RequiredInstallerResourcesMissing | secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-3
10:40:02 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-lock | LeaderElection | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k_b00268a6-0a06-4865-bb9f-1f1b7e963209 became leader
10:40:03 | openshift-etcd-operator | openshift-cluster-etcd-operator-cluster-member-controller-clustermembercontroller | etcd-operator | FoundPodToScale | found pod to add to etcd membership: etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com
10:40:03 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nClusterMemberControllerDegraded: failed to get etcd client: failed to make etcd client for endpoints [https://10.0.94.119:2379 https://10.0.95.76:2379 https://10.0.94.219:2379 https://10.0.89.186:2379]: context deadline exceeded\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:03 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 2 triggered by "configmap \"kube-apiserver-pod-1\" not found"
10:40:03 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | MemberAdd | adding new peer https://10.0.94.219:2380
10:40:03 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused"
10:40:03 (x2) | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Started | Started container kube-rbac-proxy
10:40:03 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-5 -n openshift-kube-controller-manager because it was missing
10:40:03 (x2) | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Created | Created container kube-rbac-proxy
10:40:05 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
10:40:05 | openshift-controller-manager | default-scheduler | controller-manager-5qdng | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-5qdng to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:40:05 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server",Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused"
10:40:05 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionCreate | Revision 3 created because configmap/serviceaccount-ca has changed
10:40:05 | openshift-controller-manager | default-scheduler | controller-manager-c9z4d | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-c9z4d to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:40:05 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/localhost-recovery-client-token-4 -n openshift-kube-scheduler because it was missing
10:40:05 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 4 triggered by "configmap/serviceaccount-ca has changed"
10:40:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-client-ca -n openshift-config-managed because it was missing
10:40:05 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-5 -n openshift-kube-controller-manager because it was missing
10:40:05 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": secret \"node-system-admin-client\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
10:40:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-pod-2 -n openshift-kube-apiserver because it was missing
10:40:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator-lock | LeaderElection | kube-controller-manager-operator-7bb9b6d8b8-h8hjn_d927429a-bf3d-4a4f-a9de-203494db50c9 became leader
10:40:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: configmaps: client-ca" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: configmaps: client-ca\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled"
10:40:06 (x7) | openshift-controller-manager | kubelet | controller-manager-mpm5t | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:40:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 5 triggered by "configmap/serviceaccount-ca has changed"
10:40:06 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 4"
10:40:06 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 4" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:06 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "InstallerControllerDegraded: missing required resources: [secrets: kube-scheduler-client-cert-key, secrets: localhost-recovery-client-token-3]\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready",Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 3" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 4",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 3" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 4"
10:40:07 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/config-2 -n openshift-kube-apiserver because it was missing
10:40:07 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2",Available changed from False to True ("StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2\nEtcdMembersAvailable: 1 members are available")
10:40:07 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because static pod is ready
10:40:07 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: context canceled\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:08 (x2) | openshift-etcd | controllermanager | etcd-quorum-guard | NoPods | No matching pods found
10:40:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-cert-syncer-kubeconfig-2 -n openshift-kube-apiserver because it was missing
10:40:08 (x9) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | RequiredInstallerResourcesMissing | configmaps: client-ca
10:40:08 | openshift-etcd | replicaset-controller | etcd-quorum-guard-58669bc4d5 | SuccessfulCreate | Created pod: etcd-quorum-guard-58669bc4d5-lz6pp
10:40:08 | openshift-etcd | replicaset-controller | etcd-quorum-guard-58669bc4d5 | SuccessfulCreate | Created pod: etcd-quorum-guard-58669bc4d5-rj76b
10:40:08 | openshift-etcd | replicaset-controller | etcd-quorum-guard-58669bc4d5 | SuccessfulCreate | Created pod: etcd-quorum-guard-58669bc4d5-flgvm
10:40:08 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:08 | openshift-etcd | default-scheduler | etcd-quorum-guard-58669bc4d5-rj76b | Scheduled | Successfully assigned openshift-etcd/etcd-quorum-guard-58669bc4d5-rj76b to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:40:08 | openshift-etcd | default-scheduler | etcd-quorum-guard-58669bc4d5-lz6pp | Scheduled | Successfully assigned openshift-etcd/etcd-quorum-guard-58669bc4d5-lz6pp to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:40:08 | openshift-etcd | default-scheduler | etcd-quorum-guard-58669bc4d5-flgvm | Scheduled | Successfully assigned openshift-etcd/etcd-quorum-guard-58669bc4d5-flgvm to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:40:08 | openshift-etcd | deployment-controller | etcd-quorum-guard | ScalingReplicaSet | Scaled up replica set etcd-quorum-guard-58669bc4d5 to 3
10:40:09 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-5 -n openshift-kube-controller-manager because it was missing
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-flgvm | Started | Started container guard
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-lz6pp | Created | Created container guard
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-lz6pp | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" already present on machine
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-flgvm | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" already present on machine
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-lz6pp | Started | Started container guard
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-rj76b | Started | Started container guard
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-rj76b | Created | Created container guard
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-flgvm | Created | Created container guard
10:40:09 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-resource-sync-controller-resourcesynccontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/client-ca -n openshift-kube-controller-manager because it was missing
10:40:09 | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-rj76b | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" already present on machine
10:40:10 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-0.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:10 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 4 because node master-0.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:10 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:40:10 (x5) | openshift-controller-manager | kubelet | controller-manager-vc55x | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:40:10 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: configmaps: client-ca\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled" to "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled"
10:40:10 | openshift-etcd-operator | openshift-cluster-etcd-operator-prunecontroller | etcd-operator | PodCreated | Created Pod/revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:40:10 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-3 -n openshift-kube-scheduler: cause by changes in data.status
10:40:11 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator-lock | LeaderElection | openshift-controller-manager-operator-7cbb8486b4-2jgvl_ce265787-c178-4a63-8ddb-323537d12cf9 became leader
10:40:11 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-5 -n openshift-kube-controller-manager because it was missing
10:40:12 | openshift-kube-scheduler | multus | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.17/23]
10:40:12 | openshift-etcd | multus | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.8/23]
10:40:12 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-5 -n openshift-kube-controller-manager because it was missing
10:40:12 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:39:53 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:12 | openshift-kube-apiserver-operator | kube-apiserver-operator-target-config-controller-targetconfigcontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-server-ca -n openshift-kube-apiserver because it was missing
10:40:12 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 4 because node master-1.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:12 (x5) | openshift-controller-manager | kubelet | controller-manager-5qdng | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:40:12 (x5) | openshift-controller-manager | kubelet | controller-manager-c9z4d | FailedMount | MountVolume.SetUp failed for volume "client-ca" : configmap "client-ca" not found
10:40:13 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/bound-sa-token-signing-certs-2 -n openshift-kube-apiserver because it was missing
10:40:13 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:40:13 | openshift-etcd | kubelet | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:40:13 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "Progressing: daemonset/controller-manager: observed generation is 7, desired generation is 8.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1" to "",Available message changed from "Available: no daemon pods available on any node." to ""
10:40:13 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | PodCreated | Created Pod/installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:40:13 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "" to "Progressing: daemonset/controller-manager: observed generation is 7, desired generation is 8.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1",Available message changed from "" to "Available: no daemon pods available on any node."
10:40:13 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapCreated | Created ConfigMap/client-ca -n openshift-controller-manager because it was missing
10:40:13 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-5 -n openshift-kube-controller-manager because it was missing
10:40:14 | openshift-kube-controller-manager | multus | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.18/23]
10:40:14 | openshift-kube-scheduler | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:40:14 | openshift-kube-scheduler | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:40:14 | openshift-etcd | kubelet | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:40:14 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:40:14 | openshift-etcd | kubelet | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:40:14 | openshift-kube-scheduler | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:40:15 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-controller-manager: cause by changes in data.openshift-controller-manager.client-ca.configmap
10:40:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca-2 -n openshift-kube-apiserver because it was missing
10:40:15 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-5 -n openshift-kube-controller-manager because it was missing
10:40:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-server-ca -n openshift-config-managed because it was missing
10:40:16 | openshift-etcd | multus | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.19/23]
10:40:16 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: client-ca, configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
10:40:16 | openshift-kube-scheduler | multus | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.21/23]
10:40:16 | openshift-kube-apiserver-operator | kube-apiserver-operator-node-kubeconfig-controller-nodekubeconfigcontroller | kube-apiserver-operator | SecretCreated | Created Secret/node-kubeconfigs -n openshift-kube-apiserver because it was missing
10:40:16 (x2) | openshift-insights | kubelet | insights-operator-59575c9c9d-8bw8w | Created | Created container insights-operator
10:40:16 (x2) | openshift-insights | kubelet | insights-operator-59575c9c9d-8bw8w | Started | Started container insights-operator
10:40:16 | openshift-insights | kubelet | insights-operator-59575c9c9d-8bw8w | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a380de2617755685ef62a43faefbe0041734ca0b213143bc45d29bfd0aec7a24" already present on machine
10:40:16 (x4) | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
10:40:16 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-5 -n openshift-kube-controller-manager because it was missing
10:40:16 | openshift-kube-controller-manager | kubelet | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:17 | openshift-multus | kubelet | network-metrics-daemon-djqpl | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1"
10:40:17 (x42) | openshift-apiserver-operator | openshift-apiserver-operator-workload-controller | openshift-apiserver-operator | PrereqNotReady | clusteroperator/kube-apiserver is not Available
10:40:17 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeKubeconfigControllerDegraded: \"secret/node-kubeconfigs\": configmap \"kube-apiserver-server-ca\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
10:40:17 | openshift-etcd | kubelet | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:40:17 | openshift-multus | multus | network-metrics-daemon-djqpl | AddedInterface | Add eth0 [10.128.0.22/23]
10:40:17 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-server-ca-2 -n openshift-kube-apiserver because it was missing
10:40:17 | openshift-kube-controller-manager | kubelet | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:17 | openshift-kube-controller-manager | kubelet | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 4 created because configmap/serviceaccount-ca has changed
10:40:18 | openshift-multus | multus | network-metrics-daemon-2nbpw | AddedInterface | Add eth0 [10.130.0.20/23]
10:40:18 (x3) | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | DaemonSetUpdated | Updated DaemonSet.apps/controller-manager -n openshift-controller-manager because it changed
10:40:18 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "Progressing: daemonset/controller-manager: observed generation is 9, desired generation is 10.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\nProgressing: daemonset/controller-manager: updated number scheduled is 0, desired number scheduled is 3" to "",Available message changed from "Available: no daemon pods available on any node." to ""
10:40:18 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "" to "Progressing: daemonset/controller-manager: observed generation is 9, desired generation is 10.\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\nProgressing: daemonset/controller-manager: updated number scheduled is 0, desired number scheduled is 3",Available message changed from "" to "Available: no daemon pods available on any node."
10:40:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 6 triggered by "configmap/serviceaccount-ca has changed"
10:40:18 | openshift-kube-scheduler | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369"
10:40:18 | openshift-etcd | kubelet | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-5 -n openshift-kube-controller-manager because it was missing
10:40:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-target-config-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/serviceaccount-ca -n openshift-kube-controller-manager: cause by changes in data.ca-bundle.crt
10:40:18 | openshift-multus | kubelet | network-metrics-daemon-2nbpw | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1"
10:40:18 | openshift-etcd | kubelet | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:19 | openshift-multus | kubelet | network-metrics-daemon-djqpl | Created | Created container network-metrics-daemon
10:40:19 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kubelet-serving-ca-2 -n openshift-kube-apiserver because it was missing
10:40:19 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: node-kubeconfigs, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]"
10:40:19 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/revision-status-5 -n openshift-kube-scheduler because it was missing
10:40:19 | openshift-kube-controller-manager | static-pod-installer | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 4
10:40:19 | openshift-multus | kubelet | network-metrics-daemon-djqpl | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1" in 1.542208356s
10:40:19 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/serviceaccount-ca -n openshift-kube-scheduler: cause by changes in data.ca-bundle.crt
10:40:19 (x3) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-staticpodstatecontroller | kube-controller-manager-operator | MissingVersion | no image found for operand pod
10:40:19 | openshift-multus | kubelet | network-metrics-daemon-djqpl | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:40:19 | openshift-multus | kubelet | network-metrics-daemon-djqpl | Started | Started container network-metrics-daemon
10:40:19 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 5 triggered by "configmap/serviceaccount-ca has changed"
10:40:20 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:40:20 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-4 -n openshift-kube-scheduler: cause by changes in data.status
10:40:20 | openshift-multus | kubelet | network-metrics-daemon-2nbpw | Created | Created container network-metrics-daemon
10:40:20 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:40:20 | openshift-etcd | static-pod-installer | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 2
10:40:20 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-6 -n openshift-kube-controller-manager because it was missing
10:40:20 | openshift-multus | kubelet | network-metrics-daemon-2nbpw | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1" in 1.650748084s
10:40:20 | openshift-kube-scheduler | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" in 2.39771182s
10:40:21 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/config-5 -n openshift-kube-scheduler because it was missing
10:40:21 | openshift-kube-scheduler | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:21 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-pod-5 -n openshift-kube-scheduler because it was missing
10:40:21 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 4" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 5",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 4" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 5"
10:40:21 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-5 -n openshift-kube-controller-manager: cause by changes in data.status
10:40:21 | openshift-kube-scheduler | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:21 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:40:21 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb"
10:40:21 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7"
10:40:21 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator-lock | LeaderElection | openshift-apiserver-operator-56c79799ff-ntg2n_a78899b5-f30a-4842-998c-642bd6ce9ec8 became leader
10:40:22 | openshift-kube-scheduler | static-pod-installer | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 4
10:40:22 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" in 1.897640381s
10:40:22 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pods \"etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:22 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe"
10:40:22 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-5 -n openshift-kube-scheduler because it was missing
10:40:22 | openshift-machine-api | cluster-autoscaler-operator-557599ccf8-lchg2_52ae4903-848b-4b56-98ce-217dfa7964c6 | cluster-autoscaler-operator-leader | LeaderElection | cluster-autoscaler-operator-557599ccf8-lchg2_52ae4903-848b-4b56-98ce-217dfa7964c6 became leader
10:40:22 | openshift-controller-manager | default-scheduler | controller-manager-kzgff | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-kzgff to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:40:22 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/sa-token-signing-certs-2 -n openshift-kube-apiserver because it was missing
10:40:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:40:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:40:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:40:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled" to "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled",status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"operator" "4.6.52"} {"kube-controller-manager" "1.19.16"}]
10:40:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"kube-controller-manager" "1.19.16"} {"operator" "4.6.52"}]
10:40:23 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorVersionChanged | clusteroperator/kube-controller-manager version "operator" changed from "" to "4.6.52"
10:40:23 (x9) | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | RequiredInstallerResourcesMissing | configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1
10:40:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-6 -n openshift-kube-controller-manager because it was missing
10:40:23 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorVersionChanged | clusteroperator/kube-controller-manager version "kube-controller-manager" changed from "" to "1.19.16"
10:40:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:40:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled" to "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled"
10:40:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:24 | openshift-controller-manager | default-scheduler | controller-manager-zcv97 | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-zcv97 to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:40:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:40:24 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" in 2.478996906s
10:40:24 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-ensure-env-vars
10:40:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:40:24 | openshift-controller-manager | multus | controller-manager-kzgff | AddedInterface | Add eth0 [10.129.0.9/23]
10:40:24 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/scheduler-kubeconfig-5 -n openshift-kube-scheduler because it was missing
10:40:24 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled" to "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled"
10:40:24 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-audit-policies-2 -n openshift-kube-apiserver because it was missing
10:40:24 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-6 -n openshift-kube-controller-manager because it was missing
10:40:24 | openshift-controller-manager | default-scheduler | controller-manager-n7fv8 | Scheduled | Successfully assigned openshift-controller-manager/controller-manager-n7fv8 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:40:24 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-ensure-env-vars
10:40:25 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-4 -n openshift-kube-controller-manager: cause by changes in data.status
10:40:25 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-resources-copy
10:40:25 | openshift-controller-manager | kubelet | controller-manager-kzgff | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697"
10:40:25 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:25 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:25 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-resources-copy
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcdctl
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-metrics
10:40:26 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-cert-syncer-kubeconfig-5 -n openshift-kube-scheduler because it was missing
10:40:26 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/etcd-client-2 -n openshift-kube-apiserver because it was missing
10:40:26 | openshift-controller-manager | multus | controller-manager-zcv97 | AddedInterface | Add eth0 [10.130.0.21/23]
10:40:26 | openshift-controller-manager | kubelet | controller-manager-zcv97 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697"
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcdctl
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-metrics
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd
10:40:26 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:26 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-6 -n openshift-kube-controller-manager because it was missing
10:40:26 | openshift-controller-manager | multus | controller-manager-n7fv8 | AddedInterface | Add eth0 [10.128.0.23/23]
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:26 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:27 | openshift-controller-manager | kubelet | controller-manager-n7fv8 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697"
10:40:27 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:40:27 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | MemberAdd | adding new peer https://10.0.94.119:2380
10:40:27 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/serving-cert-5 -n openshift-kube-scheduler because it was missing
10:40:27 | openshift-controller-manager | kubelet | controller-manager-kzgff | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" in 2.059654228s
10:40:27 | openshift-etcd-operator | openshift-cluster-etcd-operator-cluster-member-controller-clustermembercontroller | etcd-operator | FoundPodToScale | found pod to add to etcd membership: etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com
10:40:27 | openshift-controller-manager | openshift-controller-manager | openshift-master-controllers | LeaderElection | controller-manager-kzgff became leader
10:40:28 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "StaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending"
10:40:28 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-6 -n openshift-kube-controller-manager because it was missing
10:40:28 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending\nTargetConfigControllerDegraded: \"configmap/trusted-ca-bundle\": Get \"https://172.30.0.1:443/api/v1/namespaces/openshift-kube-controller-manager/configmaps/trusted-ca-bundle\": context canceled" to "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending"
10:40:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/localhost-recovery-client-token-5 -n openshift-kube-scheduler because it was missing
10:40:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionCreate | Revision 4 created because configmap/serviceaccount-ca has changed
10:40:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/kubelet-client-2 -n openshift-kube-apiserver because it was missing
10:40:29 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:40:26 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:29 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 4" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 5",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 4" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 5"
10:40:29 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 5 because node master-1.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:29 | openshift-controller-manager | kubelet | controller-manager-zcv97 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" in 2.226047821s
10:40:29 | openshift-controller-manager | kubelet | controller-manager-zcv97 | Created | Created container controller-manager
10:40:29 | openshift-controller-manager | kubelet | controller-manager-zcv97 | Started | Started container controller-manager
10:40:30 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-serving-certkey-2 -n openshift-kube-apiserver because it was missing
10:40:30 | openshift-kube-controller-manager | multus | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.22/23]
10:40:31 | openshift-kube-controller-manager | kubelet | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:31 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-client-token-2 -n openshift-kube-apiserver because it was missing
10:40:31 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:31 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-6 -n openshift-kube-controller-manager because it was missing
10:40:32 | openshift-controller-manager | kubelet | controller-manager-n7fv8 | Created | Created container controller-manager
10:40:32 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:40:32 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-4 -n openshift-kube-scheduler: cause by changes in data.status
10:40:32 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing changed from True to False (""),Available changed from False to True (""),status.versions changed from [] to [{"operator" "4.6.52"}]
10:40:32 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because static pod is ready
10:40:32 | openshift-controller-manager | kubelet | controller-manager-n7fv8 | Started | Started container controller-manager
10:40:32 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2" to "NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 2",Available message changed from "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2\nEtcdMembersAvailable: 1 members are available" to "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2\nEtcdMembersAvailable: 1 members are available"
10:40:32 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorVersionChanged | clusteroperator/openshift-controller-manager version "operator" changed from "" to "4.6.52"
10:40:32 | openshift-kube-controller-manager | kubelet | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:40:32 | openshift-controller-manager | kubelet | controller-manager-n7fv8 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" in 5.065832712s
10:40:32 | openshift-kube-controller-manager | kubelet | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:40:32 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" in 9.391599067s
10:40:32 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:40:32 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:40:32 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionCreate | Revision 1 created because configmap "kube-apiserver-pod-1" not found
10:40:33 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending"
10:40:33 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-1.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:33 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-6 -n openshift-kube-controller-manager because it was missing
10:40:34 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:40:26 +0000 UTC but not ready: unknown reason\nStaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:34 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:40:34 | openshift-kube-controller-manager | multus | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.24/23]
10:40:34 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:40:34 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:40:34 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:40:34 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-6 -n openshift-kube-controller-manager because it was missing
10:40:34 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nInstallerControllerDegraded: missing required resources: [configmaps: bound-sa-token-signing-certs-1,config-1,etcd-serving-ca-1,kube-apiserver-audit-policies-1,kube-apiserver-cert-syncer-kubeconfig-1,kube-apiserver-pod-1,kubelet-serving-ca-1,sa-token-signing-certs-1, secrets: etcd-client-1,kubelet-client-1,localhost-recovery-client-token-1,localhost-recovery-serving-certkey-1]" to "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:34 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:40:34 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 1" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 2"
10:40:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
10:40:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
10:40:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:40:35 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:40:35 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
10:40:35 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
10:40:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:40:36 | openshift-etcd-operator | openshift-cluster-etcd-operator-prunecontroller | etcd-operator | PodCreated | Created Pod/revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:40:36 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 5 because node master-2.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:36 | openshift-kube-scheduler | multus | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.25/23]
10:40:36 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-6 -n openshift-kube-controller-manager because it was missing
10:40:36 | openshift-kube-controller-manager | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a"
10:40:37 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-6 -n openshift-kube-controller-manager because it was missing
10:40:38 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:40:38 | openshift-kube-scheduler | kubelet | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:40:38 | openshift-kube-controller-manager | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:38 | openshift-kube-controller-manager | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" in 1.938586427s
10:40:38 | openshift-etcd | multus | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.23/23]
10:40:38 | openshift-kube-scheduler | kubelet | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:40:38 | openshift-kube-scheduler | kubelet | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:40:38 | openshift-kube-controller-manager | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:39 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-6 -n openshift-kube-controller-manager because it was missing
10:40:39 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-1.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:39 | openshift-etcd | kubelet | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:40:40 | openshift-etcd | kubelet | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:40:40 | openshift-etcd | kubelet | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:40:40 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-6 -n openshift-kube-controller-manager because it was missing
10:40:40 | openshift-kube-apiserver | multus | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.26/23]
10:40:40 | openshift-kube-controller-manager | static-pod-installer | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 5
10:40:40 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 5 created because configmap/serviceaccount-ca has changed
10:40:41 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:40:41 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:40:41 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | PodCreated | Created Pod/installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:40:42 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7"
10:40:42 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:40:42 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:40:43 | openshift-kube-apiserver | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:40:43 | openshift-kube-apiserver | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:43 | openshift-kube-apiserver | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:43 | openshift-etcd | multus | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.27/23]
10:40:43 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" in 1.739756071s
10:40:43 | openshift-kube-scheduler | multus | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.10/23]
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:40:44 | openshift-etcd | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b"
10:40:44 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 5" to "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 6",Available message changed from "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 5" to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 6"
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:40:44 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:45 | openshift-kube-scheduler | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369"
10:40:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "StaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending"
10:40:46 | openshift-etcd | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" in 1.910547789s
10:40:46 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "StaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending"
10:40:47 | openshift-etcd | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:47 | openshift-etcd | kubelet | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:47 | openshift-kube-scheduler | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" in 2.175408519s
10:40:48 | openshift-kube-scheduler | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:40:48 | openshift-kube-scheduler | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:40:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-5 -n openshift-kube-controller-manager: cause by changes in data.status
10:40:49 | openshift-kube-scheduler | static-pod-installer | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 5
10:40:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:49 | openshift-etcd | static-pod-installer | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 2
10:40:49 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending\nStaticPodsDegraded: pods \"openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending\nNodeControllerDegraded: All master nodes are ready"
10:40:50 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
10:40:50 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe"
10:40:50 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb"
10:40:50 (x7) | openshift-kube-apiserver-operator | kube-apiserver-operator-staticpodstatecontroller | kube-apiserver-operator | MissingVersion | no image found for operand pod
10:40:50 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
10:40:50 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "StaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending"
10:40:51 | openshift-kube-apiserver | static-pod-installer | installer-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 2
10:40:51 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:40:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded changed from False to True ("OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server")
10:40:51 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:40:52 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:40:52 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:40:52 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because node master-2.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:40:52 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded changed from False to True ("StaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending")
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:40:53 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-ensure-env-vars
10:40:53 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" in 2.709469225s
10:40:53 | openshift-kube-controller-manager | multus | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.28/23]
10:40:53 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-ensure-env-vars
10:40:53 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:40:54 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:54 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-resources-copy
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:40:54 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-resources-copy
10:40:54 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:40:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:40:55 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:55 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:55 | openshift-kube-controller-manager | kubelet | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:40:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:40:55 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcdctl
10:40:55 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcdctl
10:40:55 | openshift-kube-controller-manager | kubelet | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:40:55 | openshift-kube-controller-manager | kubelet | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:40:56 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready"
10:40:56 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready",status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"operator" "4.6.52"} {"kube-apiserver" "1.19.16"}]
10:40:56 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: status.versions changed from [{"raw-internal" "4.6.52"}] to [{"raw-internal" "4.6.52"} {"kube-apiserver" "1.19.16"} {"operator" "4.6.52"}]
10:40:56 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorVersionChanged | clusteroperator/kube-apiserver version "operator" changed from "" to "4.6.52"
10:40:56 | openshift-etcd-operator | openshift-cluster-etcd-operator-cluster-member-controller-clustermembercontroller | etcd-operator | FoundPodToScale | found pod to add to etcd membership: etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com
10:40:56 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:40:56 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorVersionChanged | clusteroperator/kube-apiserver version "kube-apiserver" changed from "" to "1.19.16"
10:40:56 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | MemberAdd | adding new peer https://10.0.95.76:2380
10:40:56 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd
10:40:56 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd
10:40:56 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-metrics
10:40:56 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-metrics
10:40:57 | openshift-kube-apiserver | cert-regeneration-controller | cert-regeneration-controller-lock | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_5a50137f-d94a-491b-8c5c-a1b4338e4653 became leader
10:40:58 | openshift-controller-manager | kubelet | controller-manager-mzf7c | FailedMount | Unable to attach or mount volumes: unmounted volumes=[openshift-controller-manager-sa-token-4745r config client-ca serving-cert proxy-ca-bundles], unattached volumes=[openshift-controller-manager-sa-token-4745r config client-ca serving-cert proxy-ca-bundles]: timed out waiting for the condition
10:40:58 | openshift-controller-manager | kubelet | controller-manager-fg5sh | FailedMount | Unable to attach or mount volumes: unmounted volumes=[client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r config], unattached volumes=[client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r config]: timed out waiting for the condition
10:40:58 | openshift-controller-manager | kubelet | controller-manager-9tld6 | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:40:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:40:56 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:41:00 | openshift-kube-controller-manager | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com_openshift-kube-controller-manager_6fab9328-8bac-4bbd-939c-082ee411023b_0(6e7716c94bc9f1958423f9beb575474ae43406a7faf2a470b98ffd1dfa90828f): Multus: [openshift-kube-controller-manager/installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com]: error getting pod: pods "installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com" is forbidden: User "system:serviceaccount:openshift-multus:multus" cannot get resource "pods" in API group "" in the namespace "openshift-kube-controller-manager"
10:41:00 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:41:00 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" in 10.057822703s
10:41:00 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:41:02 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2\nEtcdMembersAvailable: 1 members are available" to "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available"
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:41:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:41:03 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
10:41:03 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
10:41:03 | openshift-kube-scheduler | default-scheduler | kube-scheduler | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_66b34ec5-01a6-4107-bfec-775cb80fc6b4 became leader
10:41:05 | openshift-controller-manager | kubelet | controller-manager-9ktwp | FailedMount | Unable to attach or mount volumes: unmounted volumes=[proxy-ca-bundles openshift-controller-manager-sa-token-4745r config client-ca serving-cert], unattached volumes=[proxy-ca-bundles openshift-controller-manager-sa-token-4745r config client-ca serving-cert]: timed out waiting for the condition
10:41:05 (x29) | openshift-authentication-operator | oauth-apiserver-workload-controller | authentication-operator | PrereqNotReady | clusteroperator/kube-apiserver is not Available
10:41:05 (x17) | openshift-apiserver-operator | openshift-apiserver-operator-workload-controller | openshift-apiserver-operator | PrereqNotReady | clusteroperator/kube-apiserver is not Available
10:41:05 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
10:41:05 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
10:41:06 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because static pod is ready
10:41:06 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 2\nEtcdMembersProgressing: No unstarted etcd members found"),Available message changed from "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available"
10:41:06 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:40:56 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:41:07 | openshift-controller-manager | kubelet | controller-manager-t54r4 | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:07 | openshift-service-ca-operator | service-ca-operator | service-ca-operator-lock | LeaderElection | service-ca-operator-6fd6454f86-5bbh8_c2f21365-b6eb-4a7a-9ead-bdd839ffd2d8 became leader
10:41:07 | openshift-controller-manager | kubelet | controller-manager-ktjz4 | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:07 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-2 -n openshift-etcd: cause by changes in data.status
10:41:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from False to True ("StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found")
10:41:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because static pod is ready
10:41:11 | openshift-etcd-operator | openshift-cluster-etcd-operator-prunecontroller | etcd-operator | PodCreated | Created Pod/revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:41:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 2" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2",Available changed from False to True ("StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2")
10:41:13 | openshift-etcd | multus | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.29/23]
10:41:14 | openshift-kube-controller-manager | multus | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.11/23]
10:41:15 | openshift-etcd | kubelet | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:41:15 | openshift-etcd | kubelet | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:41:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-2.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:41:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:41:15 | openshift-etcd | kubelet | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:41:15 | openshift-controller-manager | kubelet | controller-manager-7lhx9 | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:16 | openshift-kube-controller-manager | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a"
10:41:17 | openshift-oauth-apiserver | default-scheduler | apiserver-b7585d69b-7p56k | Scheduled | Successfully assigned openshift-oauth-apiserver/apiserver-b7585d69b-7p56k to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:41:17 | openshift-kube-apiserver | multus | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.30/23]
10:41:17 | openshift-oauth-apiserver | replicaset-controller | apiserver-b7585d69b | SuccessfulCreate | Created pod: apiserver-b7585d69b-f6jdj
10:41:17 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/image-import-ca -n openshift-apiserver because it was missing
10:41:17 | openshift-oauth-apiserver | replicaset-controller | apiserver-b7585d69b | SuccessfulCreate | Created pod: apiserver-b7585d69b-l9525
10:41:17 | openshift-oauth-apiserver | default-scheduler | apiserver-b7585d69b-l9525 | Scheduled | Successfully assigned openshift-oauth-apiserver/apiserver-b7585d69b-l9525 to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:17 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (no pods found with labels \"apiserver=true,app=openshift-oauth-apiserver,oauth-apiserver-anti-affinity=true,revision=2\")",Progressing changed from Unknown to True ("APIServerDeploymentProgressing: deployment/apiserver.openshift-oauth-apiserver: observed generation is 0, desired generation is 1."),Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nAPIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node."
10:41:17 | openshift-oauth-apiserver | replicaset-controller | apiserver-b7585d69b | SuccessfulCreate | Created pod: apiserver-b7585d69b-7p56k
10:41:17 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | ConfigMapCreated | Created ConfigMap/config -n openshift-apiserver because it was missing
10:41:17 | openshift-authentication-operator | oauth-apiserver | authentication-operator | DeploymentCreated | Created Deployment.apps/apiserver -n openshift-oauth-apiserver because it was missing
10:41:17 | openshift-oauth-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled up replica set apiserver-b7585d69b to 3
10:41:17 | openshift-oauth-apiserver | default-scheduler | apiserver-b7585d69b-f6jdj | Scheduled | Successfully assigned openshift-oauth-apiserver/apiserver-b7585d69b-f6jdj to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:41:18 | openshift-kube-controller-manager | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:41:18 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:18 | openshift-kube-controller-manager | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" in 2.355797512s
10:41:19 | openshift-apiserver | default-scheduler | apiserver-686bc75759-v9g2f | Scheduled | Successfully assigned openshift-apiserver/apiserver-686bc75759-v9g2f to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:41:19 | openshift-kube-controller-manager | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:41:19 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:41:19 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:41:19 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled up replica set apiserver-686bc75759 to 3
10:41:19 | openshift-apiserver | replicaset-controller | apiserver-686bc75759 | SuccessfulCreate | Created pod: apiserver-686bc75759-77rhg
10:41:19 | openshift-apiserver | replicaset-controller | apiserver-686bc75759 | SuccessfulCreate | Created pod: apiserver-686bc75759-tbpkl
10:41:19 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Progressing changed from False to True ("OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2.")
10:41:19 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "" to "APIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-apiserver (3 containers are waiting in pending apiserver-686bc75759-v9g2f pod, 3 containers are waiting in pending apiserver-686bc75759-77rhg pod)",Progressing message changed from "OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2." to "OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2.\nAPIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 0, desired generation is 1.",Available message changed from "APIServicesAvailable: PreconditionNotReady" to "APIServicesAvailable: PreconditionNotReady\nAPIServerDeploymentAvailable: no apiserver.openshift-apiserver pods available on any node."
10:41:19 | openshift-apiserver | default-scheduler | apiserver-686bc75759-77rhg | Scheduled | Successfully assigned openshift-apiserver/apiserver-686bc75759-77rhg to master-0.case03116926.lab.upshift.rdu2.redhat.com
10:41:19 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | DeploymentCreated | Created Deployment.apps/apiserver -n openshift-apiserver because it was missing
10:41:19 | openshift-apiserver | replicaset-controller | apiserver-686bc75759 | SuccessfulCreate | Created pod: apiserver-686bc75759-v9g2f
10:41:19 | openshift-apiserver | default-scheduler | apiserver-686bc75759-tbpkl | Scheduled | Successfully assigned openshift-apiserver/apiserver-686bc75759-tbpkl to master-1.case03116926.lab.upshift.rdu2.redhat.com
10:41:20 | openshift-oauth-apiserver | multus | apiserver-b7585d69b-l9525 | AddedInterface | Add eth0 [10.128.0.31/23]
10:41:20 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:41:20 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633"
10:41:20 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633"
10:41:20 | openshift-oauth-apiserver | multus | apiserver-b7585d69b-f6jdj | AddedInterface | Add eth0 [10.130.0.24/23]
10:41:20 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (no pods found with labels \"apiserver=true,app=openshift-oauth-apiserver,oauth-apiserver-anti-affinity=true,revision=2\")" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (2 containers are waiting in pending apiserver-b7585d69b-7p56k pod, 2 containers are waiting in pending apiserver-b7585d69b-f6jdj pod, 2 containers are waiting in pending apiserver-b7585d69b-l9525 pod)",Progressing changed from True to False ("")
10:41:20 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633"
10:41:20 | openshift-oauth-apiserver | multus | apiserver-b7585d69b-7p56k | AddedInterface | Add eth0 [10.129.0.12/23]
10:41:21 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-lock | LeaderElection | kube-storage-version-migrator-operator-5999bc745c-nqrdg_537247bf-9731-4e06-adc3-75e82ecbb0c0 became leader
10:41:21 | openshift-apiserver | multus | apiserver-686bc75759-v9g2f | AddedInterface | Add eth0 [10.129.0.13/23]
10:41:21 | openshift-apiserver | multus | apiserver-686bc75759-tbpkl | AddedInterface | Add eth0 [10.128.0.32/23]
10:41:21 | openshift-kube-controller-manager | static-pod-installer | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 6
10:41:22 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Created | Created container fix-audit-permissions
10:41:22 | openshift-apiserver | multus | apiserver-686bc75759-77rhg | AddedInterface | Add eth0 [10.130.0.25/23]
10:41:22 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (2 containers are waiting in pending apiserver-b7585d69b-7p56k pod, 2 containers are waiting in pending apiserver-b7585d69b-f6jdj pod, 2 containers are waiting in pending apiserver-b7585d69b-l9525 pod)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (2 containers are waiting in pending apiserver-b7585d69b-7p56k pod, 2 containers are waiting in pending apiserver-b7585d69b-f6jdj pod, 2 containers are waiting in pending apiserver-b7585d69b-l9525 pod)"
10:41:22 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nAPIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node." to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nAPIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node."
10:41:22 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333"
10:41:22 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633" in 2.234780983s
10:41:22 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333"
10:41:22 | openshift-kube-apiserver | multus | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.14/23]
10:41:22 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333"
10:41:22 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Started | Started container fix-audit-permissions
10:41:22 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633" in 2.085352885s
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Created | Created container fix-audit-permissions
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633" already present on machine
10:41:23 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (2 containers are waiting in pending apiserver-b7585d69b-7p56k pod, 2 containers are waiting in pending apiserver-b7585d69b-f6jdj pod, 2 containers are waiting in pending apiserver-b7585d69b-l9525 pod)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (2 containers are waiting in pending apiserver-b7585d69b-7p56k pod, container is waiting in pending apiserver-b7585d69b-f6jdj pod, container is waiting in pending apiserver-b7585d69b-l9525 pod)"
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Started | Started container oauth-apiserver
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Created | Created container fix-audit-permissions
10:41:23 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "StaticPodsDegraded: pod/kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"cluster-policy-controller\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-cert-syncer\" is waiting: ContainerCreating: \nStaticPodsDegraded: pod/kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is waiting: ContainerCreating: \nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending"
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Started | Started container fix-audit-permissions
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633" already present on machine
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633" in 3.014904661s
10:41:23 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-f6jdj | Created | Created container oauth-apiserver
10:41:24 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Started | Started container oauth-apiserver
10:41:24 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19eac3f0c1f7b016f50511bbc48f807966a9690da7e461bfd7443fe7875de633" already present on machine
10:41:24 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Created | Created container oauth-apiserver
10:41:24 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Started | Started container oauth-apiserver
10:41:24 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Created | Created container oauth-apiserver
10:41:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7"
10:41:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:41:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:41:24 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" in 2.476075195s
10:41:24 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Started | Started container fix-audit-permissions
10:41:24 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:41:25 | openshift-kube-apiserver | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5"
10:41:25 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Started | Started container fix-audit-permissions
10:41:25 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
10:41:25 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Created | Created container fix-audit-permissions
10:41:25 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Created | Created container fix-audit-permissions
10:41:25 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Started | Started container fix-audit-permissions
10:41:25 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" in 2.459027562s
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5"
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Created | Created container fix-audit-permissions
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Started | Started container openshift-apiserver
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" in 4.030243635s
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Started | Started container fix-audit-permissions
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Created | Created container openshift-apiserver
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Started | Started container openshift-apiserver
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Started | Started container openshift-apiserver-check-endpoints
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Created | Created container openshift-apiserver
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Created | Created container openshift-apiserver-check-endpoints
10:41:26 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
10:41:27 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nAPIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node." to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nAPIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node."
10:41:27 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is not ready in apiserver-b7585d69b-7p56k pod, container is not ready in apiserver-b7585d69b-f6jdj pod, container is not ready in apiserver-b7585d69b-l9525 pod)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is not ready in apiserver-b7585d69b-7p56k pod, container is not ready in apiserver-b7585d69b-f6jdj pod, container is not ready in apiserver-b7585d69b-l9525 pod)"
10:41:27 | openshift-controller-manager | kubelet | controller-manager-q5w7b | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:27 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (2 containers are waiting in pending apiserver-b7585d69b-7p56k pod, container is waiting in pending apiserver-b7585d69b-f6jdj pod, container is waiting in pending apiserver-b7585d69b-l9525 pod)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is not ready in apiserver-b7585d69b-7p56k pod, container is not ready in apiserver-b7585d69b-f6jdj pod, container is not ready in apiserver-b7585d69b-l9525 pod)"
10:41:27 | openshift-controller-manager | kubelet | controller-manager-pxstf | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:28 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
10:41:28 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" in 2.308880677s
10:41:29 | openshift-kube-apiserver | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" in 3.389708687s
10:41:29 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Created | Created container openshift-apiserver
10:41:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" in 4.596902199s
10:41:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:41:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:41:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:41:29 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Started | Started container openshift-apiserver
10:41:29 | openshift-kube-apiserver | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:41:29 | openshift-kube-apiserver | kubelet | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:41:29 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Created | Created container openshift-apiserver-check-endpoints
10:41:29 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:29 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Started | Started container openshift-apiserver-check-endpoints
10:41:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:41:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:41:30 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Created | Created container openshift-apiserver-check-endpoints
10:41:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:41:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:41:30 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is not ready in apiserver-b7585d69b-7p56k pod, container is not ready in apiserver-b7585d69b-f6jdj pod, container is not ready in apiserver-b7585d69b-l9525 pod)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is not ready in apiserver-b7585d69b-f6jdj pod)",Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nAPIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node." to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
10:41:30 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Started | Started container openshift-apiserver-check-endpoints
10:41:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:41:31 | openshift-cluster-samples-operator | replicaset-controller | cluster-samples-operator-6c4b6b887b | SuccessfulCreate | Created pod: cluster-samples-operator-6c4b6b887b-cq6nw
10:41:31 | openshift-cluster-samples-operator | deployment-controller | cluster-samples-operator | ScalingReplicaSet | Scaled up replica set cluster-samples-operator-6c4b6b887b to 1
10:41:31 | openshift-cluster-samples-operator | default-scheduler | cluster-samples-operator-6c4b6b887b-cq6nw | Scheduled | Successfully assigned openshift-cluster-samples-operator/cluster-samples-operator-6c4b6b887b-cq6nw to master-2.case03116926.lab.upshift.rdu2.redhat.com
10:41:32 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | FailedMount | MountVolume.SetUp failed for volume "samples-operator-tls" : secret "samples-operator-tls" not found
10:41:32 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Progressing message changed from "OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 0, desired generation is 2.\nAPIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 0, desired generation is 1." to "APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 0, desired generation is 1."
10:41:32 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-apiserver (3 containers are waiting in pending apiserver-686bc75759-v9g2f pod, 3 containers are waiting in pending apiserver-686bc75759-77rhg pod)" to "APIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-apiserver (container is not ready in apiserver-686bc75759-v9g2f pod, container is not ready in apiserver-686bc75759-77rhg pod, container is not ready in apiserver-686bc75759-tbpkl pod)",Progressing changed from True to False ("")
10:41:34 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: status.versions changed from [] to [{"oauth-apiserver" "4.6.52"}]
10:41:34 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending")
10:41:34 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.image.openshift.io because it was missing
10:41:34 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.build.openshift.io because it was missing
10:41:34 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.authorization.openshift.io because it was missing
10:41:34 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nAPIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is not ready in apiserver-b7585d69b-f6jdj pod)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server"
10:41:34 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.oauth.openshift.io because it was missing
10:41:34 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.apps.openshift.io because it was missing
10:41:34 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.project.openshift.io because it was missing
10:41:34 | openshift-cluster-samples-operator | multus | cluster-samples-operator-6c4b6b887b-cq6nw | AddedInterface | Add eth0 [10.129.0.15/23]
10:41:34 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorVersionChanged | clusteroperator/authentication version "oauth-apiserver" changed from "" to "4.6.52"
10:41:35 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.quota.openshift.io because it was missing
10:41:35 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.route.openshift.io because it was missing
10:41:35 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9a4552e9fdd1b6f20527a63ba16eed6720901fa51d79df79dd2a0436f7c812b"
10:41:35 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.security.openshift.io because it was missing
10:41:36 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.template.openshift.io because it was missing
10:41:36 | openshift-authentication-operator | oauth-apiserver-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | authentication-operator | Updated | Updated <unknown>/v1.oauth.openshift.io because it changed
10:41:36 | openshift-authentication-operator | oauth-apiserver-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | authentication-operator | Updated | Updated <unknown>/v1.user.openshift.io because it changed
10:41:36 | openshift-authentication-operator | oauth-apiserver | authentication-operator | APIServicesToManageChanged | The new API Services list this operator will manage is [v1.oauth.openshift.io v1.user.openshift.io]
10:41:36 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | Created | Created <unknown>/v1.user.openshift.io because it was missing
10:41:37 | openshift-controller-manager | kubelet | controller-manager-mpm5t | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:37 | openshift-kube-apiserver | static-pod-installer | installer-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 2
10:41:38 (x19) | openshift-monitoring | default-scheduler | kube-state-metrics-6c8b757584-2b2dg | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:38 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Created | Created container cluster-samples-operator
10:41:38 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9a4552e9fdd1b6f20527a63ba16eed6720901fa51d79df79dd2a0436f7c812b" in 3.033926093s
10:41:38 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9a4552e9fdd1b6f20527a63ba16eed6720901fa51d79df79dd2a0436f7c812b" already present on machine
10:41:38 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Started | Started container cluster-samples-operator
10:41:39 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Started | Started container cluster-samples-operator-watch
10:41:39 | openshift-cluster-samples-operator | kubelet | cluster-samples-operator-6c4b6b887b-cq6nw | Created | Created container cluster-samples-operator-watch
10:41:39 | openshift-cluster-samples-operator | file-change-watchdog | cluster-samples-operator | FileChangeWatchdogStarted | Started watching files for process cluster-samples-operator[13]
10:41:39 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:41:39 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:41:39 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:41:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:41:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:41:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:41:41 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | OpenShiftAPICheckFailed | "apps.openshift.io.v1" failed with HTTP status code 503 (the server is currently unable to handle the request)
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:41:41 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nAPIServicesAvailable: \"oauth.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nAPIServicesAvailable: \"user.openshift.io.v1\" is not ready: 404 (the server could not find the requested resource)\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:41 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:41:42 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:41:42 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:41:44 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com\" not found\nStaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found" to "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found"
10:41:45 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: status.versions changed from [{"operator" "4.6.52"}] to [{"operator" "4.6.52"} {"openshift-apiserver" "4.6.52"}]
10:41:45 | openshift-controller-manager | kubelet | controller-manager-nv4gt | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:41:45 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorVersionChanged | clusteroperator/openshift-apiserver version "openshift-apiserver" changed from "" to "4.6.52"
10:41:45 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 3 of 3 requested instances are unavailable for apiserver.openshift-apiserver (container is not ready in apiserver-686bc75759-v9g2f pod, container is not ready in apiserver-686bc75759-77rhg pod, container is not ready in apiserver-686bc75759-tbpkl pod)" to "",Available message changed from "APIServicesAvailable: PreconditionNotReady\nAPIServerDeploymentAvailable: no apiserver.openshift-apiserver pods available on any node." to "APIServicesAvailable: PreconditionNotReady"
10:41:46 (x2) | openshift-authentication-operator | oauth-apiserver-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | authentication-operator | OpenShiftAPICheckFailed | "user.openshift.io.v1" failed with HTTP status code 404 (the server could not find the requested resource)
10:41:46 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | OpenShiftAPICheckFailed | "authorization.openshift.io.v1" failed with HTTP status code 503 (the server is currently unable to handle the request)
10:41:47 | openshift-controller-manager | kubelet | controller-manager-wf9nk | FailedMount | Unable to attach or mount volumes: unmounted volumes=[openshift-controller-manager-sa-token-4745r config client-ca serving-cert proxy-ca-bundles], unattached volumes=[openshift-controller-manager-sa-token-4745r config client-ca serving-cert proxy-ca-bundles]: timed out waiting for the condition
10:41:48 (x14) | openshift-marketplace | default-scheduler | redhat-operators-hr4dq | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:48 (x19) | openshift-monitoring | default-scheduler | telemeter-client-57c59fc45f-gsqrj | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:48 (x14) | openshift-marketplace | default-scheduler | redhat-marketplace-xm8bs | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:48 (x14) | openshift-marketplace | default-scheduler | certified-operators-p45rl | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:48 (x20) | openshift-kube-storage-version-migrator | default-scheduler | migrator-85c499d985-hznj9 | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:48 (x14) | openshift-ingress | default-scheduler | router-default-5b76b97bbb-ltvlc | FailedScheduling | 0/3 nodes are available: 3 node(s) didn't match node selector.
10:41:48 (x14) | openshift-ingress | default-scheduler | router-default-5b76b97bbb-jp2g9 | FailedScheduling | 0/3 nodes are available: 3 node(s) didn't match node selector.
10:41:48 (x19) | openshift-monitoring | default-scheduler | openshift-state-metrics-6d99649c49-4b6cb | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:48 (x14) | openshift-marketplace | default-scheduler | community-operators-mprwd | FailedScheduling | 0/3 nodes are available: 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
10:41:49 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because static pod is ready
10:41:49 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 6" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 6",Available changed from False to True ("StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 6")
10:41:51 (x3) | openshift-authentication-operator | oauth-apiserver-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | authentication-operator | OpenShiftAPICheckFailed | "oauth.openshift.io.v1" failed with HTTP status code 503 (the server is currently unable to handle the request)
10:41:51 (x12) | openshift-authentication-operator | oauth-apiserver-manage-oauth-api-controller-mangeoauthapicontroller | authentication-operator | PrereqNotReady | OAuth API servers are not ready. APIService check reporeted "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) "user.openshift.io.v1" is not ready: 404 (the server could not find the requested resource)
10:41:51 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | OpenShiftAPICheckFailed | "build.openshift.io.v1" failed with HTTP status code 503 (the server is currently unable to handle the request)
10:41:52 (x28) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SATokenSignerControllerStuck | unexpected addresses: 10.0.89.186
10:41:52 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 5 because static pod is ready
10:41:52 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 0; 0 nodes have achieved new revision 5" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 5",Available changed from False to True ("StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 5")
10:41:54 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:41:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending"
10:41:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 0 is the oldest not matching its expected revision 0
10:41:56 | openshift-authentication-operator | oauth-apiserver-manage-oauth-api-controller-mangeoauthapicontroller | authentication-operator | PrereqNotReady | OAuth API servers are not ready. APIService check reporeted "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) "user.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
10:41:56 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nAPIServicesAvailable: \"oauth.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nAPIServicesAvailable: \"user.openshift.io.v1\" is not ready: 404 (the server could not find the requested resource)\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nAPIServicesAvailable: \"oauth.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nAPIServicesAvailable: \"user.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
10:41:56 | openshift-apiserver-operator | openshift-apiserver-operator-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | openshift-apiserver-operator | OpenShiftAPICheckFailed | "image.openshift.io.v1" failed with HTTP status code 503 (the server is currently unable to handle the request)
10:41:56 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:41:56 | openshift-authentication-operator | oauth-apiserver-apiservice-openshift-apiserver-controller-apiservicecontroller_openshift-apiserver | authentication-operator | OpenShiftAPICheckFailed | "user.openshift.io.v1" failed with HTTP status code 503 (the server is currently unable to handle the request)
10:41:56 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 3 triggered by "secret/localhost-recovery-client-token has changed"
10:41:56 | openshift-kube-controller-manager | multus | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.16/23]
10:41:57 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SATokenSignerControllerOK | found expected kube-apiserver endpoints
10:41:57 | openshift-cluster-version | openshift-cluster-version | version | RetrievePayload | retrieving payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:41:57 (x27) | openshift-etcd-operator | openshift-cluster-etcd-operator-bootstrap-teardown-controller-bootstrapteardowncontroller | etcd-operator | DelayingBootstrapTeardown | cluster-bootstrap is not yet finished
10:41:57 | openshift-cluster-version | openshift-cluster-version | version | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_fddba440-4ece-4ea4-b210-f551cd82e7b6 became leader
10:41:57 | openshift-cluster-version | openshift-cluster-version | version | VerifyPayload | verifying payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:41:58 | openshift-kube-controller-manager | kubelet | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:41:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-bootstrap-teardown-controller-bootstrapteardowncontroller | etcd-operator | RemoveBootstrapEtcd | removing etcd-bootstrap member
10:41:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | MemberRemove | removing member "etcd-bootstrap"
10:41:59 | openshift-kube-controller-manager | kubelet | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:41:59 | openshift-kube-controller-manager | kubelet | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:42:00 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-7p56k | Unhealthy | Readiness probe failed: Get "https://10.129.0.12:8443/readyz": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
10:42:02 | openshift-oauth-apiserver | kubelet | apiserver-b7585d69b-l9525 | Unhealthy | Liveness probe failed: Get "https://10.128.0.31:8443/healthz": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
10:42:05 | openshift-controller-manager | kubelet | controller-manager-vc55x | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:42:06 | openshift-kube-controller-manager | cluster-policy-controller | cluster-policy-controller | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com became leader
10:42:07 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com_openshift-kube-scheduler_f7fa578f-d496-44b7-8e33-b9dd54d0ddbc_0(9d378357eda57867c6faf51217831458b2bde9441fd03809bd071bbaa179a55b): Multus: [openshift-kube-scheduler/revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com]: error setting the networks status: SetNetworkStatus: failed to query the pod revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com in out of cluster comm: etcdserver: request timed out
10:42:08 | openshift-controller-manager | kubelet | controller-manager-5qdng | FailedMount | Unable to attach or mount volumes: unmounted volumes=[openshift-controller-manager-sa-token-4745r config client-ca serving-cert proxy-ca-bundles], unattached volumes=[openshift-controller-manager-sa-token-4745r config client-ca serving-cert proxy-ca-bundles]: timed out waiting for the condition
10:42:08 | openshift-controller-manager | kubelet | controller-manager-c9z4d | FailedMount | Unable to attach or mount volumes: unmounted volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r], unattached volumes=[config client-ca serving-cert proxy-ca-bundles openshift-controller-manager-sa-token-4745r]: timed out waiting for the condition
10:42:10 | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-7cb4cb644d-wx58q | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4a090f3ea32ac6d7f815bf1c7f58c589436bc80abf968fa9bac3f5e21e83744" already present on machine
10:42:11 (x2) | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-7cb4cb644d-wx58q | Created | Created container snapshot-controller
10:42:11 (x2) | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-7cb4cb644d-wx58q | Started | Started container snapshot-controller
10:42:11 | openshift-cluster-storage-operator | snapshot-controller-leader/csi-snapshot-controller-7cb4cb644d-wx58q | snapshot-controller-leader | LeaderElection | csi-snapshot-controller-7cb4cb644d-wx58q became leader
10:42:22 | openshift-cluster-version | kubelet | cluster-version-operator-5d86d65f6b-4fm8l | Pulled | Container image "quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6" already present on machine
10:42:23 (x2) | openshift-cluster-version | kubelet | cluster-version-operator-5d86d65f6b-4fm8l | Created | Created container cluster-version-operator
10:42:23 (x2) | openshift-cluster-version | kubelet | cluster-version-operator-5d86d65f6b-4fm8l | Started | Started container cluster-version-operator
10:42:36 (x2) | openshift-cluster-storage-operator | kubelet | cluster-storage-operator-bc56464-b7kxp | Started | Started container cluster-storage-operator
10:42:36 | openshift-cluster-storage-operator | kubelet | cluster-storage-operator-bc56464-b7kxp | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1c3255b55cd57639781c529c27f98ee3e8101c31265cbd6cc63896eee70d63c" already present on machine
10:42:36 (x2) | openshift-cluster-storage-operator | kubelet | cluster-storage-operator-bc56464-b7kxp | Created | Created container cluster-storage-operator
10:42:37 | openshift-etcd-operator | kubelet | etcd-operator-859798c7f4-kb5lq | BackOff | Back-off restarting failed container
10:42:37 | openshift-image-registry | kubelet | cluster-image-registry-operator-98c4d4975-kxn6n | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf64307a7341e67a8a6cea2649cfa1734be025fd1ec515d9f739f9cc9038764c" already present on machine
10:42:37 | openshift-sdn | kubelet | sdn-controller-2pwxc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" already present on machine
10:42:37 | openshift-kube-storage-version-migrator-operator | kubelet | kube-storage-version-migrator-operator-5999bc745c-nqrdg | BackOff | Back-off restarting failed container
10:42:37 | openshift-apiserver-operator | kubelet | openshift-apiserver-operator-56c79799ff-ntg2n | BackOff | Back-off restarting failed container
10:42:37 | openshift-authentication-operator | kubelet | authentication-operator-6fcbcf79b6-5h2gz | BackOff | Back-off restarting failed container
10:42:37 | openshift-controller-manager-operator | kubelet | openshift-controller-manager-operator-7cbb8486b4-2jgvl | BackOff | Back-off restarting failed container
10:42:38 (x2) | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Unhealthy | Readiness probe failed: Get "https://10.130.0.6:8443/healthz": dial tcp 10.130.0.6:8443: connect: connection refused
10:42:38 (x2) | openshift-image-registry | kubelet | cluster-image-registry-operator-98c4d4975-kxn6n | Created | Created container cluster-image-registry-operator
10:42:38 | openshift-kube-apiserver-operator | kubelet | kube-apiserver-operator-c74696d8d-gsntz | BackOff | Back-off restarting failed container
10:42:38 | openshift-kube-scheduler-operator | kubelet | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | BackOff | Back-off restarting failed container
10:42:38 (x2) | openshift-image-registry | kubelet | cluster-image-registry-operator-98c4d4975-kxn6n | Started | Started container cluster-image-registry-operator
10:42:38 (x2) | openshift-sdn | kubelet | sdn-controller-2pwxc | Started | Started container sdn-controller
10:42:38 (x2) | openshift-sdn | kubelet | sdn-controller-2pwxc | Created | Created container sdn-controller
10:42:39 (x2) | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | BackOff | Back-off restarting failed container
10:42:40 | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-operator-76dfcb9b88-lt78x | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6b798a2aeb0d74b2644b613801feb06411c9f6444e4be462073b72df5bb35451" already present on machine
10:42:40 (x2) | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-operator-76dfcb9b88-lt78x | Created | Created container csi-snapshot-controller-operator
10:42:41 (x2) | openshift-cluster-storage-operator | kubelet | csi-snapshot-controller-operator-76dfcb9b88-lt78x | Started | Started container csi-snapshot-controller-operator
10:42:42 | openshift-kube-controller-manager-operator | kubelet | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | BackOff | Back-off restarting failed container
10:42:43 | openshift-service-ca-operator | kubelet | service-ca-operator-6fd6454f86-5bbh8 | BackOff | Back-off restarting failed container
10:42:43 | openshift-controller-manager | kubelet | controller-manager-kzgff | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" already present on machine
10:42:44 (x2) | openshift-controller-manager | kubelet | controller-manager-kzgff | Started | Started container controller-manager
10:42:44 (x2) | openshift-controller-manager | kubelet | controller-manager-kzgff | Created | Created container controller-manager
10:42:44 | openshift-controller-manager | openshift-controller-manager | openshift-master-controllers | LeaderElection | controller-manager-kzgff became leader
10:42:49 (x3) | openshift-etcd-operator | kubelet | etcd-operator-859798c7f4-kb5lq | Started | Started container etcd-operator
10:42:49 (x3) | openshift-kube-apiserver-operator | kubelet | kube-apiserver-operator-c74696d8d-gsntz | Created | Created container kube-apiserver-operator
10:42:49 (x3) | openshift-etcd-operator | kubelet | etcd-operator-859798c7f4-kb5lq | Created | Created container etcd-operator
10:42:49 (x2) | openshift-kube-apiserver-operator | kubelet | kube-apiserver-operator-c74696d8d-gsntz | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:42:49 (x3) | openshift-apiserver-operator | kubelet | openshift-apiserver-operator-56c79799ff-ntg2n | Started | Started container openshift-apiserver-operator
10:42:49 (x2) | openshift-etcd-operator | kubelet | etcd-operator-859798c7f4-kb5lq | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:42:49 (x3) | openshift-apiserver-operator | kubelet | openshift-apiserver-operator-56c79799ff-ntg2n | Created | Created container openshift-apiserver-operator
10:42:49 (x2) | openshift-apiserver-operator | kubelet | openshift-apiserver-operator-56c79799ff-ntg2n | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46c56da38fc8caba5ee400ff9aea7dd30fa6cc23283517e912e0a054250b200c" already present on machine
10:42:49 (x3) | openshift-kube-apiserver-operator | kubelet | kube-apiserver-operator-c74696d8d-gsntz | Started | Started container kube-apiserver-operator
10:42:50 (x2) | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7932986aa8485857de987ccfd248146e06d18cf52f65d28fa607f47cab34a49" already present on machine
10:42:50 (x3) | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Created | Created container openshift-config-operator
10:42:50 (x3) | openshift-config-operator | kubelet | openshift-config-operator-697f744f69-g6ck2 | Started | Started container openshift-config-operator
10:42:52 (x2) | openshift-kube-storage-version-migrator-operator | kubelet | kube-storage-version-migrator-operator-5999bc745c-nqrdg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10ba9f188dfb4e2a087bce84c8b0efb93313cd194aa5d4542d478c1154210cf7" already present on machine
10:42:52 (x2) | openshift-controller-manager-operator | kubelet | openshift-controller-manager-operator-7cbb8486b4-2jgvl | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f22313adee458c064e331dc1dbd0846895ed6ad3e1607f0b523d5a89ab34ed96" already present on machine
10:42:52 (x3) | openshift-controller-manager-operator | kubelet | openshift-controller-manager-operator-7cbb8486b4-2jgvl | Started | Started container openshift-controller-manager-operator
10:42:52 (x3) | openshift-kube-storage-version-migrator-operator | kubelet | kube-storage-version-migrator-operator-5999bc745c-nqrdg | Started | Started container kube-storage-version-migrator-operator
10:42:52 (x3) | openshift-authentication-operator | kubelet | authentication-operator-6fcbcf79b6-5h2gz | Started | Started container authentication-operator
10:42:52 (x3) | openshift-kube-storage-version-migrator-operator | kubelet | kube-storage-version-migrator-operator-5999bc745c-nqrdg | Created | Created container kube-storage-version-migrator-operator
10:42:52 (x3) | openshift-controller-manager-operator | kubelet | openshift-controller-manager-operator-7cbb8486b4-2jgvl | Created | Created container openshift-controller-manager-operator
10:42:52 (x3) | openshift-authentication-operator | kubelet | authentication-operator-6fcbcf79b6-5h2gz | Created | Created container authentication-operator
10:42:52 (x2) | openshift-authentication-operator | kubelet | authentication-operator-6fcbcf79b6-5h2gz | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0e9ab39a035468cdc17b59fe61931ca6df3509099b6c2267338215499105524" already present on machine
10:42:53 (x3) | openshift-kube-controller-manager-operator | kubelet | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | Created | Created container kube-controller-manager-operator
10:42:53 (x3) | openshift-kube-controller-manager-operator | kubelet | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | Started | Started container kube-controller-manager-operator
10:42:53 (x2) | openshift-kube-controller-manager-operator | kubelet | kube-controller-manager-operator-7bb9b6d8b8-h8hjn | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:42:54 (x3) | openshift-kube-scheduler-operator | kubelet | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | Created | Created container kube-scheduler-operator-container
10:42:54 (x3) | openshift-kube-scheduler-operator | kubelet | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | Started | Started container kube-scheduler-operator-container
10:42:54 (x2) | openshift-kube-scheduler-operator | kubelet | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:42:56 (x2) | openshift-service-ca-operator | kubelet | service-ca-operator-6fd6454f86-5bbh8 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf9539d6f39fdfc4049ed5c3f16ed3b9b97c11408b9b36fc7c2090d1cbddf178" already present on machine
10:42:56 (x3) | openshift-service-ca-operator | kubelet | service-ca-operator-6fd6454f86-5bbh8 | Started | Started container service-ca-operator
10:42:56 (x3) | openshift-service-ca-operator | kubelet | service-ca-operator-6fd6454f86-5bbh8 | Created | Created container service-ca-operator
10:42:58 (x2) | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Created | Created container machine-api-operator
10:42:58 (x2) | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Started | Started container machine-api-operator
10:42:58 | openshift-machine-api | kubelet | machine-api-operator-784c77b6d4-xfllc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ed187e9c9e7585720b3b407ceeeb827d2d4d38fad8843cd914677eb3ab53c13b" already present on machine
10:43:05 | openshift-cluster-version | openshift-cluster-version | version | PayloadLoaded | payload loaded version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:43:05 | openshift-cluster-version | openshift-cluster-version | version | VerifyPayload | verifying payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:43:05 | openshift-cluster-version | openshift-cluster-version | version | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_de14dae2-3d86-4a63-90ce-75039f864b9e became leader
10:43:05 | openshift-cluster-version | openshift-cluster-version | version | RetrievePayload | retrieving payload version="4.6.52" image="quay.io/openshift-release-dev/ocp-release@sha256:58c2e7bd1b248b1ffd9829b84b4ee89bb93e3e47b997d312e7e1ade5517eb1a6"
10:43:10 (x2) | openshift-machine-config-operator | kubelet | machine-config-operator-65cd958fc7-b5cm6 | Created | Created container machine-config-operator
10:43:10 (x2) | openshift-machine-config-operator | kubelet | machine-config-operator-65cd958fc7-b5cm6 | Started | Started container machine-config-operator
10:43:10 (x2) | openshift-machine-config-operator | kubelet | machine-config-operator-65cd958fc7-b5cm6 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:43:20 (x2) | openshift-machine-config-operator | kubelet | machine-config-controller-7df85d4c4-lsgx2 | Started | Started container machine-config-controller
10:43:20 (x2) | openshift-machine-config-operator | kubelet | machine-config-controller-7df85d4c4-lsgx2 | Created | Created container machine-config-controller
10:43:20 (x2) | openshift-machine-config-operator | kubelet | machine-config-controller-7df85d4c4-lsgx2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:43:22 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com_openshift-kube-scheduler_f7fa578f-d496-44b7-8e33-b9dd54d0ddbc_0(7fc03d992a3c2aea864df04a4a4dd374128ec4da682cfbb8d3d7fe429c25c582): [openshift-kube-scheduler/revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com:openshift-sdn]: error adding container to network "openshift-sdn": CNI request failed with status 400: 'the server was unable to return a response in the time allotted, but may still be processing the request (get pods revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com) '
10:43:25 | openshift-console | deployment-controller | downloads | ScalingReplicaSet | Scaled up replica set downloads-88d4c8dfb to 2
10:43:25 | openshift-console | replicaset-controller | downloads-88d4c8dfb | SuccessfulCreate | Created pod: downloads-88d4c8dfb-5g9qb
10:43:25 | openshift-console | replicaset-controller | downloads-88d4c8dfb | SuccessfulCreate | Created pod: downloads-88d4c8dfb-w49d2
10:43:37 | openshift-cluster-storage-operator | cluster-storage-operator | cluster-storage-operator-lock | LeaderElection | cluster-storage-operator-bc56464-b7kxp_d61446dd-e276-4868-9220-a191acd60139 became leader
10:43:42 | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c41fd0616e68c5a87537186bc935c5d846d2beaef2c8b2d9343ceec4b6263be" already present on machine
10:43:43 | openshift-cluster-storage-operator | csi-snapshot-controller-operator-loggingsyncer | csi-snapshot-controller-operator | OperatorLogLevelChange | Operator log level changed from "Debug" to "Normal"
10:43:43 (x2) | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Started | Started container cluster-autoscaler-operator
10:43:43 | openshift-cluster-storage-operator | csi-snapshot-controller-operator | csi-snapshot-controller-operator-lock | LeaderElection | csi-snapshot-controller-operator-76dfcb9b88-lt78x_0866e10d-9698-4a2f-a8e6-19709977fa74 became leader
10:43:43 (x2) | openshift-machine-api | kubelet | cluster-autoscaler-operator-557599ccf8-lchg2 | Created | Created container cluster-autoscaler-operator
10:43:50 | openshift-image-registry | daemonset-controller | node-ca | SuccessfulCreate | Created pod: node-ca-tqqc8
10:43:50 | openshift-image-registry | daemonset-controller | node-ca | SuccessfulCreate | Created pod: node-ca-gjw8n
10:43:50 | openshift-image-registry | daemonset-controller | node-ca | SuccessfulCreate | Created pod: node-ca-n2hfh
10:43:50 | openshift-image-registry | image-registry-operator | openshift-master-controllers | LeaderElection | cluster-image-registry-operator-98c4d4975-kxn6n_ad9eda44-99ba-40ba-8519-749b280b6d75 became leader
10:43:51 | openshift-image-registry | kubelet | node-ca-gjw8n | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5"
10:43:51 | openshift-image-registry | kubelet | node-ca-tqqc8 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5"
10:43:51 | openshift-image-registry | kubelet | node-ca-n2hfh | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5"
10:43:52 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator-lock | LeaderElection | openshift-apiserver-operator-56c79799ff-ntg2n_b04cd2ff-a2dc-4ac8-859d-f1a6e9ef52c7 became leader
10:43:53 | openshift-image-registry | kubelet | node-ca-tqqc8 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" in 2.66150562s
10:43:53 | openshift-image-registry | kubelet | node-ca-gjw8n | Created | Created container node-ca
10:43:53 | openshift-image-registry | kubelet | node-ca-n2hfh | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" in 2.312645672s
10:43:53 | openshift-image-registry | kubelet | node-ca-gjw8n | Started | Started container node-ca
10:43:53 | openshift-image-registry | kubelet | node-ca-n2hfh | Started | Started container node-ca
10:43:53 | openshift-image-registry | kubelet | node-ca-gjw8n | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" in 2.34254321s
10:43:53 | openshift-image-registry | kubelet | node-ca-n2hfh | Created | Created container node-ca
10:43:54 | openshift-image-registry | kubelet | node-ca-tqqc8 | Started | Started container node-ca
10:43:54 | openshift-image-registry | kubelet | node-ca-tqqc8 | Created | Created container node-ca
10:43:54 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | APIServicesToManageChanged | The new API Services list this operator will manage is [v1.apps.openshift.io v1.authorization.openshift.io v1.build.openshift.io v1.image.openshift.io v1.project.openshift.io v1.quota.openshift.io v1.route.openshift.io v1.security.openshift.io v1.template.openshift.io]
10:43:55 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Available changed from False to True ("")
10:43:57 | openshift-etcd-operator | openshift-cluster-etcd-operator-loggingsyncer | etcd-operator | OperatorLogLevelChange | Operator log level changed from "Debug" to "Normal"
10:43:57 (x3) | openshift-etcd-operator | openshift-cluster-etcd-operator-member-observer-controller-etcdmemberscontroller | etcd-operator | ReportEtcdMembersErrorUpdatingStatus | etcds.operator.openshift.io "cluster" not found
10:43:57 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:43:57 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:43:57 | openshift-etcd-operator | openshift-cluster-etcd-operator | openshift-cluster-etcd-operator-lock | LeaderElection | etcd-operator-859798c7f4-kb5lq_18fcd5e6-c8d0-4c4d-aba7-24ba745ea443 became leader
10:43:57 | openshift-config-operator | config-operator | config-operator-lock | LeaderElection | openshift-config-operator-697f744f69-g6ck2_884cac45-b475-4481-a56f-5ff591022954 became leader
10:43:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator | kube-controller-manager-operator-lock | LeaderElection | kube-controller-manager-operator-7bb9b6d8b8-h8hjn_e776a105-4aea-4dfc-b77a-0dec7ceec2e5 became leader
10:43:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "ScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:43:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 4 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available"
10:43:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-endpoints-controller-etcdendpointscontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/etcd-endpoints -n openshift-etcd:
10:43:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 7 triggered by "secret/localhost-recovery-client-token has changed"
10:43:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:43:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded changed from False to True ("NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending")
10:43:58 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "EtcdMembersControllerDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:43:59 | openshift-kube-apiserver-operator | kube-apiserver-operator | kube-apiserver-operator-lock | LeaderElection | kube-apiserver-operator-c74696d8d-gsntz_c1a11fb1-129e-4286-b963-123e7d08019b became leader
10:43:59 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 3 triggered by "secret/localhost-recovery-client-token has changed"
10:44:00 | openshift-service-ca-operator | service-ca-operator | service-ca-operator-lock | LeaderElection | service-ca-operator-6fd6454f86-5bbh8_a648c38b-f022-4936-8d1b-cc3800e408c6 became leader
10:44:00 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-7 -n openshift-kube-controller-manager because it was missing
10:44:00 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "ScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:44:01 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:44:02 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found"
10:44:02 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-lock | LeaderElection | kube-storage-version-migrator-operator-5999bc745c-nqrdg_eb06ad9a-d9e9-402c-a683-6f28655b48a0 became leader
10:44:02 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded changed from False to True ("NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending")
10:44:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SATokenSignerControllerOK | found expected kube-apiserver endpoints
10:44:02 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found"
10:44:02 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionTriggered | new revision 6 triggered by "secret/localhost-recovery-client-token has changed"
10:44:02 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-lock | LeaderElection | openshift-kube-scheduler-operator-85dfcb9bcd-s2b7k_f0e49ba3-fb34-48f9-9d43-6232be93fca7 became leader
10:44:02 | openshift-authentication-operator | cluster-authentication-operator | cluster-authentication-operator-lock | LeaderElection | authentication-operator-6fcbcf79b6-5h2gz_2b2b4071-b75c-445f-9908-bb5cb3e800a2 became leader
10:44:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-6 -n openshift-kube-controller-manager: cause by changes in data.status
10:44:03 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-3 -n openshift-kube-apiserver because it was missing
10:44:03 | openshift-etcd-operator | openshift-cluster-etcd-operator-script-controller-scriptcontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/etcd-scripts -n openshift-etcd: cause by changes in data.etcd.env
10:44:03 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/revision-status-6 -n openshift-kube-scheduler because it was missing
10:44:04 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SecretCreated | Created Secret/next-service-account-private-key -n openshift-kube-controller-manager-operator because it was missing
10:44:04 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-7 -n openshift-kube-controller-manager because it was missing
10:44:05 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nAPIServicesAvailable: \"oauth.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nAPIServicesAvailable: \"user.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
10:44:05 | openshift-authentication-operator | oauth-apiserver | authentication-operator | APIServicesToManageChanged | The new API Services list this operator will manage is [v1.oauth.openshift.io v1.user.openshift.io]
10:44:05 (x10) | openshift-authentication-operator | oauth-apiserver-manage-oauth-api-controller-mangeoauthapicontroller | authentication-operator | PrereqNotReady | OAuth API servers are not ready. APIService check reporeted "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) "user.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
10:44:05 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-5 -n openshift-kube-scheduler: cause by changes in data.status
10:44:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-2 -n openshift-kube-apiserver: cause by changes in data.status
10:44:05 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/sa-token-signing-certs -n openshift-config-managed: cause by changes in data.service-account-002.pub
10:44:05 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-7 -n openshift-kube-controller-manager because it was missing
10:44:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending\nSATokenSignerDegraded: Operation cannot be fulfilled on configmaps \"sa-token-signing-certs\": the object has been modified; please apply your changes to the latest version and try again"
10:44:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-7 -n openshift-kube-controller-manager because it was missing
10:44:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending\nSATokenSignerDegraded: Operation cannot be fulfilled on configmaps \"sa-token-signing-certs\": the object has been modified; please apply your changes to the latest version and try again" to "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending"
10:44:06 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-pod-6 -n openshift-kube-scheduler because it was missing
10:44:06 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: route.route.openshift.io \"oauth-openshift\" not found"
10:44:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | ConfigMapUpdateFailed | Failed to update ConfigMap/sa-token-signing-certs -n openshift-config-managed: Operation cannot be fulfilled on configmaps "sa-token-signing-certs": the object has been modified; please apply your changes to the latest version and try again
10:44:06 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found"
10:44:06 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator-lock | LeaderElection | openshift-controller-manager-operator-7cbb8486b4-2jgvl_16eca5d7-25cb-4e81-8d4c-ce8c2f9799b3 became leader
10:44:07 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: route.route.openshift.io \"oauth-openshift\" not found"
10:44:07 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-pod-3 -n openshift-kube-apiserver because it was missing
10:44:07 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-7 -n openshift-kube-controller-manager because it was missing
10:44:07 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because static pod is ready
10:44:07 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/config-6 -n openshift-kube-scheduler because it was missing
10:44:07 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 2" to "NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 2",Available message changed from "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 2" to "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2"
10:44:08 (x17) | openshift-authentication-operator | cluster-authentication-operator | authentication-operator | OAuthRouteCheck | failed to get oauth route: route.route.openshift.io "oauth-openshift" not found
10:44:08 | openshift-kube-controller-manager | multus | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.26/23]
10:44:08 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found"
10:44:08 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:08 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-6 -n openshift-kube-scheduler because it was missing
10:44:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/sa-token-signing-certs -n openshift-kube-apiserver: cause by changes in data.service-account-002.pub
10:44:08 | openshift-kube-scheduler | multus | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.18/23]
10:44:08 | openshift-etcd-operator | openshift-cluster-etcd-operator-target-config-controller-targetconfigcontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/etcd-pod -n openshift-etcd: cause by changes in data.pod.yaml
10:44:08 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | RevisionTriggered | new revision 3 triggered by "configmap/etcd-pod has changed"
10:44:08 | openshift-console | kubelet | downloads-88d4c8dfb-5g9qb | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_downloads-88d4c8dfb-5g9qb_openshift-console_3bcf3885-72f6-4040-b2a4-83de4d09fc51_0(b0247d3367a86d9414b8ca3c8f2ee5b7339453ae57fb293a4eff844126c5ff79): [openshift-console/downloads-88d4c8dfb-5g9qb:openshift-sdn]: error adding container to network "openshift-sdn": CNI request failed with status 400: 'Get "https://api-int.case03116926.lab.upshift.rdu2.redhat.com:6443/api/v1/namespaces/openshift-console/pods/downloads-88d4c8dfb-5g9qb": unexpected EOF '
10:44:08 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_downloads-88d4c8dfb-w49d2_openshift-console_52d9b14f-2245-482d-9833-6d97258abe03_0(72262e23129b6b1d2a66926ac338d72e5393fd30a18c7eb5c22627343c930a56): [openshift-console/downloads-88d4c8dfb-w49d2:openshift-sdn]: error adding container to network "openshift-sdn": CNI request failed with status 400: 'Get "https://api-int.case03116926.lab.upshift.rdu2.redhat.com:6443/api/v1/namespaces/openshift-console/pods/downloads-88d4c8dfb-w49d2": unexpected EOF '
10:44:09 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-pod-3 -n openshift-etcd because it was missing
10:44:09 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:09 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:09 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/revision-status-3 -n openshift-etcd because it was missing
10:44:09 | openshift-etcd-operator | openshift-cluster-etcd-operator-target-config-controller-targetconfigcontroller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/restore-etcd-pod -n openshift-etcd: cause by changes in data.pod.yaml
10:44:09 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:09 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:09 | openshift-console-operator | deployment-controller | console-operator | ScalingReplicaSet | Scaled up replica set console-operator-68cc7b855f to 1
10:44:09 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nOAuthVersionRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty",Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:44:09 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:44:09 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: route.route.openshift.io \"oauth-openshift\" not found\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:09 (x2) | openshift-operator-lifecycle-manager | operator-lifecycle-manager | packageserver | InstallSucceeded | install strategy completed with no errors
10:44:09 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending"
10:44:09 | openshift-console-operator | replicaset-controller | console-operator-68cc7b855f | SuccessfulCreate | Created pod: console-operator-68cc7b855f-nldb7
10:44:09 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found"
10:44:10 | openshift-kube-controller-manager | kubelet | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:44:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nResourceSyncControllerDegraded: Operation cannot be fulfilled on configmaps \"sa-token-signing-certs\": the object has been modified; please apply your changes to the latest version and try again" to "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found"
10:44:10 | openshift-kube-controller-manager | kubelet | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:44:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/config-3 -n openshift-kube-apiserver because it was missing
10:44:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapUpdateFailed | Failed to update ConfigMap/sa-token-signing-certs -n openshift-kube-apiserver: Operation cannot be fulfilled on configmaps "sa-token-signing-certs": the object has been modified; please apply your changes to the latest version and try again
10:44:10 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-7 -n openshift-kube-controller-manager because it was missing
10:44:10 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/scheduler-kubeconfig-6 -n openshift-kube-scheduler because it was missing
10:44:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found" to "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nResourceSyncControllerDegraded: Operation cannot be fulfilled on configmaps \"sa-token-signing-certs\": the object has been modified; please apply your changes to the latest version and try again"
10:44:10 | openshift-kube-controller-manager | kubelet | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:44:10 | openshift-authentication-operator | cluster-authentication-operator-payload-config-controller-payloadconfig | authentication-operator | SecretCreated | Created Secret/v4-0-config-system-session -n openshift-authentication because it was missing
10:44:10 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/config-3 -n openshift-etcd because it was missing
10:44:11 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:44:11 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:44:11 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-7 -n openshift-kube-controller-manager because it was missing
10:44:11 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:44:11 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca-3 -n openshift-etcd because it was missing
10:44:12 | openshift-monitoring | deployment-controller | prometheus-operator | ScalingReplicaSet | Scaled up replica set prometheus-operator-79ffc97ccd to 1
10:44:12 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-peer-client-ca-3 -n openshift-etcd because it was missing
10:44:12 | openshift-console-operator | multus | console-operator-68cc7b855f-nldb7 | AddedInterface | Add eth0 [10.130.0.27/23]
10:44:12 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | ConfigMapCreated | Created ConfigMap/kube-scheduler-cert-syncer-kubeconfig-6 -n openshift-kube-scheduler because it was missing
10:44:12 | openshift-monitoring | replicaset-controller | prometheus-operator-79ffc97ccd | SuccessfulCreate | Created pod: prometheus-operator-79ffc97ccd-5h2pl
10:44:12 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-cert-syncer-kubeconfig-3 -n openshift-kube-apiserver because it was missing
10:44:13 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:44:13 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:44:13 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:44:13 | openshift-monitoring | deployment-controller | prometheus-adapter | ScalingReplicaSet | Scaled up replica set prometheus-adapter-5fdd4bd886 to 2
10:44:13 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Unable to get oauth-openshift route: route.route.openshift.io \"oauth-openshift\" not found" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:44:13 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-7 -n openshift-kube-controller-manager because it was missing
10:44:13 | openshift-kube-controller-manager | static-pod-installer | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 6
10:44:13 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/serving-cert-6 -n openshift-kube-scheduler because it was missing
10:44:13 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-peer-3 -n openshift-etcd because it was missing
10:44:13 | openshift-console-operator | kubelet | console-operator-68cc7b855f-nldb7 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6ef1b92b9d6147acd487c1230256c06910ae96071778fad99eec8545ec38fb4"
10:44:13 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-metrics-proxy-client-ca-3 -n openshift-etcd because it was missing
10:44:13 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:44:13 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | ConfigMapCreated | Created ConfigMap/etcd-metrics-proxy-serving-ca-3 -n openshift-etcd because it was missing
10:44:14 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | RevisionCreate | Revision 2 created because configmap/etcd-pod has changed
10:44:14 | openshift-authentication-operator | cluster-authentication-operator-payload-config-controller-payloadconfig | authentication-operator | ConfigMapCreated | Created ConfigMap/v4-0-config-system-cliconfig -n openshift-authentication because it was missing
10:44:14 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | RevisionCreate | Revision 5 created because secret/localhost-recovery-client-token has changed
10:44:14 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-revisioncontroller | openshift-kube-scheduler-operator | SecretCreated | Created Secret/localhost-recovery-client-token-6 -n openshift-kube-scheduler because it was missing
10:44:14 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-serving-3 -n openshift-etcd because it was missing
10:44:14 | openshift-monitoring | replicaset-controller | prometheus-adapter-5fdd4bd886 | SuccessfulCreate | Created pod: prometheus-adapter-5fdd4bd886-nnfsr
10:44:14 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-7 -n openshift-kube-controller-manager because it was missing
10:44:14 | openshift-monitoring | replicaset-controller | prometheus-adapter-5fdd4bd886 | SuccessfulCreate | Created pod: prometheus-adapter-5fdd4bd886-bzs9k
10:44:14 | openshift-etcd-operator | openshift-cluster-etcd-operator-revisioncontroller | etcd-operator | SecretCreated | Created Secret/etcd-all-serving-metrics-3 -n openshift-etcd because it was missing
10:44:14 | openshift-monitoring | multus | prometheus-operator-79ffc97ccd-5h2pl | AddedInterface | Add eth0 [10.130.0.28/23]
10:44:15 | openshift-console-operator | kubelet | console-operator-68cc7b855f-nldb7 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6ef1b92b9d6147acd487c1230256c06910ae96071778fad99eec8545ec38fb4" in 2.856813282s
10:44:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 2 because node master-0.case03116926.lab.upshift.rdu2.redhat.com static pod not found
10:44:15 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-7 -n openshift-kube-controller-manager because it was missing
10:44:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:44:15 | openshift-monitoring | kubelet | prometheus-operator-79ffc97ccd-5h2pl | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c651f0af4ba650af214a59e3f974b72caeda32535ce116617ea588e3fdb762e0" already present on machine
10:44:16 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 5" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 5; 0 nodes have achieved new revision 6",Available message changed from "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 5" to "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 5; 0 nodes have achieved new revision 6"
10:44:16 | openshift-console-operator | kubelet | console-operator-68cc7b855f-nldb7 | Created | Created container console-operator
10:44:16 (x2) | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 5 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 0 is the oldest not matching its expected revision 0
10:44:16 | openshift-console-operator | console-operator | console-operator-lock | LeaderElection | console-operator-68cc7b855f-nldb7_8e137d95-eaa9-4842-8860-c032b5a79857 became leader
10:44:16 | openshift-monitoring | kubelet | prometheus-operator-79ffc97ccd-5h2pl | Started | Started container kube-rbac-proxy
10:44:16 | openshift-monitoring | kubelet | prometheus-operator-79ffc97ccd-5h2pl | Created | Created container kube-rbac-proxy
10:44:16 | openshift-monitoring | kubelet | prometheus-operator-79ffc97ccd-5h2pl | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:44:16 | openshift-monitoring | kubelet | prometheus-operator-79ffc97ccd-5h2pl | Started | Started container prometheus-operator
10:44:16 | openshift-monitoring | kubelet | prometheus-operator-79ffc97ccd-5h2pl | Created | Created container prometheus-operator
10:44:16 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-7 -n openshift-kube-controller-manager because it was missing
10:44:16 | openshift-console-operator | kubelet | console-operator-68cc7b855f-nldb7 | Started | Started container console-operator
10:44:17 | openshift-console-operator | console-operator | console-operator | ServiceCreated | Created Service/console -n openshift-console because it was missing
10:44:17 | openshift-console-operator | console-operator-resource-sync-controller-resourcesynccontroller | console-operator | ConfigMapCreated | Created ConfigMap/default-ingress-cert -n openshift-console because it was missing
10:44:17 | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Killing | Stopping container prometheus-operator
10:44:17 | openshift-monitoring | kubelet | prometheus-operator-6db8fc66c7-rgvbc | Killing | Stopping container kube-rbac-proxy
10:44:17 | openshift-monitoring | replicaset-controller | prometheus-operator-6db8fc66c7 | SuccessfulDelete | Deleted pod: prometheus-operator-6db8fc66c7-rgvbc
10:44:17 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing changed from Unknown to False ("")
10:44:17 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 6 created because secret/localhost-recovery-client-token has changed
10:44:17 | openshift-kube-apiserver | multus | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.19/23]
10:44:17 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Upgradeable changed from Unknown to True ("")
10:44:17 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-7 -n openshift-kube-controller-manager because it was missing
10:44:17 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Degraded set to Unknown (""),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to Unknown (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "consoles" "" "cluster"} {"config.openshift.io" "consoles" "" "cluster"} {"config.openshift.io" "infrastructures" "" "cluster"} {"config.openshift.io" "proxies" "" "cluster"} {"config.openshift.io" "oauths" "" "cluster"} {"oauth.openshift.io" "oauthclients" "" "console"} {"" "namespaces" "" "openshift-console-operator"} {"" "namespaces" "" "openshift-console"} {"" "configmaps" "openshift-config-managed" "console-public"}],status.versions changed from [] to [{"operator" "4.6.52"}]
10:44:17 | openshift-monitoring | deployment-controller | prometheus-operator | ScalingReplicaSet | Scaled down replica set prometheus-operator-6db8fc66c7 to 0
10:44:17 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorVersionChanged | clusteroperator/console version "operator" changed from "" to "4.6.52"
10:44:17 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Degraded changed from Unknown to False ("")
10:44:18 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:44:18 | openshift-console-operator | console-operator | console-operator | ConfigMapCreated | Created ConfigMap/console-config -n openshift-console because it was missing
10:44:19 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:44:19 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available"
10:44:19 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 6" to "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 6; 0 nodes have achieved new revision 7",Available message changed from "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 6" to "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 6; 0 nodes have achieved new revision 7"
10:44:19 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing changed from False to True ("DefaultRouteSyncProgressing: route \"console\" is not available at canonical host []")
10:44:19 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
10:44:20 | openshift-kube-scheduler | multus | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.29/23]
10:44:20 | openshift-console-operator | console-operator | console-operator | SecretCreated | Created Secret/console-oauth-config -n openshift-console because it was missing
10:44:20 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:44:20 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:44:21 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/bound-sa-token-signing-certs-3 -n openshift-kube-apiserver because it was missing
10:44:21 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending; 1 nodes are failing on revision 6:\nNodeInstallerDegraded: static pod of revision 6 has been installed, but is not ready while new revision 7 is pending"
10:44:22 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-6 -n openshift-kube-controller-manager: cause by changes in data.status
10:44:22 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing message changed from "DefaultRouteSyncProgressing: route \"console\" is not available at canonical host []" to "DefaultRouteSyncProgressing: route \"console\" is not available at canonical host []\nOAuthClientSyncProgressing: route \"console\" is not available at canonical host []"
10:44:22 | openshift-console | multus | downloads-88d4c8dfb-w49d2 | AddedInterface | Add eth0 [10.129.0.20/23]
10:44:22 | openshift-kube-scheduler | kubelet | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:44:22 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | PodCreated | Created Pod/installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:44:22 | openshift-kube-scheduler | kubelet | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:44:23 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6494c710e14aaa44d36c1f0b16cfa170b905a290b88707b5caac15ae5e5de916"
10:44:23 | openshift-kube-scheduler | static-pod-installer | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 5
10:44:23 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-cert-syncer
10:44:23 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-recovery-controller
10:44:23 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler
10:44:23 | openshift-kube-scheduler | kubelet | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:44:24 | openshift-etcd | multus | installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.21/23]
10:44:24 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca-3 -n openshift-kube-apiserver because it was missing
10:44:24 | openshift-console | multus | downloads-88d4c8dfb-5g9qb | AddedInterface | Add eth0 [10.130.0.30/23]
10:44:25 | openshift-console | kubelet | downloads-88d4c8dfb-5g9qb | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6494c710e14aaa44d36c1f0b16cfa170b905a290b88707b5caac15ae5e5de916"
10:44:25 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: ; 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending"
10:44:26 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:44:26 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-server-ca-3 -n openshift-kube-apiserver because it was missing
10:44:26 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-5 -n openshift-kube-scheduler: cause by changes in data.status
10:44:26 | openshift-etcd | kubelet | installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:44:27 | openshift-etcd | kubelet | installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:44:27 | openshift-etcd | kubelet | installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:44:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:44:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kubelet-serving-ca-3 -n openshift-kube-apiserver because it was missing
10:44:28 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending; 1 nodes are failing on revision 6:\nNodeInstallerDegraded: static pod of revision 6 has been installed, but is not ready while new revision 7 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending; 1 nodes are failing on revision 6:\nNodeInstallerDegraded: "
10:44:28 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:44:28 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 7 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 0 is the oldest not matching its expected revision 0
10:44:28 | openshift-kube-controller-manager | multus | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.31/23]
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:44:29 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:44:30 | openshift-kube-apiserver | multus | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.33/23]
10:44:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:44:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:44:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:44:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:44:30 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:44:30 | openshift-kube-scheduler | multus | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.32/23]
10:44:30 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcd-metrics
10:44:30 | openshift-kube-controller-manager | kubelet | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:44:30 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcd
10:44:30 | openshift-etcd | static-pod-installer | installer-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 3
10:44:31 | openshift-kube-controller-manager | kubelet | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:44:31 | openshift-kube-controller-manager | kubelet | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:44:31 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-7-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:44:31 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/sa-token-signing-certs-3 -n openshift-kube-apiserver because it was missing
10:44:32 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:44:33 | openshift-kube-controller-manager | multus | installer-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.34/23]
10:44:33 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: "
10:44:33 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 0 is the oldest not matching its expected revision 0
10:44:34 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-audit-policies-3 -n openshift-kube-apiserver because it was missing
10:44:36 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/etcd-client-3 -n openshift-kube-apiserver because it was missing
10:44:37 | openshift-kube-apiserver | kubelet | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:44:37 | openshift-console | kubelet | downloads-88d4c8dfb-5g9qb | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6494c710e14aaa44d36c1f0b16cfa170b905a290b88707b5caac15ae5e5de916" in 12.59038799s
10:44:38 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:44:38 | openshift-kube-apiserver | kubelet | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:44:38 | openshift-kube-apiserver | kubelet | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:44:38 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | Created | Created container download-server
10:44:38 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:44:38 | openshift-console | kubelet | downloads-88d4c8dfb-5g9qb | Started | Started container download-server
10:44:38 | openshift-console | kubelet | downloads-88d4c8dfb-5g9qb | Created | Created container download-server
10:44:38 | openshift-kube-scheduler | kubelet | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:44:38 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/kubelet-client-3 -n openshift-kube-apiserver because it was missing
10:44:38 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6494c710e14aaa44d36c1f0b16cfa170b905a290b88707b5caac15ae5e5de916" in 15.324113934s
10:44:38 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | Started | Started container download-server
10:44:38 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:44:38 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:44:39 | openshift-kube-controller-manager | kubelet | installer-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:44:40 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | Unhealthy | Liveness probe failed: Get "http://10.129.0.20:8080/": dial tcp 10.129.0.20:8080: connect: connection refused
10:44:40 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:44:40 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:44:40 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:44:40 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:44:40 | openshift-kube-controller-manager | kubelet | installer-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:44:40 | openshift-kube-controller-manager | kubelet | installer-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:44:41 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:44:41 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
10:44:41 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-serving-certkey-3 -n openshift-kube-apiserver because it was missing
10:44:41 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:44:41 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:44:41 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
10:44:41 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:44:42 | openshift-console | kubelet | downloads-88d4c8dfb-w49d2 | Unhealthy | Readiness probe failed: Get "http://10.129.0.20:8080/": dial tcp 10.129.0.20:8080: connect: connection refused
10:44:43 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:44:43 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:44:43 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionCreate | Revision 2 created because secret/localhost-recovery-client-token has changed
10:44:43 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:44:43 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:44:43 | openshift-kube-scheduler | multus | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.35/23]
10:44:43 | openshift-kube-controller-manager | static-pod-installer | installer-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 7
10:44:43 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-client-token-3 -n openshift-kube-apiserver because it was missing
10:44:45 | openshift-kube-apiserver | static-pod-installer | installer-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 2
10:44:45 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 2" to "NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 2; 0 nodes have achieved new revision 3",Available message changed from "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2" to "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2; 0 nodes have achieved new revision 3"
10:44:45 | openshift-kube-scheduler | kubelet | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:44:46 | openshift-kube-scheduler | static-pod-installer | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 6
10:44:46 | openshift-kube-scheduler | kubelet | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:44:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:44:46 | openshift-kube-scheduler | kubelet | installer-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:44:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:44:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-cert-syncer
10:44:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler
10:44:46 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-recovery-controller
10:44:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:44:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:44:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:44:50 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found" to "StaticPodsDegraded: pods \"kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: static pod of revision 2 has been installed, but is not ready while new revision 3 is pending"
10:44:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: static pod of revision 2 has been installed, but is not ready while new revision 3 is pending")
10:44:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-2 -n openshift-kube-apiserver: cause by changes in data.status
10:44:58 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:44:59 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:45:01 | openshift-kube-controller-manager | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com_7e9fdc19-9b69-4241-a23b-cdb72d30f545 became leader
10:45:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:45:02 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:45:03 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:45:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:45:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:45:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:45:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:45:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:45:05 | openshift-kube-scheduler | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com_b6e69e1b-adee-4091-af55-97b89876046f became leader
10:45:05 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:45:05 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:45:06 | openshift-kube-apiserver | multus | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.36/23]
10:45:07 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:45:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 3 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest not matching its expected revision 0
10:45:08 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:45:09 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:45:09 | openshift-kube-apiserver | kubelet | revision-pruner-2-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:45:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: static pod of revision 2 has been installed, but is not ready while new revision 3 is pending" to "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: "
10:45:12 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:45:12 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:12 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:13 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-ensure-env-vars
10:45:13 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:45:13 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:45:13 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-ensure-env-vars
10:45:14 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:45:14 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:45:14 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-resources-copy
10:45:14 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-resources-copy
10:45:14 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcdctl
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-metrics
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcdctl
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd
10:45:15 | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-metrics
10:45:16 | openshift-kube-apiserver | multus | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.37/23]
10:45:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 7 because static pod is ready
10:45:18 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:18 | openshift-kube-apiserver | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:45:18 | openshift-kube-apiserver | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:45:18 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending; 1 nodes are failing on revision 6:\nNodeInstallerDegraded: " to "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending",Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 6; 0 nodes have achieved new revision 7" to "NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 6; 1 nodes are at revision 7",Available message changed from "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 6; 0 nodes have achieved new revision 7" to "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 1 nodes are at revision 6; 1 nodes are at revision 7"
10:45:19 | openshift-kube-apiserver | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:45:20 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-7-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:45:22 | openshift-kube-controller-manager | multus | revision-pruner-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.38/23]
10:45:24 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: static pod of revision 5 has been installed, but is not ready while new revision 6 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 5:\nNodeInstallerDegraded: "
10:45:24 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 7 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 1 is the oldest not matching its expected revision 0
10:45:24 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:45:25 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:45:25 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:45:26 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
10:45:26 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
10:45:26 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
10:45:26 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
10:45:26 | openshift-kube-apiserver | static-pod-installer | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 3
10:45:26 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
10:45:27 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-7-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:45:29 | openshift-kube-controller-manager | multus | installer-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.33/23]
10:45:30 | openshift-kube-controller-manager | kubelet | installer-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:45:31 | openshift-kube-controller-manager | kubelet | installer-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:45:31 (x4) | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | UnhealthyEtcdMember | unhealthy members: master-2.case03116926.lab.upshift.rdu2.redhat.com
10:45:31 | openshift-kube-controller-manager | kubelet | installer-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:45:33 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:45:33 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:45:33 | openshift-kube-controller-manager | static-pod-installer | installer-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 7
10:45:33 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:45:33 (x14) | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-flgvm | Unhealthy | Readiness probe failed:
10:45:33 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:45:34 (x4) | openshift-etcd | kubelet | etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com | Unhealthy | Readiness probe failed: dial tcp 10.0.94.219:2380: connect: connection refused
10:45:39 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 2; 0 nodes have achieved new revision 3" to "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3",Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 2; 0 nodes have achieved new revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:39 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:45:15 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:39 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because static pod is ready
10:45:40 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "ClusterMemberControllerDegraded: rpc error: code = Canceled desc = grpc: the client connection is closing\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:41 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "ClusterMemberControllerDegraded: rpc error: code = Canceled desc = grpc: the client connection is closing\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:45:42 | openshift-etcd-operator | openshift-cluster-etcd-operator-prunecontroller | etcd-operator | PodCreated | Created Pod/revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:45:44 | openshift-etcd | multus | revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.22/23]
10:45:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
10:45:46 | openshift-etcd | kubelet | revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:45:47 | openshift-etcd | kubelet | revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:45:47 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:45:47 | openshift-etcd | kubelet | revision-pruner-3-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:45:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | PodCreated | Created Pod/installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:45:48 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:45:51 | openshift-etcd | multus | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.39/23]
10:45:53 | openshift-etcd | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:45:53 | openshift-etcd | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:45:53 | openshift-etcd | kubelet | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:45:56 | openshift-etcd | static-pod-installer | installer-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 3
10:45:56 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcdctl
10:45:56 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcd-metrics
10:45:56 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcd
10:46:00 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because static pod is ready
10:46:00 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending; 1 nodes are failing on revision 5:\nNodeInstallerDegraded: " to "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending",Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 5; 0 nodes have achieved new revision 6" to "NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 5; 1 nodes are at revision 6",Available message changed from "StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 5; 0 nodes have achieved new revision 6" to "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 1 nodes are at revision 5; 1 nodes are at revision 6"
10:46:03 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:46:04 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 7 because static pod is ready
10:46:04 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready"),Progressing message changed from "NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 6; 1 nodes are at revision 7" to "NodeInstallerProgressing: 1 nodes are at revision 6; 2 nodes are at revision 7",Available message changed from "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 1 nodes are at revision 6; 1 nodes are at revision 7" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 6; 2 nodes are at revision 7"
10:46:05 | openshift-kube-scheduler | multus | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.40/23]
10:46:07 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:46:07 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-7-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:46:07 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:46:07 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: static pod of revision 4 has been installed, but is not ready while new revision 5 is pending" to "NodeInstallerDegraded: 1 nodes are failing on revision 4:\nNodeInstallerDegraded: "
10:46:07 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 1 is the oldest not matching its expected revision 0
10:46:07 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:46:07 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:46:07 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:46:09 | openshift-kube-controller-manager | multus | revision-pruner-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.34/23]
10:46:10 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 6 to 7 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 6 is the oldest
10:46:10 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:46:11 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:46:11 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:46:12 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:46:12 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-2.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:46:12 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:46:12 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:46:12 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-7-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:46:13 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:46:13 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:46:14 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-6-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:46:14 | openshift-kube-controller-manager | multus | installer-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.23/23]
10:46:16 | openshift-kube-controller-manager | kubelet | installer-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:46:16 | openshift-kube-scheduler | multus | installer-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.35/23]
10:46:17 | openshift-kube-controller-manager | kubelet | installer-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:46:17 | openshift-kube-controller-manager | kubelet | installer-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:46:18 | openshift-kube-scheduler | kubelet | installer-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:46:18 | openshift-kube-scheduler | kubelet | installer-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:46:18 | openshift-kube-scheduler | kubelet | installer-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:46:19 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-recovery-controller
10:46:19 | openshift-kube-scheduler | static-pod-installer | installer-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 6
10:46:19 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-cert-syncer
10:46:19 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler
10:46:20 | openshift-kube-controller-manager | static-pod-installer | installer-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 7
10:46:20 | openshift-machine-api | cluster-autoscaler-operator-557599ccf8-lchg2_43d57b03-4899-4255-8da3-98db85a63355 | cluster-autoscaler-operator-leader | LeaderElection | cluster-autoscaler-operator-557599ccf8-lchg2_43d57b03-4899-4255-8da3-98db85a63355 became leader
10:46:20 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:46:20 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:46:20 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:46:20 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:46:27 | openshift-dns | daemonset-controller | dns-default | SuccessfulCreate | Created pod: dns-default-zwxfh
10:46:27 | openshift-multus | daemonset-controller | multus | SuccessfulCreate | Created pod: multus-kr75l
10:46:27 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 1 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:46:28 | openshift-machine-config-operator | daemonset-controller | machine-config-daemon | SuccessfulCreate | Created pod: machine-config-daemon-2xgfc
10:46:28 | openshift-sdn | daemonset-controller | ovs | SuccessfulCreate | Created pod: ovs-cb7xs
10:46:28 | openshift-image-registry | daemonset-controller | node-ca | SuccessfulCreate | Created pod: node-ca-lg2rm
10:46:28 | openshift-multus | daemonset-controller | network-metrics-daemon | SuccessfulCreate | Created pod: network-metrics-daemon-pkjvx
10:46:28 | openshift-cluster-node-tuning-operator | daemonset-controller | tuned | SuccessfulCreate | Created pod: tuned-97fwx
10:46:28 | openshift-cluster-node-tuning-operator | daemonset-controller | tuned | SuccessfulCreate | Created pod: tuned-2kbh9
10:46:28 | openshift-machine-config-operator | daemonset-controller | machine-config-daemon | SuccessfulCreate | Created pod: machine-config-daemon-5qfjm
10:46:28 | openshift-multus | daemonset-controller | multus | SuccessfulCreate | Created pod: multus-vrcmx
10:46:28 | openshift-monitoring | daemonset-controller | node-exporter | SuccessfulCreate | Created pod: node-exporter-tg54f
10:46:28 | openshift-sdn | daemonset-controller | ovs | SuccessfulCreate | Created pod: ovs-8mdrj
10:46:28 | openshift-dns | daemonset-controller | dns-default | SuccessfulCreate | Created pod: dns-default-xbgq5
10:46:28 | openshift-sdn | daemonset-controller | sdn | SuccessfulCreate | Created pod: sdn-9n262
10:46:28 | openshift-sdn | daemonset-controller | sdn | SuccessfulCreate | Created pod: sdn-2ff5l
10:46:28 | openshift-image-registry | daemonset-controller | node-ca | SuccessfulCreate | Created pod: node-ca-2qml7
10:46:28 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 1 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 2 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:46:28 | openshift-multus | daemonset-controller | network-metrics-daemon | SuccessfulCreate | Created pod: network-metrics-daemon-m86r5
10:46:28 | openshift-monitoring | daemonset-controller | node-exporter | SuccessfulCreate | Created pod: node-exporter-x72zx
10:46:29 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 2 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:46:29 | openshift-dns | daemonset-controller | dns-default | SuccessfulCreate | Created pod: dns-default-fcp4w
10:46:29 | openshift-sdn | daemonset-controller | sdn | SuccessfulCreate | Created pod: sdn-5vdnk
10:46:29 | openshift-machine-config-operator | daemonset-controller | machine-config-daemon | SuccessfulCreate | Created pod: machine-config-daemon-8ttrt
10:46:29 | openshift-cluster-node-tuning-operator | daemonset-controller | tuned | SuccessfulCreate | Created pod: tuned-dqzh5
10:46:29 | openshift-multus | daemonset-controller | multus | SuccessfulCreate | Created pod: multus-t6jxt
10:46:29 | openshift-image-registry | daemonset-controller | node-ca | SuccessfulCreate | Created pod: node-ca-xzbkf
10:46:29 | openshift-sdn | daemonset-controller | ovs | SuccessfulCreate | Created pod: ovs-dpkmc
10:46:29 | openshift-monitoring | daemonset-controller | node-exporter | SuccessfulCreate | Created pod: node-exporter-bvmqv
10:46:29 | openshift-multus | daemonset-controller | network-metrics-daemon | SuccessfulCreate | Created pod: network-metrics-daemon-xxx4f
10:46:32 (x11) | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-rj76b | Unhealthy | Readiness probe failed:
10:46:32 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:46:32 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-ensure-env-vars
10:46:32 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:46:33 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-ensure-env-vars
10:46:33 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:46:34 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:46:34 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:46:34 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:46:34 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:46:34 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:46:34 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:46:34 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-resources-copy
10:46:34 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-resources-copy
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-metrics
10:46:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcdctl
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:46:35 | openshift-kube-controller-manager | cluster-policy-controller | cluster-policy-controller | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com became leader
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcdctl
10:46:35 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-metrics
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:46:35 | openshift-etcd | kubelet | etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:46:35 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:46:37 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:46:37 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:46:37 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:46:37 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:46:37 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:46:37 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:46:39 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:46:35 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:46:41 (x3) | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | UnhealthyEtcdMember | unhealthy members: master-0.case03116926.lab.upshift.rdu2.redhat.com
10:46:42 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:46:35 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:46:43 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because static pod is ready
10:46:43 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3" to "NodeInstallerProgressing: 1 nodes are at revision 2; 2 nodes are at revision 3",Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy"
10:46:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-prunecontroller | etcd-operator | PodCreated | Created Pod/revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:46:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
10:46:50 | openshift-etcd | multus | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.41/23]
10:46:50 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from False to True ("NodeInstallerDegraded: 1 nodes are failing on revision 2:\nNodeInstallerDegraded: ")
10:46:52 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | PodCreated | Created Pod/installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:46:52 | openshift-etcd | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:46:52 | openshift-etcd | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:46:52 | openshift-etcd | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:46:54 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 7"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 6; 2 nodes are at revision 7" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 7"
10:46:54 | openshift-etcd | multus | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.36/23]
10:46:54 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 6 to 7 because static pod is ready
10:46:55 | openshift-sdn | kubelet | ovs-cb7xs | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:46:55 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-7 -n openshift-kube-controller-manager: cause by changes in data.status
10:46:56 | openshift-sdn | kubelet | ovs-8mdrj | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:46:56 | openshift-multus | kubelet | multus-vrcmx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe"
10:46:56 | openshift-cluster-node-tuning-operator | kubelet | tuned-2kbh9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2"
10:46:56 | openshift-etcd | kubelet | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:46:56 | openshift-image-registry | kubelet | node-ca-2qml7 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5"
10:46:56 | openshift-cluster-node-tuning-operator | kubelet | tuned-97fwx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2"
10:46:56 | openshift-multus | kubelet | multus-kr75l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe"
10:46:56 | openshift-monitoring | kubelet | node-exporter-x72zx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708"
10:46:56 | openshift-etcd | kubelet | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:46:56 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:46:56 | openshift-sdn | kubelet | sdn-9n262 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:46:56 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Created | Created container machine-config-daemon
10:46:56 | openshift-monitoring | kubelet | node-exporter-tg54f | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708"
10:46:56 | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd"
10:46:56 | openshift-sdn | kubelet | sdn-2ff5l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:46:56 | openshift-image-registry | kubelet | node-ca-lg2rm | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5"
10:46:57 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd"
10:46:57 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available"
10:46:57 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: 2 of 3 members are available, master-0.case03116926.lab.upshift.rdu2.redhat.com is unhealthy" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:46:57 | openshift-etcd | kubelet | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:46:57 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Started | Started container machine-config-daemon
10:46:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-7-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:46:59 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcd-metrics
10:46:59 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcd
10:46:59 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container etcdctl
10:46:59 | openshift-etcd | static-pod-installer | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 3
10:47:00 | openshift-kube-controller-manager | multus | revision-pruner-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.24/23]
10:47:01 | openshift-monitoring | kubelet | node-exporter-tg54f | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" in 5.585347542s
10:47:02 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:47:02 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:47:02 | openshift-kube-controller-manager | kubelet | revision-pruner-7-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:47:04 | openshift-cluster-node-tuning-operator | kubelet | tuned-2kbh9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" in 8.708613089s
10:47:04 | openshift-sdn | kubelet | sdn-2ff5l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:47:04 | openshift-image-registry | kubelet | node-ca-lg2rm | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" in 8.614601101s
10:47:04 (x2) | openshift-monitoring | kubelet | node-exporter-tg54f | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:04 | openshift-multus | kubelet | multus-vrcmx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" in 8.521381913s
10:47:04 | openshift-sdn | kubelet | sdn-2ff5l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.713207365s
10:47:04 | openshift-sdn | kubelet | ovs-cb7xs | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.851437681s
10:47:04 | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" in 8.573116709s
10:47:05 (x2) | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:05 | openshift-sdn | kubelet | ovs-8mdrj | Created | Created container openvswitch
10:47:05 | openshift-sdn | kubelet | ovs-8mdrj | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.881443736s
10:47:05 (x2) | openshift-sdn | kubelet | ovs-cb7xs | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:05 | openshift-multus | kubelet | multus-kr75l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" in 8.677849057s
10:47:05 | openshift-multus | kubelet | multus-kr75l | Created | Created container multus-binary-copy
10:47:05 | openshift-image-registry | kubelet | node-ca-2qml7 | Started | Started container node-ca
10:47:05 (x2) | openshift-cluster-node-tuning-operator | kubelet | tuned-2kbh9 | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:05 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" in 8.500238917s
10:47:05 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Created | Created container oauth-proxy
10:47:05 | openshift-cluster-node-tuning-operator | kubelet | tuned-97fwx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" in 8.852373331s
10:47:05 | openshift-sdn | kubelet | sdn-9n262 | Created | Created container sdn
10:47:05 | openshift-sdn | kubelet | sdn-9n262 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.935536278s
10:47:05 | openshift-image-registry | kubelet | node-ca-2qml7 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" in 8.661133547s
10:47:05 | openshift-monitoring | kubelet | node-exporter-x72zx | Created | Created container init-textfile
10:47:05 | openshift-monitoring | kubelet | node-exporter-x72zx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" in 8.64491569s
10:47:05 (x2) | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:05 (x2) | openshift-image-registry | kubelet | node-ca-lg2rm | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:05 (x2) | openshift-multus | kubelet | multus-vrcmx | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:05 | openshift-image-registry | kubelet | node-ca-2qml7 | Created | Created container node-ca
10:47:06 | openshift-monitoring | kubelet | node-exporter-x72zx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:47:06 | openshift-cluster-node-tuning-operator | kubelet | tuned-97fwx | Created | Created container tuned
10:47:06 | openshift-cluster-node-tuning-operator | kubelet | tuned-97fwx | Started | Started container tuned
10:47:06 | openshift-multus | kubelet | multus-kr75l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398"
10:47:06 | openshift-machine-config-operator | kubelet | machine-config-daemon-2xgfc | Started | Started container oauth-proxy
10:47:06 (x2) | openshift-sdn | kubelet | sdn-2ff5l | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:06 | openshift-sdn | kubelet | sdn-2ff5l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 1.508702187s
10:47:06 | openshift-sdn | kubelet | ovs-8mdrj | Started | Started container openvswitch
10:47:06 | openshift-monitoring | kubelet | node-exporter-x72zx | Created | Created container node-exporter
10:47:06 | openshift-multus | kubelet | multus-kr75l | Started | Started container multus-binary-copy
10:47:06 | openshift-monitoring | kubelet | node-exporter-x72zx | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:47:06 | openshift-monitoring | kubelet | node-exporter-x72zx | Started | Started container init-textfile
10:47:06 | openshift-sdn | kubelet | sdn-9n262 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:47:06 | openshift-sdn | kubelet | sdn-9n262 | Started | Started container sdn
10:47:06 (x2) | openshift-sdn | kubelet | sdn-2ff5l | Failed | Error: services have not yet been read at least once, cannot construct envvars
10:47:06 | openshift-monitoring | kubelet | node-exporter-x72zx | Started | Started container node-exporter
10:47:07 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:47:07 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:47:08 | openshift-monitoring | kubelet | node-exporter-x72zx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 2.482987948s
10:47:08 | openshift-sdn | kubelet | sdn-9n262 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 2.945189579s
10:47:09 | openshift-sdn | kubelet | sdn-9n262 | Started | Started container kube-rbac-proxy
10:47:09 | openshift-monitoring | kubelet | node-exporter-x72zx | Created | Created container kube-rbac-proxy
10:47:09 | openshift-monitoring | kubelet | node-exporter-x72zx | Started | Started container kube-rbac-proxy
10:47:09 | openshift-sdn | kubelet | sdn-9n262 | Created | Created container kube-rbac-proxy
10:47:12 | openshift-multus | kubelet | multus-kr75l | Started | Started container cni-plugins
10:47:12 | openshift-multus | kubelet | multus-kr75l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398" in 5.806090359s
10:47:12 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:47:12 | openshift-multus | kubelet | multus-kr75l | Created | Created container cni-plugins
10:47:12 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:47:13 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:47:13 | openshift-multus | kubelet | multus-kr75l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9"
10:47:14 | openshift-multus | kubelet | multus-kr75l | Started | Started container routeoverride-cni
10:47:14 | openshift-multus | kubelet | multus-kr75l | Created | Created container routeoverride-cni
10:47:14 | openshift-multus | kubelet | multus-kr75l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9" in 997.966181ms
10:47:14 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:47:15 | openshift-multus | kubelet | multus-kr75l | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754"
10:47:16 (x2) | openshift-image-registry | kubelet | node-ca-lg2rm | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" already present on machine
10:47:16 | openshift-image-registry | kubelet | node-ca-lg2rm | Created | Created container node-ca
10:47:16 | openshift-image-registry | kubelet | node-ca-lg2rm | Started | Started container node-ca
10:47:17 | openshift-monitoring | kubelet | node-exporter-tg54f | Started | Started container node-exporter
10:47:17 | openshift-monitoring | kubelet | node-exporter-tg54f | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:17 | openshift-monitoring | kubelet | node-exporter-tg54f | Started | Started container init-textfile
10:47:17 | openshift-monitoring | kubelet | node-exporter-tg54f | Created | Created container init-textfile
10:47:17 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:47:17 | openshift-sdn | kubelet | ovs-cb7xs | Created | Created container openvswitch
10:47:17 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
10:47:17 (x2) | openshift-monitoring | kubelet | node-exporter-tg54f | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:47:17 | openshift-monitoring | kubelet | node-exporter-tg54f | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:47:17 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
10:47:17 | openshift-sdn | kubelet | ovs-cb7xs | Started | Started container openvswitch
10:47:17 | openshift-monitoring | kubelet | node-exporter-tg54f | Created | Created container node-exporter
10:47:17 (x2) | openshift-sdn | kubelet | ovs-cb7xs | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" already present on machine
10:47:18 | openshift-multus | kubelet | multus-kr75l | Started | Started container whereabouts-cni-bincopy
10:47:18 | openshift-sdn | kubelet | sdn-2ff5l | Started | Started container sdn
10:47:18 | openshift-sdn | kubelet | sdn-2ff5l | Created | Created container sdn
10:47:18 | openshift-sdn | kubelet | sdn-2ff5l | Started | Started container kube-rbac-proxy
10:47:18 | openshift-monitoring | kubelet | node-exporter-tg54f | Started | Started container kube-rbac-proxy
10:47:18 (x2) | openshift-sdn | kubelet | sdn-2ff5l | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" already present on machine
10:47:18 | default | machineconfigdaemon | worker-2.case03116926.lab.upshift.rdu2.redhat.com | NodeDone | Setting node worker-2.case03116926.lab.upshift.rdu2.redhat.com, currentConfig rendered-worker-9cba68d1b9fde6c79c41dcb10dfd7e0d to Done
10:47:18 (x2) | openshift-cluster-node-tuning-operator | kubelet | tuned-2kbh9 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" already present on machine
10:47:18 | openshift-multus | kubelet | multus-kr75l | Created | Created container whereabouts-cni-bincopy
10:47:18 | openshift-monitoring | kubelet | node-exporter-tg54f | Created | Created container kube-rbac-proxy
10:47:18 | openshift-cluster-node-tuning-operator | kubelet | tuned-2kbh9 | Created | Created container tuned
10:47:18 (x2) | openshift-sdn | kubelet | sdn-2ff5l | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:18 | openshift-multus | kubelet | multus-kr75l | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" in 2.807013463s
10:47:18 | openshift-cluster-node-tuning-operator | kubelet | tuned-2kbh9 | Started | Started container tuned
10:47:18 | openshift-sdn | kubelet | sdn-2ff5l | Created | Created container kube-rbac-proxy
10:47:19 | openshift-multus | kubelet | multus-kr75l | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" already present on machine
10:47:19 | openshift-multus | kubelet | multus-kr75l | Created | Created container whereabouts-cni
10:47:19 | openshift-multus | kubelet | multus-kr75l | Started | Started container whereabouts-cni
10:47:19 (x3) | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:47:19 | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Created | Created container machine-config-daemon
10:47:19 | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Created | Created container oauth-proxy
10:47:19 | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Started | Started container oauth-proxy
10:47:19 | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Started | Started container machine-config-daemon
10:47:19 (x2) | openshift-machine-config-operator | kubelet | machine-config-daemon-5qfjm | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:20 | openshift-multus | kubelet | multus-vrcmx | Started | Started container multus-binary-copy
10:47:20 (x14) | openshift-multus | kubelet | network-metrics-daemon-m86r5 | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:47:20 | openshift-multus | kubelet | multus-kr75l | Started | Started container kube-multus
10:47:20 | openshift-multus | kubelet | multus-vrcmx | Created | Created container multus-binary-copy
10:47:20 | default | openshift-sdn | worker-0.case03116926.lab.upshift.rdu2.redhat.com | Starting | openshift-sdn done initializing node networking.
10:47:20 (x2) | openshift-multus | kubelet | multus-vrcmx | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:47:20 | openshift-multus | kubelet | multus-kr75l | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:47:20 | openshift-multus | kubelet | multus-kr75l | Created | Created container kube-multus
10:47:20 (x14) | openshift-dns | kubelet | dns-default-zwxfh | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:47:20 | openshift-multus | kubelet | multus-vrcmx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398"
10:47:21 | default | machineconfigdaemon | worker-0.case03116926.lab.upshift.rdu2.redhat.com | NodeDone | Setting node worker-0.case03116926.lab.upshift.rdu2.redhat.com, currentConfig rendered-worker-9cba68d1b9fde6c79c41dcb10dfd7e0d to Done
10:47:24 | openshift-dns | multus | dns-default-zwxfh | AddedInterface | Add eth0 [10.131.0.2/23]
10:47:24 | openshift-multus | multus | network-metrics-daemon-m86r5 | AddedInterface | Add eth0 [10.131.0.3/23]
10:47:25 | openshift-cluster-node-tuning-operator | kubelet | tuned-dqzh5 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2"
10:47:25 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Started | Started container machine-config-daemon
10:47:25 | openshift-dns | kubelet | dns-default-zwxfh | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf"
10:47:25 | openshift-multus | kubelet | multus-t6jxt | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe"
10:47:25 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44af0373c6d9ac03c0453fd50cc78ee9b2d8782ed3fccc5bf4e510fd0937d6a9" already present on machine
10:47:25 | openshift-sdn | kubelet | sdn-5vdnk | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:47:25 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1"
10:47:25 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd"
10:47:25 | openshift-image-registry | kubelet | node-ca-xzbkf | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5"
10:47:25 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Created | Created container machine-config-daemon
10:47:25 | openshift-monitoring | kubelet | node-exporter-bvmqv | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708"
10:47:25 | openshift-sdn | kubelet | ovs-dpkmc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e"
10:47:26 | openshift-multus | kubelet | multus-vrcmx | Started | Started container cni-plugins
10:47:26 | openshift-multus | kubelet | multus-vrcmx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398" in 5.496319029s
10:47:26 | openshift-multus | kubelet | multus-vrcmx | Created | Created container cni-plugins
10:47:26 | openshift-multus | kubelet | multus-vrcmx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9"
10:47:27 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1" in 2.247796299s
10:47:27 | openshift-dns | kubelet | dns-default-zwxfh | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf" in 2.438869265s
10:47:27 | openshift-dns | kubelet | dns-default-zwxfh | Created | Created container dns
10:47:27 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:27 | openshift-multus | kubelet | multus-vrcmx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9" in 968.038781ms
10:47:27 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Created | Created container kube-rbac-proxy
10:47:27 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Started | Started container network-metrics-daemon
10:47:27 | openshift-dns | kubelet | dns-default-zwxfh | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:27 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Created | Created container network-metrics-daemon
10:47:27 | openshift-dns | kubelet | dns-default-zwxfh | Started | Started container dns
10:47:27 | openshift-dns | kubelet | dns-default-zwxfh | Created | Created container kube-rbac-proxy
10:47:27 | openshift-multus | kubelet | multus-vrcmx | Created | Created container routeoverride-cni
10:47:28 | openshift-multus | kubelet | multus-vrcmx | Started | Started container routeoverride-cni
10:47:28 | openshift-multus | kubelet | multus-vrcmx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754"
10:47:28 | openshift-monitoring | kubelet | node-exporter-bvmqv | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" in 3.226111214s
10:47:28 | openshift-dns | kubelet | dns-default-zwxfh | Started | Started container kube-rbac-proxy
10:47:28 | openshift-dns | kubelet | dns-default-zwxfh | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d"
10:47:28 | openshift-multus | kubelet | network-metrics-daemon-m86r5 | Started | Started container kube-rbac-proxy
10:47:30 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready"),Progressing message changed from "NodeInstallerProgressing: 1 nodes are at revision 0; 1 nodes are at revision 5; 1 nodes are at revision 6" to "NodeInstallerProgressing: 1 nodes are at revision 5; 2 nodes are at revision 6",Available message changed from "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 1 nodes are at revision 5; 1 nodes are at revision 6" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 5; 2 nodes are at revision 6"
10:47:30 (x2) | openshift-etcd-operator | openshift-cluster-etcd-operator-etcd-client | etcd-operator | UnhealthyEtcdMember | unhealthy members: master-1.case03116926.lab.upshift.rdu2.redhat.com
10:47:30 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 3 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)"
10:47:30 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 6 because static pod is ready
10:47:30 | openshift-dns | kubelet | dns-default-zwxfh | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" in 2.874776417s
10:47:31 | openshift-marketplace | taint-controller | redhat-marketplace-xm8bs | TaintManagerEviction | Cancelling deletion of Pod openshift-marketplace/redhat-marketplace-xm8bs
10:47:31 | openshift-dns | kubelet | dns-default-zwxfh | Started | Started container dns-node-resolver
10:47:31 | openshift-dns | kubelet | dns-default-zwxfh | Created | Created container dns-node-resolver
10:47:31 | openshift-multus | kubelet | multus-vrcmx | Started | Started container whereabouts-cni
10:47:31 | openshift-monitoring | taint-controller | prometheus-adapter-5fdd4bd886-bzs9k | TaintManagerEviction | Cancelling deletion of Pod openshift-monitoring/prometheus-adapter-5fdd4bd886-bzs9k
10:47:31 | openshift-kube-storage-version-migrator | taint-controller | migrator-85c499d985-hznj9 | TaintManagerEviction | Cancelling deletion of Pod openshift-kube-storage-version-migrator/migrator-85c499d985-hznj9
10:47:31 | openshift-multus | kubelet | multus-vrcmx | Created | Created container whereabouts-cni-bincopy
10:47:31 | openshift-marketplace | taint-controller | redhat-operators-hr4dq | TaintManagerEviction | Cancelling deletion of Pod openshift-marketplace/redhat-operators-hr4dq
10:47:31 | openshift-monitoring | taint-controller | kube-state-metrics-6c8b757584-2b2dg | TaintManagerEviction | Cancelling deletion of Pod openshift-monitoring/kube-state-metrics-6c8b757584-2b2dg
10:47:31 | openshift-ingress | kubelet | router-default-5b76b97bbb-ltvlc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:570ef2ff0e37e0933d0f9bbffd82bf3d119e96dfa0eb9c3f11ea9892c5d68384"
10:47:31 | openshift-marketplace | taint-controller | community-operators-mprwd | TaintManagerEviction | Cancelling deletion of Pod openshift-marketplace/community-operators-mprwd
10:47:31 | openshift-monitoring | taint-controller | openshift-state-metrics-6d99649c49-4b6cb | TaintManagerEviction | Cancelling deletion of Pod openshift-monitoring/openshift-state-metrics-6d99649c49-4b6cb
10:47:31 | openshift-multus | kubelet | multus-vrcmx | Started | Started container whereabouts-cni-bincopy
10:47:31 | openshift-marketplace | taint-controller | certified-operators-p45rl | TaintManagerEviction | Cancelling deletion of Pod openshift-marketplace/certified-operators-p45rl
10:47:31 | openshift-multus | kubelet | multus-vrcmx | Created | Created container whereabouts-cni
10:47:31 | openshift-monitoring | taint-controller | telemeter-client-57c59fc45f-gsqrj | TaintManagerEviction | Cancelling deletion of Pod openshift-monitoring/telemeter-client-57c59fc45f-gsqrj
10:47:31 | openshift-ingress | taint-controller | router-default-5b76b97bbb-ltvlc | TaintManagerEviction | Cancelling deletion of Pod openshift-ingress/router-default-5b76b97bbb-ltvlc
10:47:31 | openshift-multus | kubelet | multus-vrcmx | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" already present on machine
10:47:31 | openshift-multus | kubelet | multus-vrcmx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" in 2.70987094s
10:47:32 | openshift-marketplace | multus | community-operators-mprwd | AddedInterface | Add eth0 [10.131.0.8/23]
10:47:32 | openshift-kube-storage-version-migrator | multus | migrator-85c499d985-hznj9 | AddedInterface | Add eth0 [10.131.0.5/23]
10:47:32 | openshift-monitoring | multus | prometheus-adapter-5fdd4bd886-bzs9k | AddedInterface | Add eth0 [10.131.0.9/23]
10:47:32 | openshift-marketplace | kubelet | community-operators-mprwd | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
10:47:32 | openshift-multus | kubelet | multus-vrcmx | Created | Created container kube-multus
10:47:32 | openshift-multus | kubelet | multus-vrcmx | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:47:32 | openshift-marketplace | multus | certified-operators-p45rl | AddedInterface | Add eth0 [10.131.0.7/23]
10:47:32 (x47) | openshift-authentication-operator | cluster-authentication-operator | authentication-operator | OAuthRouteCheck | route status does not have host address
10:47:33 | openshift-monitoring | multus | telemeter-client-57c59fc45f-gsqrj | AddedInterface | Add eth0 [10.131.0.11/23]
10:47:33 | openshift-marketplace | multus | redhat-operators-hr4dq | AddedInterface | Add eth0 [10.131.0.12/23]
10:47:33 | openshift-multus | kubelet | multus-t6jxt | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" in 8.338679783s
10:47:33 | openshift-cluster-node-tuning-operator | kubelet | tuned-dqzh5 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:577393e4d4d8b953c80ca4a0536740f0880cebe22f2e65cce35d8f53f60f94d2" in 8.401820999s
10:47:33 | openshift-marketplace | multus | redhat-marketplace-xm8bs | AddedInterface | Add eth0 [10.131.0.6/23]
10:47:33 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" in 8.256722066s
10:47:33 | openshift-sdn | kubelet | sdn-5vdnk | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.584959707s
10:47:33 | openshift-monitoring | multus | kube-state-metrics-6c8b757584-2b2dg | AddedInterface | Add eth0 [10.131.0.10/23]
10:47:33 | openshift-marketplace | kubelet | certified-operators-p45rl | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
10:47:33 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-bzs9k | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c479c728c1a03595cc10e3f3ab3e400f588dc662fb0b7b81ac9b62eddee3dd80"
10:47:33 | openshift-kube-storage-version-migrator | kubelet | migrator-85c499d985-hznj9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b268776e5b0b054bcac9ca4b57ce629c6d852d12792cf72d8c3adc2cb88037df"
10:47:33 | openshift-monitoring | multus | openshift-state-metrics-6d99649c49-4b6cb | AddedInterface | Add eth0 [10.131.0.4/23]
10:47:33 | openshift-sdn | kubelet | ovs-dpkmc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:980d20b31cdcd9db0a94e393b53799985ea399bbe8dbd552bf2e27e32cf0be7e" in 8.623181694s
10:47:33 | openshift-image-registry | kubelet | node-ca-xzbkf | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" in 8.347625329s
10:47:34 | openshift-sdn | kubelet | sdn-5vdnk | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:47:34 | openshift-sdn | kubelet | ovs-dpkmc | Started | Started container openvswitch
10:47:34 | openshift-image-registry | kubelet | node-ca-xzbkf | Started | Started container node-ca
10:47:34 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:47:34 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:34 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Started | Started container oauth-proxy
10:47:34 | openshift-machine-config-operator | kubelet | machine-config-daemon-8ttrt | Created | Created container oauth-proxy
10:47:34 (x2) | openshift-etcd | endpoint-controller | etcd | FailedToUpdateEndpoint | Failed to update endpoint openshift-etcd/etcd: Operation cannot be fulfilled on endpoints "etcd": the object has been modified; please apply your changes to the latest version and try again
10:47:34 | openshift-multus | kubelet | multus-t6jxt | Created | Created container multus-binary-copy
10:47:34 | openshift-multus | kubelet | multus-t6jxt | Started | Started container multus-binary-copy
10:47:34 | openshift-image-registry | kubelet | node-ca-xzbkf | Created | Created container node-ca
10:47:34 | openshift-cluster-node-tuning-operator | kubelet | tuned-dqzh5 | Created | Created container tuned
10:47:34 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:47:34 | openshift-sdn | kubelet | sdn-5vdnk | Created | Created container sdn
10:47:34 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Created | Created container kube-rbac-proxy-main
10:47:34 | openshift-sdn | kubelet | sdn-5vdnk | Started | Started container sdn
10:47:34 | openshift-ingress | kubelet | router-default-5b76b97bbb-ltvlc | Created | Created container router
10:47:34 | openshift-ingress | kubelet | router-default-5b76b97bbb-ltvlc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:570ef2ff0e37e0933d0f9bbffd82bf3d119e96dfa0eb9c3f11ea9892c5d68384" in 2.264856786s
10:47:34 | openshift-monitoring | kubelet | node-exporter-bvmqv | Created | Created container init-textfile
10:47:34 | openshift-cluster-node-tuning-operator | kubelet | tuned-dqzh5 | Started | Started container tuned
10:47:34 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-6-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:47:34 | openshift-monitoring | kubelet | node-exporter-bvmqv | Started | Started container init-textfile
10:47:34 | openshift-sdn | kubelet | ovs-dpkmc | Created | Created container openvswitch
10:47:35 | openshift-sdn | kubelet | sdn-5vdnk | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 1.565641948s
10:47:35 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Degraded message changed from "" to "RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com/health): Get \"https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com/health\": EOF",Progressing message changed from "DefaultRouteSyncProgressing: route \"console\" is not available at canonical host []\nOAuthClientSyncProgressing: route \"console\" is not available at canonical host []" to "OAuthClientSyncProgressing: route \"console\" is not available at canonical host []"
10:47:35 | openshift-monitoring | kubelet | node-exporter-bvmqv | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9025a18059c20d9b2e4214eb8852784961d00fd68e8a938b422a34b84632e708" already present on machine
10:47:35 | openshift-monitoring | kubelet | node-exporter-bvmqv | Created | Created container node-exporter
10:47:35 | openshift-monitoring | kubelet | node-exporter-bvmqv | Started | Started container node-exporter
10:47:35 | openshift-marketplace | kubelet | redhat-marketplace-xm8bs | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
10:47:35 | openshift-monitoring | kubelet | node-exporter-bvmqv | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" in 396.134234ms
10:47:35 (x17) | openshift-etcd | kubelet | etcd-quorum-guard-58669bc4d5-lz6pp | Unhealthy | Readiness probe failed:
10:47:35 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f1862f267aba64b61a7b61b2035513c17bf047adb602a4f09f201e2d47199f4"
10:47:35 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:35 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9f4b1e422e2599f4af9b47b175738aa7110eb94dc886b250b3ce7c4941b8005e"
10:47:35 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: route status does not have host address\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:47:35 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:35 | openshift-ingress | kubelet | router-default-5b76b97bbb-ltvlc | Started | Started container router
10:47:35 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:47:35 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-ensure-env-vars
10:47:35 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-resources-copy
10:47:35 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-resources-copy
10:47:35 | default | openshift-sdn | worker-1.case03116926.lab.upshift.rdu2.redhat.com | Starting | openshift-sdn done initializing node networking.
10:47:35 | openshift-multus | kubelet | multus-t6jxt | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398"
10:47:35 | openshift-monitoring | kubelet | node-exporter-bvmqv | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874"
10:47:35 | openshift-marketplace | kubelet | redhat-operators-hr4dq | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
10:47:35 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Started | Started container kube-rbac-proxy-main
10:47:35 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-ensure-env-vars
10:47:36 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcdctl
10:47:36 | openshift-monitoring | kubelet | node-exporter-bvmqv | Started | Started container kube-rbac-proxy
10:47:36 (x22) | openshift-multus | kubelet | network-metrics-daemon-pkjvx | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:47:36 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcdctl
10:47:36 | openshift-sdn | kubelet | sdn-5vdnk | Created | Created container kube-rbac-proxy
10:47:36 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:47:36 | openshift-monitoring | kubelet | node-exporter-bvmqv | Created | Created container kube-rbac-proxy
10:47:36 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:47:36 (x22) | openshift-dns | kubelet | dns-default-xbgq5 | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:47:36 | openshift-sdn | kubelet | sdn-5vdnk | Started | Started container kube-rbac-proxy
10:47:37 | openshift-monitoring | statefulset-controller | alertmanager-main | SuccessfulCreate | create Pod alertmanager-main-1 in StatefulSet alertmanager-main successful
10:47:37 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 5 to 6 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 5 is the oldest
10:47:37 | openshift-kube-scheduler | multus | revision-pruner-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.37/23]
10:47:37 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd-metrics
10:47:37 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff68bba6a0c20745df387a2000e21e46bbf21af46f995d0315039f0f069977fb" already present on machine
10:47:37 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd
10:47:37 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container etcd
10:47:37 | openshift-console | replicaset-controller | console-7fdcf89666 | SuccessfulCreate | Created pod: console-7fdcf89666-hh29k
10:47:37 | openshift-monitoring | statefulset-controller | alertmanager-main | SuccessfulCreate | create Pod alertmanager-main-2 in StatefulSet alertmanager-main successful
10:47:37 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Degraded message changed from "RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com/health): Get \"https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com/health\": EOF" to "RouteHealthDegraded: route not yet available, https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com/health returns '503 Service Unavailable'"
10:47:37 | openshift-console | replicaset-controller | console-7fdcf89666 | SuccessfulCreate | Created pod: console-7fdcf89666-fslmv
10:47:37 | openshift-console-operator | console-operator | console-operator | DeploymentCreated | Created Deployment.apps/console -n openshift-console because it was missing
10:47:37 | openshift-monitoring | statefulset-controller | alertmanager-main | SuccessfulCreate | create Pod alertmanager-main-0 in StatefulSet alertmanager-main successful
10:47:37 | openshift-console | deployment-controller | console | ScalingReplicaSet | Scaled up replica set console-7fdcf89666 to 2
10:47:37 | openshift-etcd | kubelet | etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container etcd-metrics
10:47:38 | openshift-authentication-operator | cluster-authentication-operator-metadata-controller-metadatacontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/v4-0-config-system-metadata -n openshift-authentication because it was missing
10:47:38 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ "admission": map[string]interface{}{"pluginConfig": map[string]interface{}{"network.openshift.io/ExternalIPRanger": map[string]interface{}{"configuration": map[string]interface{}{"allowIngressIP": bool(false), "apiVersion": string("network.openshift.io/v1"), "kind": string("ExternalIPRangerAdmissionConfig")}}, "network.openshift.io/RestrictedEndpointsAdmission": map[string]interface{}{"configuration": map[string]interface{}{"apiVersion": string("network.openshift.io/v1"), "kind": string("RestrictedEndpointsAdmissionConfig"), "restrictedCIDRs": []interface{}{string("10.128.0.0/14"), string("172.30.0.0/16")}}}}}, "apiServerArguments": map[string]interface{}{"audit-policy-file": []interface{}{string("/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-audit-policies/default.yaml")}, "etcd-servers": []interface{}{string("https://10.0.94.119:2379"), string("https://10.0.94.219:2379"), string("https://10.0.95.76:2379"), string("https://localhost:2379")}, "feature-gates": []interface{}{string("APIPriorityAndFairness=true"), string("RotateKubeletServerCertificate=true"), string("SupportPodPidsLimit=true"), string("NodeDisruptionExclusion=true"), string("ServiceNodeExclusion=true"), string("SCTPSupport=true"), string("LegacyNodeRoleBehavior=false")}}, + "authConfig": map[string]interface{}{ + "oauthMetadataFile": string("/etc/kubernetes/static-pod-resources/configmaps/oauth-metadata/oauthMetadata"), + }, "corsAllowedOrigins": []interface{}{string(`//127\.0\.0\.1(:|$)`), string("//localhost(:|$)")}, "servicesSubnet": string("172.30.0.0/16"), "servingInfo": map[string]interface{}{"bindAddress": string("0.0.0.0:6443"), "bindNetwork": string("tcp4"), "cipherSuites": []interface{}{string("TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"), string("TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"), string("TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"), string("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"), string("TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"), string("TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256")}, "minTLSVersion": string("VersionTLS12"), "namedCertificates": []interface{}{map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key")}}}, }
10:47:38 | openshift-monitoring | kubelet | alertmanager-main-1 | FailedMount | MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" : failed to sync secret cache: timed out waiting for the condition
10:47:38 | openshift-monitoring | kubelet | alertmanager-main-2 | FailedMount | MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" : failed to sync secret cache: timed out waiting for the condition
10:47:38 | openshift-monitoring | kubelet | alertmanager-main-0 | FailedMount | MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" : failed to sync secret cache: timed out waiting for the condition
10:47:38 | openshift-console-operator | console-operator | console-operator | ConfigMapUpdated | Updated ConfigMap/console-public -n openshift-config-managed: cause by changes in data.consoleURL
10:47:38 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty"
10:47:38 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pods \"etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com\" not found\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:47:38 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing message changed from "OAuthClientSyncProgressing: route \"console\" is not available at canonical host []" to "SyncLoopRefreshProgressing: Changes made during sync updates, additional sync expected.",Available changed from Unknown to False ("DeploymentAvailable: 0 pods available for console deployment")
10:47:38 | openshift-monitoring | kubelet | alertmanager-main-0 | FailedMount | MountVolume.SetUp failed for volume "alertmanager-main-token-jfgs5" : failed to sync secret cache: timed out waiting for the condition
10:47:39 | openshift-console | multus | console-7fdcf89666-hh29k | AddedInterface | Add eth0 [10.128.0.38/23]
10:47:39 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:47:39 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:47:39 | openshift-console | kubelet | console-7fdcf89666-fslmv | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d"
10:47:39 | openshift-console | multus | console-7fdcf89666-fslmv | AddedInterface | Add eth0 [10.129.0.25/23]
10:47:39 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:47:39 | openshift-console | kubelet | console-7fdcf89666-hh29k | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d"
10:47:40 | openshift-monitoring | deployment-controller | thanos-querier | ScalingReplicaSet | Scaled up replica set thanos-querier-5c8f998999 to 2
10:47:40 | openshift-dns | multus | dns-default-xbgq5 | AddedInterface | Add eth0 [10.128.2.3/23]
10:47:40 | openshift-dns | kubelet | dns-default-xbgq5 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf"
10:47:40 | openshift-multus | kubelet | network-metrics-daemon-pkjvx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1"
10:47:40 | openshift-multus | multus | network-metrics-daemon-pkjvx | AddedInterface | Add eth0 [10.128.2.2/23]
10:47:40 | openshift-monitoring | replicaset-controller | thanos-querier-5c8f998999 | SuccessfulCreate | Created pod: thanos-querier-5c8f998999-bwnht
10:47:40 | openshift-monitoring | replicaset-controller | thanos-querier-5c8f998999 | SuccessfulCreate | Created pod: thanos-querier-5c8f998999-vrvtc
10:47:41 | openshift-authentication | replicaset-controller | oauth-openshift-765c977d67 | SuccessfulCreate | Created pod: oauth-openshift-765c977d67-lh7jg
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-resource-sync-controller-resourcesynccontroller | authentication-operator | ConfigMapCreated | Created ConfigMap/oauth-openshift -n openshift-config-managed because it was missing
10:47:41 | openshift-ingress | taint-controller | router-default-5b76b97bbb-jp2g9 | TaintManagerEviction | Cancelling deletion of Pod openshift-ingress/router-default-5b76b97bbb-jp2g9
10:47:41 | openshift-multus | kubelet | multus-t6jxt | Created | Created container cni-plugins
10:47:41 | openshift-ingress | kubelet | router-default-5b76b97bbb-jp2g9 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:570ef2ff0e37e0933d0f9bbffd82bf3d119e96dfa0eb9c3f11ea9892c5d68384"
10:47:41 | openshift-multus | kubelet | multus-t6jxt | Started | Started container cni-plugins
10:47:41 | openshift-multus | kubelet | multus-t6jxt | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9290248773de78bc750709a850ef66bd0d32a64ab1ed5228d344557e8bade398" in 6.113447976s
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1" to "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1"
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:41 | openshift-monitoring | taint-controller | thanos-querier-5c8f998999-bwnht | TaintManagerEviction | Cancelling deletion of Pod openshift-monitoring/thanos-querier-5c8f998999-bwnht
10:47:41 | openshift-kube-controller-manager | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_f72bf8ca-4e18-44bb-b968-44542c58abf8 became leader
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1" to "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1"
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-deployment-controller-deployment | authentication-operator | DeploymentCreated | Created Deployment.apps/oauth-openshift -n openshift-authentication because it was missing
10:47:41 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled up replica set oauth-openshift-765c977d67 to 2
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)",Progressing message changed from "OAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1" to "OAuthVersionDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1",Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:41 | openshift-monitoring | deployment-controller | grafana | ScalingReplicaSet | Scaled up replica set grafana-c4597b685 to 1
10:47:41 | openshift-authentication | replicaset-controller | oauth-openshift-765c977d67 | SuccessfulCreate | Created pod: oauth-openshift-765c977d67-9ztxx
10:47:41 | openshift-monitoring | replicaset-controller | grafana-c4597b685 | SuccessfulCreate | Created pod: grafana-c4597b685-hvrqk
10:47:41 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthServerDeploymentDegraded: deployments.apps \"oauth-openshift\" not found\nOAuthServerRouteDegraded: Route is not available at canonical host oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com: route status ingress is empty" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nOAuthVersionDeploymentDegraded: Unable to get OAuth server deployment: deployment.apps \"oauth-openshift\" not found\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)",Progressing changed from False to True ("OAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1"),Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nWellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:42 (x2) | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObserveConsoleURL | assetPublicURL changed from to https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com
10:47:42 (x2) | openshift-authentication-operator | cluster-authentication-operator-config-observer-configobserver | authentication-operator | ObservedConfigChanged | Writing updated section ("oauthServer") of observed config: "\u00a0\u00a0map[string]interface{}{\n\u00a0\u00a0\t\"corsAllowedOrigins\": []interface{}{string(`//127\\.0\\.0\\.1(:|$)`), string(\"//localhost(:|$)\")},\n\u00a0\u00a0\t\"oauthConfig\": map[string]interface{}{\n-\u00a0\t\t\"assetPublicURL\": string(\"\"),\n+\u00a0\t\t\"assetPublicURL\": string(\"https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com\"),\n\u00a0\u00a0\t\t\"loginURL\": string(\"https://api.case03116926.lab.upshift.rdu2.redhat.com:6443\"),\n\u00a0\u00a0\t\t\"templates\": map[string]interface{}{\"error\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/errors.html\"), \"login\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/login.html\"), \"providerSelection\": string(\"/var/config/system/secrets/v4-0-config-system-ocp-branding-template/providers.html\")},\n\u00a0\u00a0\t\t\"tokenConfig\": map[string]interface{}{\"accessTokenMaxAgeSeconds\": float64(86400), \"authorizeTokenMaxAgeSeconds\": float64(300)},\n\u00a0\u00a0\t},\n\u00a0\u00a0\t\"servingInfo\": map[string]interface{}{\"cipherSuites\": []interface{}{string(\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\"), string(\"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\"), string(\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\"), string(\"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\"), string(\"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\"), string(\"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\")}, \"minTLSVersion\": string(\"VersionTLS12\"), \"namedCertificates\": []interface{}{map[string]interface{}{\"certFile\": string(\"/var/config/system/secrets/v4-0-config-system-router-certs/apps.case03116926.lab.upshift.rdu2.redhat.com\"), \"keyFile\": string(\"/var/config/system/secrets/v4-0-config-system-router-certs/apps.case03116926.lab.upshift.rdu2.redhat.com\"), \"names\": []interface{}{string(\"*.apps.case03116926.lab.upshift.rdu2.redhat.com\")}}}},\n\u00a0\u00a0\t\"volumesToMount\": map[string]interface{}{\"identityProviders\": string(\"{}\")},\n\u00a0\u00a0}\n"
10:47:42 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd-metrics\" is not ready: PodInitializing: \nStaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcdctl\" is not ready: PodInitializing: \nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:47:37 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:47:42 | openshift-monitoring | multus | thanos-querier-5c8f998999-bwnht | AddedInterface | Add eth0 [10.128.2.4/23]
10:47:42 | openshift-multus | kubelet | multus-t6jxt | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9"
10:47:43 | openshift-console | kubelet | console-7fdcf89666-hh29k | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d" in 4.107380862s
10:47:43 | openshift-multus | kubelet | multus-t6jxt | Started | Started container routeoverride-cni
10:47:43 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap \"oauth-openshift\" not found (check authentication operator, it is supposed to create this)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:47:43 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:47:43 | openshift-multus | kubelet | network-metrics-daemon-pkjvx | Created | Created container network-metrics-daemon
10:47:43 | openshift-multus | kubelet | multus-t6jxt | Created | Created container routeoverride-cni
10:47:43 | openshift-dns | kubelet | dns-default-xbgq5 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf" in 2.665802994s
10:47:43 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5"
10:47:43 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:47:43 | openshift-multus | kubelet | multus-t6jxt | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3816ed95856807e1d3d7d6194adcb7a1df78972d512253d5b9b621fd395c0fe9" in 927.945857ms
10:47:43 | openshift-multus | kubelet | network-metrics-daemon-pkjvx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1" in 2.657543975s
10:47:43 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Created | Created container kube-rbac-proxy-self
10:47:43 | openshift-authentication | multus | oauth-openshift-765c977d67-9ztxx | AddedInterface | Add eth0 [10.129.0.26/23]
10:47:43 | openshift-dns | kubelet | dns-default-xbgq5 | Created | Created container dns
10:47:44 | openshift-console | kubelet | console-7fdcf89666-hh29k | Created | Created container console
10:47:44 | openshift-authentication | kubelet | oauth-openshift-765c977d67-9ztxx | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931"
10:47:44 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1" to "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1"
10:47:44 | openshift-kube-storage-version-migrator | kubelet | migrator-85c499d985-hznj9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b268776e5b0b054bcac9ca4b57ce629c6d852d12792cf72d8c3adc2cb88037df" in 10.776242362s
10:47:44 | openshift-multus | kubelet | multus-t6jxt | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754"
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:47:44 | openshift-console | kubelet | console-7fdcf89666-hh29k | Started | Started container console
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:47:44 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c032715c7f67b0dbf0f30b446aa7d29de9b3725b0a8d6fa58fe901c5b7c2c007"
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:47:44 | openshift-console | kubelet | console-7fdcf89666-fslmv | Started | Started container console
10:47:44 | openshift-authentication | kubelet | oauth-openshift-765c977d67-lh7jg | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931"
10:47:44 | openshift-console | kubelet | console-7fdcf89666-fslmv | Created | Created container console
10:47:44 | openshift-authentication | multus | oauth-openshift-765c977d67-lh7jg | AddedInterface | Add eth0 [10.128.0.39/23]
10:47:44 | openshift-monitoring | multus | alertmanager-main-2 | AddedInterface | Add eth0 [10.131.0.13/23]
10:47:44 | openshift-ingress | kubelet | router-default-5b76b97bbb-jp2g9 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:570ef2ff0e37e0933d0f9bbffd82bf3d119e96dfa0eb9c3f11ea9892c5d68384" in 3.175319321s
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:47:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:47:44 | openshift-authentication | replicaset-controller | oauth-openshift-d5d844698 | SuccessfulCreate | Created pod: oauth-openshift-d5d844698-kh5nv
10:47:44 | openshift-ingress | kubelet | router-default-5b76b97bbb-jp2g9 | Created | Created container router
10:47:44 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Started | Started container kube-rbac-proxy-self
10:47:44 | openshift-console | kubelet | console-7fdcf89666-fslmv | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d" in 4.473243071s
10:47:44 | openshift-ingress | kubelet | router-default-5b76b97bbb-jp2g9 | Started | Started container router
10:47:44 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled up replica set oauth-openshift-d5d844698 to 1
10:47:44 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 0 to match expected generation 1" to "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2"
10:47:45 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 4 triggered by "configmap/oauth-metadata has changed"
10:47:45 | openshift-kube-apiserver-operator | kube-apiserver-operator-resource-sync-controller-resourcesynccontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/oauth-metadata -n openshift-kube-apiserver because it was missing
10:47:45 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfaec558476cba803fcfe0fbae364aa5bedc9de51d412cf242c35319b9dbc0b6"
10:47:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:47:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:47:45 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | PodCreated | Created Pod/installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:47:45 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 2 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2" to "OAuthVersionDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2"
10:47:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:47:45 | openshift-monitoring | multus | grafana-c4597b685-hvrqk | AddedInterface | Add eth0 [10.128.2.5/23]
10:47:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:47:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:47:46 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Degraded message changed from "StaticPodsDegraded: pod/etcd-master-1.case03116926.lab.upshift.rdu2.redhat.com container \"etcd\" is running since 2022-03-08 10:47:37 +0000 UTC but not ready: unknown reason\nNodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found" to "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found"
10:47:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:47:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:47:46 | openshift-kube-scheduler | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_7eb4f86a-5976-47fd-9cb2-03150827a969 became leader
10:47:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:47:46 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2"
10:47:47 | openshift-authentication | kubelet | oauth-openshift-765c977d67-9ztxx | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931" in 2.627832801s
10:47:47 | openshift-monitoring | statefulset-controller | prometheus-k8s | SuccessfulCreate | create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s successful
10:47:47 | openshift-multus | kubelet | multus-t6jxt | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" in 3.077257071s
10:47:47 | openshift-authentication-operator | cluster-authentication-operator-payload-config-controller-payloadconfig | authentication-operator | ConfigMapUpdated | Updated ConfigMap/v4-0-config-system-cliconfig -n openshift-authentication: cause by changes in data.v4-0-config-system-cliconfig
10:47:47 | openshift-multus | kubelet | multus-t6jxt | Created | Created container whereabouts-cni-bincopy
10:47:47 | openshift-authentication | kubelet | oauth-openshift-765c977d67-lh7jg | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931" in 2.398567763s
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5" in 4.146962026s
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Started | Started container oauth-proxy
10:47:47 | openshift-authentication | kubelet | oauth-openshift-765c977d67-lh7jg | Created | Created container oauth-openshift
10:47:47 | openshift-authentication | kubelet | oauth-openshift-765c977d67-lh7jg | Started | Started container oauth-openshift
10:47:47 | openshift-multus | kubelet | multus-t6jxt | Started | Started container whereabouts-cni-bincopy
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Created | Created container oauth-proxy
10:47:47 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2"
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Started | Started container thanos-query
10:47:47 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Created | Created container thanos-query
10:47:47 | openshift-kube-scheduler | multus | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.27/23]
10:47:47 | openshift-authentication | multus | oauth-openshift-d5d844698-kh5nv | AddedInterface | Add eth0 [10.130.0.42/23]
10:47:47 | openshift-authentication | kubelet | oauth-openshift-d5d844698-kh5nv | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931"
10:47:47 | openshift-authentication | kubelet | oauth-openshift-765c977d67-9ztxx | Created | Created container oauth-openshift
10:47:48 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3"
10:47:48 | openshift-monitoring | statefulset-controller | prometheus-k8s | SuccessfulCreate | create Pod prometheus-k8s-1 in StatefulSet prometheus-k8s successful
10:47:48 | openshift-multus | kubelet | multus-t6jxt | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42aff16d4f81f800424c77d3200b5952f0ff3c58ca1ea01dbceb37ef25df0754" already present on machine
10:47:48 | openshift-multus | kubelet | multus-t6jxt | Created | Created container whereabouts-cni
10:47:48 | openshift-multus | kubelet | multus-t6jxt | Started | Started container whereabouts-cni
10:47:48 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 1 to match expected generation 2" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)"
10:47:48 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)"
10:47:48 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-4 -n openshift-kube-apiserver because it was missing
10:47:48 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)"
10:47:48 | openshift-authentication | kubelet | oauth-openshift-765c977d67-9ztxx | Started | Started container oauth-openshift
10:47:48 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Started | Started container kube-rbac-proxy
10:47:48 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Created | Created container kube-rbac-proxy
10:47:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-status-controller-statussyncer_etcd | etcd-operator | OperatorStatusChanged | Status for clusteroperator/etcd changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 3\nEtcdMembersProgressing: No unstarted etcd members found"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 2 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 3\nEtcdMembersAvailable: 3 members are available"
10:47:48 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because static pod is ready
10:47:49 | openshift-kube-scheduler | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:47:49 | openshift-multus | kubelet | multus-t6jxt | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f54f352668f99e4776c12a0b9477bf1910af43bc27f7d17cdf6d9dd494b408fe" already present on machine
10:47:49 | openshift-etcd-operator | openshift-cluster-etcd-operator-installer-controller | etcd-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-3 -n openshift-etcd: cause by changes in data.status
10:47:49 | openshift-multus | kubelet | multus-t6jxt | Started | Started container kube-multus
10:47:49 | openshift-multus | kubelet | multus-t6jxt | Created | Created container kube-multus
10:47:49 | openshift-kube-scheduler | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:47:49 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:47:50 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:50 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Created | Created container grafana-proxy
10:47:50 | openshift-kube-scheduler | static-pod-installer | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 6
10:47:50 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-bzs9k | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c479c728c1a03595cc10e3f3ab3e400f588dc662fb0b7b81ac9b62eddee3dd80" in 16.727364761s
10:47:50 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Started | Started container grafana-proxy
10:47:50 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler
10:47:50 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" in 2.641015173s
10:47:50 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b7f8a5c0c45e29642dda6d575de9b971c192935c3599470487fd0e8b4732be7"
10:47:50 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfaec558476cba803fcfe0fbae364aa5bedc9de51d412cf242c35319b9dbc0b6" in 4.713989882s
10:47:50 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-recovery-controller
10:47:50 | openshift-kube-scheduler | kubelet | installer-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:47:50 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Created | Created container grafana
10:47:50 | openshift-monitoring | kubelet | grafana-c4597b685-hvrqk | Started | Started container grafana
10:47:50 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-scheduler-cert-syncer
10:47:50 | openshift-authentication | kubelet | oauth-openshift-d5d844698-kh5nv | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931" in 2.793616866s
10:47:50 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-3 -n openshift-kube-apiserver: cause by changes in data.status
10:47:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)",Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.219:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:47:51 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e"
10:47:51 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Created | Created container prom-label-proxy
10:47:51 | openshift-kube-scheduler | default-scheduler | kube-scheduler | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com_988dec17-1268-40c9-88db-2de81fdffca8 became leader
10:47:51 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Started | Started container prom-label-proxy
10:47:51 | openshift-authentication | kubelet | oauth-openshift-d5d844698-kh5nv | Created | Created container oauth-openshift
10:47:51 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:51 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Started | Started container kube-rbac-proxy-rules
10:47:51 | openshift-authentication | kubelet | oauth-openshift-d5d844698-kh5nv | Started | Started container oauth-openshift
10:47:51 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-bwnht | Created | Created container kube-rbac-proxy-rules
10:47:51 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)"
10:47:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3" to "OAuthVersionDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3"
10:47:52 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled down replica set oauth-openshift-765c977d67 to 1
10:47:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3"
10:47:52 | openshift-authentication | replicaset-controller | oauth-openshift-765c977d67 | SuccessfulDelete | Deleted pod: oauth-openshift-765c977d67-9ztxx
10:47:52 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3"
10:47:52 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled up replica set oauth-openshift-667c95946d to 1
10:47:52 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f1862f267aba64b61a7b61b2035513c17bf047adb602a4f09f201e2d47199f4" in 17.616868141s
10:47:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is waiting in pending oauth-openshift-765c977d67-9ztxx pod,container is waiting in pending oauth-openshift-765c977d67-lh7jg pod,container is waiting in pending oauth-openshift-d5d844698-kh5nv pod)" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3"
10:47:52 | openshift-authentication | replicaset-controller | oauth-openshift-667c95946d | SuccessfulCreate | Created pod: oauth-openshift-667c95946d-9m5z4
10:47:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-9ztxx pod,container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3"
10:47:52 | openshift-authentication | kubelet | oauth-openshift-765c977d67-9ztxx | Killing | Stopping container oauth-openshift
10:47:53 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b7f8a5c0c45e29642dda6d575de9b971c192935c3599470487fd0e8b4732be7"
10:47:53 | openshift-marketplace | kubelet | certified-operators-p45rl | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 20.415936709s
10:47:53 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-nnfsr | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c479c728c1a03595cc10e3f3ab3e400f588dc662fb0b7b81ac9b62eddee3dd80" already present on machine
10:47:53 | openshift-marketplace | kubelet | redhat-marketplace-xm8bs | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 18.108556291s
10:47:53 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9f4b1e422e2599f4af9b47b175738aa7110eb94dc886b250b3ce7c4941b8005e" in 17.751108176s
10:47:53 (x16) | openshift-dns | kubelet | dns-default-fcp4w | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:47:53 | openshift-marketplace | kubelet | community-operators-mprwd | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 20.954308912s
10:47:53 | openshift-kube-storage-version-migrator | kubelet | migrator-85c499d985-hznj9 | Created | Created container migrator
10:47:53 | openshift-kube-storage-version-migrator | kubelet | migrator-85c499d985-hznj9 | Started | Started container migrator
10:47:53 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b7f8a5c0c45e29642dda6d575de9b971c192935c3599470487fd0e8b4732be7"
10:47:53 (x16) | openshift-multus | kubelet | network-metrics-daemon-xxx4f | NetworkNotReady | network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
10:47:53 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5"
10:47:53 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c032715c7f67b0dbf0f30b446aa7d29de9b3725b0a8d6fa58fe901c5b7c2c007" in 9.935469869s
10:47:53 | openshift-etcd-operator | openshift-cluster-etcd-operator-prunecontroller | etcd-operator | PodCreated | Created Pod/revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-etcd because it was missing
10:47:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-pod-4 -n openshift-kube-apiserver because it was missing
10:47:54 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Created | Created container kube-state-metrics
10:47:54 | openshift-marketplace | kubelet | redhat-operators-hr4dq | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 19.011345065s
10:47:54 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-bzs9k | Created | Created container prometheus-adapter
10:47:54 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-bzs9k | Started | Started container prometheus-adapter
10:47:54 (x2) | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-status-controller-statussyncer_kube-storage-version-migrator | kube-storage-version-migrator-operator | OperatorStatusChanged | Status for clusteroperator/kube-storage-version-migrator changed: Available changed from False to True ("")
10:47:54 | openshift-marketplace | kubelet | redhat-marketplace-xm8bs | Started | Started container registry-server
10:47:54 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Created | Created container telemeter-client
10:47:54 | openshift-marketplace | kubelet | redhat-marketplace-xm8bs | Created | Created container registry-server
10:47:54 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b7f8a5c0c45e29642dda6d575de9b971c192935c3599470487fd0e8b4732be7" in 3.662709988s
10:47:54 | openshift-authentication | multus | oauth-openshift-667c95946d-9m5z4 | AddedInterface | Add eth0 [10.129.0.28/23]
10:47:54 | openshift-kube-storage-version-migrator-operator | openshift-kube-storage-version-migrator-operator-status-controller-statussyncer_kube-storage-version-migrator | kube-storage-version-migrator-operator | OperatorStatusChanged | Status for clusteroperator/kube-storage-version-migrator changed: status.versions changed from [{"operator" "4.6.52"}] to [{"operator" "4.6.52"} {"kube-storage-version-migrator" ""}]
10:47:54 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba303d6ca96abd0073e2f0c03f1db898665aa7d9f8e56a4814ba7625eb48ceec"
10:47:54 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Started | Started container telemeter-client
10:47:54 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Started | Started container kube-state-metrics
10:47:54 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:54 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b7f8a5c0c45e29642dda6d575de9b971c192935c3599470487fd0e8b4732be7" in 711.453445ms
10:47:54 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-nnfsr | Created | Created container prometheus-adapter
10:47:54 | openshift-marketplace | kubelet | certified-operators-p45rl | Created | Created container registry-server
10:47:54 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b7f8a5c0c45e29642dda6d575de9b971c192935c3599470487fd0e8b4732be7" in 657.825868ms
10:47:54 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" in 2.952523403s
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4"
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-1 | Created | Created container alertmanager
10:47:55 | openshift-marketplace | kubelet | certified-operators-p45rl | Started | Started container registry-server
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-0 | Started | Started container alertmanager
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-0 | Created | Created container alertmanager
10:47:55 | openshift-authentication | kubelet | oauth-openshift-667c95946d-9m5z4 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931" already present on machine
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-2 | Created | Created container alertmanager
10:47:55 | openshift-authentication | kubelet | oauth-openshift-667c95946d-9m5z4 | Created | Created container oauth-openshift
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-2 | Started | Started container alertmanager
10:47:55 | openshift-marketplace | kubelet | redhat-operators-hr4dq | Created | Created container registry-server
10:47:55 | openshift-authentication | kubelet | oauth-openshift-667c95946d-9m5z4 | Started | Started container oauth-openshift
10:47:55 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for OAuth server observed generation 2 to match expected generation 3" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)"
10:47:55 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Created | Created container openshift-state-metrics
10:47:55 | openshift-marketplace | kubelet | redhat-operators-hr4dq | Started | Started container registry-server
10:47:55 | openshift-monitoring | kubelet | openshift-state-metrics-6d99649c49-4b6cb | Started | Started container openshift-state-metrics
10:47:55 | openshift-marketplace | kubelet | community-operators-mprwd | Created | Created container registry-server
10:47:55 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/config-4 -n openshift-kube-apiserver because it was missing
10:47:55 | openshift-marketplace | kubelet | community-operators-mprwd | Started | Started container registry-server
10:47:55 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4"
10:47:55 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Created | Created container kube-rbac-proxy-main
10:47:55 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Started | Started container kube-rbac-proxy-main
10:47:55 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4"
10:47:55 | openshift-monitoring | kubelet | prometheus-adapter-5fdd4bd886-nnfsr | Started | Started container prometheus-adapter
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container prometheus-proxy
10:47:56 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)"
10:47:56 | openshift-etcd | multus | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.40/23]
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" in 1.180481934s
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container config-reloader
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container config-reloader
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e"
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5" already present on machine
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container thanos-sidecar
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container prometheus-proxy
10:47:56 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container thanos-sidecar
10:47:56 | openshift-monitoring | kubelet | alertmanager-main-1 | Started | Started container alertmanager
10:47:56 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4"
10:47:57 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-cert-syncer-kubeconfig-4 -n openshift-kube-apiserver because it was missing
10:47:57 | openshift-dns | multus | dns-default-fcp4w | AddedInterface | Add eth0 [10.129.2.2/23]
10:47:57 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container kube-rbac-proxy
10:47:57 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Started | Started container reload
10:47:57 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" already present on machine
10:47:57 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba303d6ca96abd0073e2f0c03f1db898665aa7d9f8e56a4814ba7625eb48ceec" in 2.859377054s
10:47:57 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container prom-label-proxy
10:47:57 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container prom-label-proxy
10:47:57 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Started | Started container kube-rbac-proxy-self
10:47:57 | openshift-dns | kubelet | dns-default-fcp4w | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf"
10:47:57 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5" in 3.89138763s
10:47:57 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Created | Created container thanos-query
10:47:57 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Started | Started container thanos-query
10:47:57 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Created | Created container kube-rbac-proxy-self
10:47:57 | openshift-multus | multus | network-metrics-daemon-xxx4f | AddedInterface | Add eth0 [10.129.2.3/23]
10:47:57 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1"
10:47:57 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" already present on machine
10:47:57 | openshift-monitoring | kubelet | kube-state-metrics-6c8b757584-2b2dg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:57 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container kube-rbac-proxy
10:47:57 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Created | Created container reload
10:47:58 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)"
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" in 2.341385695s
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-0 | Created | Created container config-reloader
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-0 | Started | Started container config-reloader
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:58 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 3 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nIngressStateEndpointsDegraded: All 3 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:47:58 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)" to "OAuthVersionDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)"
10:47:58 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceEndpointsCheckEndpointAccessibleControllerAvailable: Failed to get oauth-openshift enpoints\nOAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-1 | Created | Created container config-reloader
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-1 | Started | Started container config-reloader
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" in 2.412402487s
10:47:58 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: oauth service endpoints are not ready\nIngressStateEndpointsDegraded: All 3 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-2 | Created | Created container config-reloader
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-2 | Started | Started container config-reloader
10:47:58 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:58 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Started | Started container oauth-proxy
10:47:58 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Created | Created container oauth-proxy
10:47:58 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:58 (x2) | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container prometheus
10:47:58 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Started | Started container kube-rbac-proxy
10:47:58 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Created | Created container kube-rbac-proxy
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:47:58 | openshift-monitoring | kubelet | telemeter-client-57c59fc45f-gsqrj | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:47:58 | openshift-etcd | kubelet | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:47:58 | openshift-etcd | kubelet | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:47:58 | openshift-etcd | kubelet | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ac56740ab9d9685e2630c017f14237a53b3e1f8903e9c077f48a78d6c7c134b" already present on machine
10:47:58 (x2) | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container prometheus
10:47:58 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" in 2.098866274s
10:47:59 | openshift-monitoring | kubelet | alertmanager-main-2 | Created | Created container alertmanager-proxy
10:47:59 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Started | Started container kube-rbac-proxy
10:47:59 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3"
10:47:59 | openshift-monitoring | kubelet | alertmanager-main-0 | Started | Started container alertmanager-proxy
10:47:59 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for 3 replicas of OAuth server to be available (container is not ready in oauth-openshift-765c977d67-lh7jg pod,container is not ready in oauth-openshift-d5d844698-kh5nv pod,container is waiting in pending oauth-openshift-667c95946d-9m5z4 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)",Available message changed from "OAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthServerDeploymentAvailable: availableReplicas==0\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:47:59 | openshift-monitoring | kubelet | alertmanager-main-1 | Created | Created container alertmanager-proxy
10:47:59 | openshift-monitoring | kubelet | alertmanager-main-2 | Started | Started container alertmanager-proxy
10:47:59 | openshift-monitoring | kubelet | alertmanager-main-1 | Started | Started container alertmanager-proxy
10:47:59 | openshift-monitoring | kubelet | alertmanager-main-0 | Created | Created container alertmanager-proxy
10:47:59 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Created | Created container kube-rbac-proxy
10:48:00 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Created | Created container kube-rbac-proxy
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Created | Created container kube-rbac-proxy
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d"
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Created | Created container dns
10:48:00 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Started | Started container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3"
10:48:00 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a97c9a38c7b4900d593ddeb44e98e6351f7b8ebbc0dffc406ead0dc4cb7a1ed1" in 2.248149054s
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-2 | Created | Created container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:00 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" already present on machine
10:48:00 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Started | Started container network-metrics-daemon
10:48:00 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:00 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/oauth-metadata-4 -n openshift-kube-apiserver because it was missing
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3"
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-1 | Started | Started container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-1 | Created | Created container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:00 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" in 3.873905992s
10:48:00 | openshift-multus | kubelet | network-metrics-daemon-xxx4f | Created | Created container network-metrics-daemon
10:48:00 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "OAuthServiceCheckEndpointAccessibleControllerAvailable: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nOAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "OAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Started | Started container dns
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3"
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-0 | Started | Started container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-0 | Created | Created container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4bfcb018947b57966ce4bc55f9d96768ffc0127195ae207324f3f8cde8ee1ebf" in 2.273665161s
10:48:00 | openshift-dns | kubelet | dns-default-fcp4w | Started | Started container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | alertmanager-main-2 | Started | Started container kube-rbac-proxy
10:48:00 | openshift-monitoring | kubelet | prometheus-k8s-0 | FailedMount | MountVolume.SetUp failed for volume "prometheus-k8s-rulefiles-0" : configmap "prometheus-k8s-rulefiles-0" not found
10:48:01 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Started | Started container kube-rbac-proxy-rules
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-0 | Started | Started container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container thanos-sidecar
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container thanos-sidecar
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container prometheus-proxy
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container prometheus-proxy
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" in 218.229027ms
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-1 | Created | Created container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-1 | Started | Started container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-0 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" in 328.63825ms
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-0 | Created | Created container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-2 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" in 306.966662ms
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-2 | Created | Created container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | alertmanager-main-2 | Started | Started container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5" already present on machine
10:48:01 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Created | Created container kube-rbac-proxy-rules
10:48:01 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
10:48:01 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Started | Started container prom-label-proxy
10:48:01 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Created | Created container prom-label-proxy
10:48:01 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready"),Progressing message changed from "NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 2; 0 nodes have achieved new revision 3" to "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3",Available message changed from "StaticPodsAvailable: 2 nodes are active; 1 nodes are at revision 0; 2 nodes are at revision 2; 0 nodes have achieved new revision 3" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3"
10:48:01 | openshift-monitoring | kubelet | thanos-querier-5c8f998999-vrvtc | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" in 1.653914285s
10:48:01 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 0 to 3 because static pod is ready
10:48:01 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthServiceCheckEndpointAccessibleControllerDegraded: Get \"https://172.30.82.175:443/healthz\": dial tcp 172.30.82.175:443: connect: connection refused\nIngressStateEndpointsDegraded: All 3 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "IngressStateEndpointsDegraded: All 3 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container config-reloader
10:48:01 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container config-reloader
10:48:02 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container prom-label-proxy
10:48:02 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container kube-rbac-proxy
10:48:02 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" already present on machine
10:48:02 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container kube-rbac-proxy
10:48:02 | openshift-dns | kubelet | dns-default-fcp4w | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f95ac948edf7e12a3d46e88cbab8c7010ef9c3af6df2ff002af28ba76445247d" in 2.149278287s
10:48:02 (x2) | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container prometheus
10:48:02 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled up replica set oauth-openshift-667c95946d to 2
10:48:02 (x2) | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container prometheus
10:48:02 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled down replica set oauth-openshift-765c977d67 to 0
10:48:02 | openshift-authentication | replicaset-controller | oauth-openshift-765c977d67 | SuccessfulDelete | Deleted pod: oauth-openshift-765c977d67-lh7jg
10:48:02 | openshift-authentication | kubelet | oauth-openshift-765c977d67-lh7jg | Killing | Stopping container oauth-openshift
10:48:02 | openshift-authentication | replicaset-controller | oauth-openshift-667c95946d | SuccessfulCreate | Created pod: oauth-openshift-667c95946d-kk679
10:48:02 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)\nOAuthVersionRouteProgressing: Request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" not successfull yet\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, )\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)",Available message changed from "OAuthVersionRouteAvailable: HTTP request to \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\" failed: EOF\nWellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "WellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF"
10:48:02 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 3 endpoints for oauth-server are reporting 'not ready'\nOAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "OAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:48:02 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" already present on machine
10:48:02 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container prom-label-proxy
10:48:03 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "OAuthRouteCheckEndpointAccessibleControllerDegraded: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF\nWellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "WellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:48:03 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "WellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)\nOAuthRouteCheckEndpointAccessibleControllerAvailable: Get \"https://oauth-openshift.apps.case03116926.lab.upshift.rdu2.redhat.com/healthz\": EOF" to "WellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)"
10:48:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:48:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container wait-for-host-port
10:48:04 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, )\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, container is not ready in oauth-openshift-667c95946d-9m5z4 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, )\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)"
10:48:04 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container wait-for-host-port
10:48:05 | openshift-authentication | multus | oauth-openshift-667c95946d-kk679 | AddedInterface | Add eth0 [10.128.0.41/23]
10:48:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/bound-sa-token-signing-certs-4 -n openshift-kube-apiserver because it was missing
10:48:05 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (2 not ready, )\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)"
10:48:05 | openshift-authentication | kubelet | oauth-openshift-667c95946d-kk679 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0e3fc052d02f74ef44eda4392d91a6708e3efeb81e9286694f24c788b8ee931" already present on machine
10:48:06 | openshift-authentication | kubelet | oauth-openshift-667c95946d-kk679 | Started | Started container oauth-openshift
10:48:06 | openshift-authentication | kubelet | oauth-openshift-667c95946d-kk679 | Created | Created container oauth-openshift
10:48:06 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:48:06 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)"
10:48:07 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-cert-syncer
10:48:07 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler
10:48:07 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler
10:48:07 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:48:07 | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-cert-syncer
10:48:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca-4 -n openshift-kube-apiserver because it was missing
10:48:08 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is waiting in pending oauth-openshift-667c95946d-kk679 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)"
10:48:10 (x2) | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, )\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)" to "OAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)"
10:48:10 | openshift-authentication | deployment-controller | oauth-openshift | ScalingReplicaSet | Scaled down replica set oauth-openshift-d5d844698 to 0
10:48:10 | openshift-authentication | kubelet | oauth-openshift-d5d844698-kh5nv | Killing | Stopping container oauth-openshift
10:48:10 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing message changed from "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)" to "OAuthVersionDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, )\nOAuthServerDeploymentProgressing: Waiting for all OAuth server replicas to be ready (1 not ready, container is not ready in oauth-openshift-667c95946d-kk679 pod)"
10:48:10 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-server-ca-4 -n openshift-kube-apiserver because it was missing
10:48:10 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorVersionChanged | clusteroperator/authentication version "operator" changed from "" to "4.6.52"
10:48:10 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorVersionChanged | clusteroperator/authentication version "oauth-openshift" changed from "" to "4.6.52_openshift"
10:48:10 | openshift-authentication | replicaset-controller | oauth-openshift-d5d844698 | SuccessfulDelete | Deleted pod: oauth-openshift-d5d844698-kh5nv
10:48:10 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: status.versions changed from [{"oauth-apiserver" "4.6.52"}] to [{"oauth-apiserver" "4.6.52"} {"operator" "4.6.52"} {"oauth-openshift" "4.6.52_openshift"}]
10:48:11 (x8) | openshift-authentication-operator | cluster-authentication-operator-deployment-controller-deployment | authentication-operator | DeploymentUpdated | Updated Deployment.apps/oauth-openshift -n openshift-authentication because it changed
10:48:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:48:11 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Progressing changed from True to False ("")
10:48:13 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kubelet-serving-ca-4 -n openshift-kube-apiserver because it was missing
10:48:13 | openshift-kube-apiserver | multus | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.43/23]
10:48:14 | openshift-kube-apiserver | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:48:14 | openshift-console-operator | console-operator | console-operator | ConfigMapUpdated | Updated ConfigMap/console-config -n openshift-console: cause by changes in data.console-config.yaml
10:48:15 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing message changed from "SyncLoopRefreshProgressing: Working toward version 4.6.52" to "SyncLoopRefreshProgressing: Changes made during sync updates, additional sync expected."
10:48:15 | openshift-console-operator | console-operator | console-operator | DeploymentUpdated | Updated Deployment.apps/console -n openshift-console because it changed
10:48:15 | openshift-console | deployment-controller | console | ScalingReplicaSet | Scaled up replica set console-7d6f969d46 to 1
10:48:15 | openshift-console | replicaset-controller | console-7d6f969d46 | SuccessfulCreate | Created pod: console-7d6f969d46-crrrm
10:48:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/sa-token-signing-certs-4 -n openshift-kube-apiserver because it was missing
10:48:15 | openshift-kube-apiserver | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:48:15 | openshift-kube-apiserver | kubelet | revision-pruner-3-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:48:17 | openshift-console | multus | console-7d6f969d46-crrrm | AddedInterface | Add eth0 [10.130.0.44/23]
10:48:18 | openshift-console | kubelet | console-7d6f969d46-crrrm | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d"
10:48:18 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-audit-policies-4 -n openshift-kube-apiserver because it was missing
10:48:19 | openshift-authentication | kubelet | oauth-openshift-765c977d67-9ztxx | Unhealthy | Liveness probe failed: Get "https://10.129.0.26:6443/healthz": dial tcp 10.129.0.26:6443: i/o timeout
10:48:19 (x2) | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing message changed from "SyncLoopRefreshProgressing: Changes made during sync updates, additional sync expected." to "SyncLoopRefreshProgressing: Working toward version 4.6.52"
10:48:21 | openshift-console | kubelet | console-7d6f969d46-crrrm | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d" in 3.308775941s
10:48:21 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/etcd-client-4 -n openshift-kube-apiserver because it was missing
10:48:21 | openshift-console | kubelet | console-7d6f969d46-crrrm | Started | Started container console
10:48:21 | openshift-console | kubelet | console-7d6f969d46-crrrm | Created | Created container console
10:48:24 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/kubelet-client-4 -n openshift-kube-apiserver because it was missing
10:48:26 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-serving-certkey-4 -n openshift-kube-apiserver because it was missing
10:48:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionCreate | Revision 3 created because configmap/oauth-metadata has changed
10:48:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-client-token-4 -n openshift-kube-apiserver because it was missing
10:48:28 | openshift-authentication | kubelet | oauth-openshift-765c977d67-lh7jg | Unhealthy | Readiness probe failed: Get "https://10.128.0.39:6443/healthz": dial tcp 10.128.0.39:6443: connect: connection refused
10:48:31 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3" to "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3; 0 nodes have achieved new revision 4",Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3; 0 nodes have achieved new revision 4"
10:48:31 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 3 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
10:48:36 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:48:38 | openshift-kube-apiserver | multus | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.42/23]
10:48:40 | openshift-kube-apiserver | kubelet | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:48:41 | openshift-kube-apiserver | kubelet | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:48:41 | openshift-kube-apiserver | kubelet | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:48:48 | openshift-kube-apiserver | static-pod-installer | installer-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 3
10:48:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
10:48:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
10:48:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
10:48:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
10:48:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
10:48:50 | openshift-kube-apiserver | cert-regeneration-controller | cert-regeneration-controller-lock | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_3507af22-d386-4ea0-b055-d6c45df13215 became leader
10:48:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending"
10:48:55 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-3 -n openshift-kube-apiserver: cause by changes in data.status
10:48:57 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:48:59 | openshift-kube-apiserver | multus | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.43/23]
10:49:00 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 5 to 6 because static pod is ready
10:49:00 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 6"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 5; 2 nodes are at revision 6" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6"
10:49:01 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-installer-controller | openshift-kube-scheduler-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-6 -n openshift-kube-scheduler: cause by changes in data.status
10:49:01 | openshift-kube-apiserver | kubelet | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:49:02 | openshift-kube-apiserver | kubelet | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:49:02 | openshift-kube-apiserver | kubelet | revision-pruner-3-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:49:04 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-prunecontroller | openshift-kube-scheduler-operator | PodCreated | Created Pod/revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-scheduler because it was missing
10:49:07 | openshift-kube-scheduler | multus | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.29/23]
10:49:09 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:49:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: static pod of revision 3 has been installed, but is not ready while new revision 4 is pending" to "NodeControllerDegraded: All master nodes are ready\nNodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: "
10:49:09 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 4 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
10:49:09 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:49:09 | openshift-kube-scheduler | kubelet | revision-pruner-6-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:49:13 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:49:15 | openshift-kube-apiserver | multus | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.44/23]
10:49:17 | openshift-kube-apiserver | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:49:18 | openshift-kube-apiserver | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:49:18 | openshift-kube-apiserver | kubelet | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:49:25 | openshift-kube-apiserver | static-pod-installer | installer-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 4
10:49:35 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Degraded changed from False to True ("RouteHealthDegraded: route not yet available, https://console-openshift-console.apps.case03116926.lab.upshift.rdu2.redhat.com/health returns '503 Service Unavailable'")
10:50:00 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/revision-status-8 -n openshift-kube-controller-manager because it was missing
10:50:00 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-sa-token-signer-controller | kube-controller-manager-operator | SecretUpdated | Updated Secret/service-account-private-key -n openshift-kube-controller-manager because it changed
10:50:01 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-manager-pod-8 -n openshift-kube-controller-manager because it was missing
10:50:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/cluster-policy-controller-config-8 -n openshift-kube-controller-manager because it was missing
10:50:02 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/config-8 -n openshift-kube-controller-manager because it was missing
10:50:03 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/controller-manager-kubeconfig-8 -n openshift-kube-controller-manager because it was missing
10:50:03 | openshift-marketplace | multus | redhat-operators-ds5xd | AddedInterface | Add eth0 [10.129.2.4/23]
10:50:04 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/kube-controller-cert-syncer-kubeconfig-8 -n openshift-kube-controller-manager because it was missing
10:50:04 | openshift-marketplace | kubelet | redhat-operators-ds5xd | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
10:50:05 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/serviceaccount-ca-8 -n openshift-kube-controller-manager because it was missing
10:50:05 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/service-ca-8 -n openshift-kube-controller-manager because it was missing
10:50:06 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | ConfigMapCreated | Created ConfigMap/recycler-config-8 -n openshift-kube-controller-manager because it was missing
10:50:07 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/serving-cert-8 -n openshift-kube-controller-manager because it was missing
10:50:07 | openshift-marketplace | multus | community-operators-hs279 | AddedInterface | Add eth0 [10.129.2.5/23]
10:50:07 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/service-account-private-key-8 -n openshift-kube-controller-manager because it was missing
10:50:08 | openshift-marketplace | kubelet | community-operators-hs279 | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
10:50:08 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionTriggered | new revision 8 triggered by "secret/service-account-private-key has changed"
10:50:08 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | SecretCreated | Created Secret/localhost-recovery-client-token-8 -n openshift-kube-controller-manager because it was missing
10:50:08 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-revisioncontroller | kube-controller-manager-operator | RevisionCreate | Revision 7 created because secret/service-account-private-key has changed
10:50:09 | openshift-marketplace | kubelet | certified-operators-trz5k | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
10:50:09 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 8" to "NodeControllerDegraded: All master nodes are ready"
10:50:09 | openshift-marketplace | multus | certified-operators-trz5k | AddedInterface | Add eth0 [10.129.2.6/23]
10:50:09 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 8"
10:50:15 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 7 to 8 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 7 is the oldest
10:50:15 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 7; 0 nodes have achieved new revision 8"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 7" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 7; 0 nodes have achieved new revision 8"
10:50:17 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-8-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:50:19 | openshift-kube-controller-manager | multus | installer-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.45/23]
10:50:19 | openshift-marketplace | kubelet | certified-operators-trz5k | Started | Started container registry-server
10:50:19 | openshift-marketplace | kubelet | redhat-operators-ds5xd | Created | Created container registry-server
10:50:19 | openshift-marketplace | kubelet | community-operators-hs279 | Started | Started container registry-server
10:50:19 | openshift-marketplace | kubelet | certified-operators-trz5k | Created | Created container registry-server
10:50:19 | openshift-marketplace | kubelet | redhat-operators-ds5xd | Started | Started container registry-server
10:50:19 | openshift-marketplace | kubelet | community-operators-hs279 | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 11.357157106s
10:50:19 | openshift-marketplace | kubelet | community-operators-hs279 | Created | Created container registry-server
10:50:19 | openshift-marketplace | kubelet | redhat-operators-ds5xd | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 15.180871838s
10:50:19 | openshift-marketplace | kubelet | certified-operators-trz5k | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 9.477245917s
10:50:20 | openshift-kube-controller-manager | kubelet | installer-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:50:21 | openshift-kube-controller-manager | kubelet | installer-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:50:21 | openshift-kube-controller-manager | kubelet | installer-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:50:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:50:23 | openshift-kube-controller-manager | static-pod-installer | installer-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 8
10:50:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:50:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:50:23 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:50:24 (x2) | openshift-marketplace | kubelet | certified-operators-trz5k | Killing | Stopping container registry-server
10:50:26 | default | node-controller | worker-2.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node worker-2.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node worker-2.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:50:26 | default | node-controller | master-1.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node master-1.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node master-1.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:50:26 | default | node-controller | master-0.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node master-0.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node master-0.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:50:26 | default | node-controller | worker-0.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node worker-0.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node worker-0.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:50:26 | default | node-controller | worker-1.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node worker-1.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node worker-1.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:50:26 | default | node-controller | master-2.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node master-2.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node master-2.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:50:26 | kube-system | kube-controller-manager | kube-controller-manager | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_b18b0120-46d7-4e3a-8a67-1a7bca6f1b14 became leader
10:50:29 (x2) | openshift-marketplace | kubelet | community-operators-hs279 | Killing | Stopping container registry-server
10:50:31 | openshift-marketplace | multus | community-operators-mljlz | AddedInterface | Add eth0 [10.129.2.7/23]
10:50:31 (x2) | openshift-marketplace | kubelet | redhat-operators-ds5xd | Killing | Stopping container registry-server
10:50:32 | openshift-marketplace | kubelet | community-operators-mljlz | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 450.455336ms
10:50:32 | openshift-marketplace | kubelet | community-operators-mljlz | Started | Started container registry-server
10:50:32 | openshift-marketplace | kubelet | community-operators-mljlz | Created | Created container registry-server
10:50:32 | openshift-marketplace | kubelet | community-operators-mljlz | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
10:50:34 (x17) | openshift-console | kubelet | console-7fdcf89666-fslmv | Unhealthy | Readiness probe failed: Get "https://10.129.0.25:8443/health": dial tcp 10.129.0.25:8443: connect: connection refused
10:50:36 (x3) | openshift-console | kubelet | console-7fdcf89666-fslmv | Unhealthy | Liveness probe failed: Get "https://10.129.0.25:8443/health": dial tcp 10.129.0.25:8443: connect: connection refused
10:50:36 | openshift-console | kubelet | console-7fdcf89666-fslmv | Killing | Container console failed liveness probe, will be restarted
10:50:37 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:50:37 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:50:38 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:50:39 (x3) | openshift-console | kubelet | console-7fdcf89666-hh29k | Unhealthy | Liveness probe failed: Get "https://10.128.0.38:8443/health": dial tcp 10.128.0.38:8443: connect: connection refused
10:50:39 (x18) | openshift-console | kubelet | console-7fdcf89666-hh29k | Unhealthy | Readiness probe failed: Get "https://10.128.0.38:8443/health": dial tcp 10.128.0.38:8443: connect: connection refused
10:50:39 (x2) | openshift-marketplace | kubelet | community-operators-mljlz | Killing | Stopping container registry-server
10:50:41 | openshift-marketplace | multus | community-operators-xc9nn | AddedInterface | Add eth0 [10.129.2.8/23]
10:50:41 | openshift-marketplace | kubelet | community-operators-xc9nn | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
10:50:42 (x2) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
10:50:42 | openshift-marketplace | kubelet | community-operators-xc9nn | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 410.875265ms
10:50:42 | openshift-marketplace | kubelet | community-operators-xc9nn | Created | Created container registry-server
10:50:42 (x2) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
10:50:42 (x2) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:50:42 | openshift-marketplace | kubelet | community-operators-xc9nn | Started | Started container registry-server
10:50:53 (x2) | openshift-marketplace | kubelet | community-operators-xc9nn | Killing | Stopping container registry-server
10:50:53 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from False to True ("NodeInstallerDegraded: 1 nodes are failing on revision 3:\nNodeInstallerDegraded: ")
10:50:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 7 to 8 because static pod is ready
10:50:58 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 7; 0 nodes have achieved new revision 8" to "NodeInstallerProgressing: 2 nodes are at revision 7; 1 nodes are at revision 8",Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 7; 0 nodes have achieved new revision 8" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 7; 1 nodes are at revision 8"
10:51:00 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-8-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:51:02 | openshift-kube-controller-manager | multus | revision-pruner-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.46/23]
10:51:02 | openshift-kube-scheduler | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com_4715076d-4ebf-4f56-ba86-b55879497fa1 became leader
10:51:04 | openshift-kube-apiserver | endpoint-controller | apiserver | FailedToUpdateEndpoint | Failed to update endpoint openshift-kube-apiserver/apiserver: Operation cannot be fulfilled on endpoints "apiserver": the object has been modified; please apply your changes to the latest version and try again
10:51:04 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:51:04 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:51:04 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:51:04 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:51:05 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:51:05 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:51:05 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:51:05 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:51:05 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:51:05 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 7 to 8 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 7 is the oldest
10:51:05 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:51:06 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:51:07 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:51:07 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-8-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:51:07 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:51:07 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:51:07 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:51:11 | openshift-kube-controller-manager | multus | installer-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.45/23]
10:51:13 | openshift-kube-controller-manager | kubelet | installer-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:51:13 | openshift-kube-controller-manager | kubelet | installer-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:51:14 | openshift-kube-controller-manager | kubelet | installer-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:51:14 (x18) | openshift-console | kubelet | console-7d6f969d46-crrrm | Unhealthy | Readiness probe failed: Get "https://10.130.0.44:8443/health": dial tcp 10.130.0.44:8443: connect: connection refused
10:51:16 | openshift-kube-controller-manager | static-pod-installer | installer-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 8
10:51:16 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:51:16 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:51:16 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:51:16 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:51:18 (x3) | openshift-console | kubelet | console-7d6f969d46-crrrm | Unhealthy | Liveness probe failed: Get "https://10.130.0.44:8443/health": dial tcp 10.130.0.44:8443: connect: connection refused
10:51:20 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Degraded changed from True to False ("")
10:51:23 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Available message changed from "DeploymentAvailable: 0 pods available for console deployment" to "DeploymentAvailable: 1 replicas ready at version 4.6.52"
10:51:25 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 4 because static pod is ready
10:51:25 | openshift-console | replicaset-controller | console-7fdcf89666 | SuccessfulDelete | Deleted pod: console-7fdcf89666-fslmv
10:51:25 | openshift-console | deployment-controller | console | ScalingReplicaSet | Scaled down replica set console-7fdcf89666 to 1
10:51:25 | openshift-console | replicaset-controller | console-7d6f969d46 | SuccessfulCreate | Created pod: console-7d6f969d46-7jdhw
10:51:25 | openshift-console | deployment-controller | console | ScalingReplicaSet | Scaled up replica set console-7d6f969d46 to 2
10:51:25 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded changed from True to False ("NodeControllerDegraded: All master nodes are ready"),Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 2; 1 nodes are at revision 3; 0 nodes have achieved new revision 4" to "NodeInstallerProgressing: 1 nodes are at revision 2; 1 nodes are at revision 3; 1 nodes are at revision 4",Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 2; 1 nodes are at revision 3; 0 nodes have achieved new revision 4" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 1 nodes are at revision 3; 1 nodes are at revision 4"
10:51:26 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Available message changed from "DeploymentAvailable: 1 replicas ready at version 4.6.52" to "DeploymentAvailable: 2 replicas ready at version 4.6.52"
10:51:27 | openshift-console | multus | console-7d6f969d46-7jdhw | AddedInterface | Add eth0 [10.128.0.46/23]
10:51:27 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:51:27 | openshift-console | kubelet | console-7d6f969d46-7jdhw | Started | Started container console
10:51:27 | openshift-console | kubelet | console-7d6f969d46-7jdhw | Created | Created container console
10:51:27 | openshift-console | kubelet | console-7d6f969d46-7jdhw | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08259a3570f43a3dc707ead84f600d27dfe881532c1a4fc6f0aadb6cc297ae8d" already present on machine
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:51:28 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:51:29 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:51:31 | openshift-kube-apiserver | multus | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.47/23]
10:51:33 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:51:34 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:51:34 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:51:36 | openshift-console | deployment-controller | console | ScalingReplicaSet | Scaled down replica set console-7fdcf89666 to 0
10:51:36 | openshift-console | replicaset-controller | console-7fdcf89666 | SuccessfulDelete | Deleted pod: console-7fdcf89666-hh29k
10:51:37 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Progressing changed from True to False (""),Available changed from False to True ("")
10:51:44 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Available changed from True to False ("DeploymentAvailable: 1 replicas ready at version 4.6.52")
10:51:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 7; 1 nodes are at revision 8" to "NodeInstallerProgressing: 1 nodes are at revision 7; 2 nodes are at revision 8",Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 7; 1 nodes are at revision 8" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 7; 2 nodes are at revision 8"
10:51:45 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 4 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 2 is the oldest
10:51:45 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 7 to 8 because static pod is ready
10:51:48 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-8-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:51:49 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-4-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:51:50 | openshift-kube-controller-manager | multus | revision-pruner-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.48/23]
10:51:51 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 7 to 8 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 7 is the oldest
10:51:51 | openshift-kube-apiserver | multus | installer-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.30/23]
10:51:51 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:51:52 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:51:52 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:51:53 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | PodCreated | Created Pod/installer-8-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:51:53 | openshift-kube-apiserver | kubelet | installer-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:51:54 | openshift-kube-apiserver | kubelet | installer-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:51:54 | openshift-kube-apiserver | kubelet | installer-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:51:55 | openshift-kube-controller-manager | multus | installer-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.31/23]
10:51:57 | openshift-kube-controller-manager | kubelet | installer-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:51:58 | openshift-kube-controller-manager | kubelet | installer-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:51:58 | openshift-kube-controller-manager | kubelet | installer-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:52:01 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
10:52:01 | openshift-kube-controller-manager | static-pod-installer | installer-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 8
10:52:01 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
10:52:01 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
10:52:01 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
10:52:01 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-recovery-controller
10:52:01 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager-cert-syncer
10:52:01 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container cluster-policy-controller
10:52:01 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
10:52:01 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-controller-manager
10:52:01 | openshift-kube-controller-manager | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_73259534-44e5-497f-80fe-fcb84540baf2 became leader
10:52:01 | openshift-kube-apiserver | static-pod-installer | installer-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 4
10:52:02 | kube-system | kube-controller-manager | kube-controller-manager | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_180299a7-8a37-4932-9718-c20bea8a86b2 became leader
10:52:03 | default | node-controller | master-0.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node master-0.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node master-0.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:52:03 | default | node-controller | worker-1.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node worker-1.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node worker-1.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:52:03 | default | node-controller | master-1.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node master-1.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node master-1.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:52:03 | default | node-controller | worker-2.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node worker-2.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node worker-2.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:52:03 | default | node-controller | worker-0.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node worker-0.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node worker-0.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:52:03 | default | node-controller | master-2.case03116926.lab.upshift.rdu2.redhat.com | RegisteredNode | Node master-2.case03116926.lab.upshift.rdu2.redhat.com event: Registered Node master-2.case03116926.lab.upshift.rdu2.redhat.com in Controller
10:52:05 | openshift-console-operator | console-operator-status-controller-statussyncer_console | console-operator | OperatorStatusChanged | Status for clusteroperator/console changed: Available changed from False to True ("")
10:52:07 | openshift-kube-apiserver | cert-regeneration-controller | cert-regeneration-controller-lock | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com_638f1e24-535c-4f23-bf01-569651546c3d became leader
10:52:13 | openshift-kube-controller-manager | cluster-policy-controller | cluster-policy-controller | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com became leader
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container cluster-policy-controller
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f47007f136492ddd3574d720493da6624ddfae842363e6f96ecda6bb0af8cfe7" already present on machine
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container cluster-policy-controller
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:52:14 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager
10:52:15 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-cert-syncer
10:52:15 | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-cert-syncer
10:52:33 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 7 to 8 because static pod is ready
10:52:33 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 8"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 7; 2 nodes are at revision 8" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 8"
10:52:34 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-installer-controller | kube-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-8 -n openshift-kube-controller-manager: cause by changes in data.status
10:52:37 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-prunecontroller | kube-controller-manager-operator | PodCreated | Created Pod/revision-pruner-8-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-controller-manager because it was missing
10:52:39 | openshift-kube-controller-manager | multus | revision-pruner-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.32/23]
10:52:41 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
10:52:42 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:52:42 | openshift-kube-controller-manager | kubelet | revision-pruner-8-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:53:18 | openshift-marketplace | kubelet | redhat-marketplace-dkmx8 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
10:53:18 | openshift-marketplace | multus | redhat-marketplace-dkmx8 | AddedInterface | Add eth0 [10.129.2.9/23]
10:53:21 | openshift-marketplace | kubelet | redhat-marketplace-dkmx8 | Started | Started container registry-server
10:53:21 | openshift-marketplace | kubelet | redhat-marketplace-dkmx8 | Created | Created container registry-server
10:53:21 | openshift-marketplace | kubelet | redhat-marketplace-dkmx8 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 2.865372095s
10:53:33 (x2) | openshift-marketplace | kubelet | redhat-marketplace-dkmx8 | Killing | Stopping container registry-server
10:53:35 | openshift-marketplace | multus | redhat-marketplace-p8mr5 | AddedInterface | Add eth0 [10.129.2.10/23]
10:53:35 | openshift-marketplace | kubelet | redhat-marketplace-p8mr5 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
10:53:36 | openshift-marketplace | kubelet | redhat-marketplace-p8mr5 | Created | Created container registry-server
10:53:36 | openshift-marketplace | kubelet | redhat-marketplace-p8mr5 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 772.197045ms
10:53:36 | openshift-marketplace | kubelet | redhat-marketplace-p8mr5 | Started | Started container registry-server
10:53:47 (x2) | openshift-marketplace | kubelet | redhat-marketplace-p8mr5 | Killing | Stopping container registry-server
10:54:22 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:54:22 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:54:22 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:55:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:55:18 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:55:18 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:55:18 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:55:18 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:55:18 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:55:19 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:55:19 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:55:35 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 2 to 4 because static pod is ready
10:55:35 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 1 nodes are at revision 2; 1 nodes are at revision 3; 1 nodes are at revision 4" to "NodeInstallerProgressing: 1 nodes are at revision 3; 2 nodes are at revision 4",Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 2; 1 nodes are at revision 3; 1 nodes are at revision 4" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 3; 2 nodes are at revision 4"
10:55:42 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-4-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:55:45 | openshift-kube-apiserver | multus | revision-pruner-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.33/23]
10:55:47 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:55:48 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
10:55:48 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:55:55 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 3 to 4 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 3 is the oldest
10:56:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:56:14 | openshift-kube-apiserver | multus | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.47/23]
10:56:16 | openshift-kube-apiserver | kubelet | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:56:16 | openshift-kube-apiserver | kubelet | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
10:56:16 | openshift-kube-apiserver | kubelet | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
10:56:24 | openshift-kube-apiserver | static-pod-installer | installer-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 4
10:56:24 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
10:56:24 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
10:56:24 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
10:56:24 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
10:56:24 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
10:56:24 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available message changed from "WellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "WellKnownAvailable: The well-known endpoint is not yet available: need at least 3 kube-apiservers, got 2"
10:56:24 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded message changed from "WellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://10.0.94.119:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)" to "WellKnownReadyControllerDegraded: need at least 3 kube-apiservers, got 2"
10:56:26 | openshift-kube-apiserver | cert-regeneration-controller | cert-regeneration-controller-lock | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_ca90cd0e-2330-4311-a739-c18959923a1d became leader
10:58:19 (x3) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
10:58:19 (x3) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
10:58:19 (x3) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
10:58:21 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pod/openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-scheduler-recovery-controller\" is terminated: Completed: \nNodeControllerDegraded: All master nodes are ready"
10:58:24 | openshift-kube-scheduler-operator | openshift-cluster-kube-scheduler-operator-status-controller-statussyncer_kube-scheduler | openshift-kube-scheduler-operator | OperatorStatusChanged | Status for clusteroperator/kube-scheduler changed: Degraded message changed from "StaticPodsDegraded: pod/openshift-kube-scheduler-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-scheduler-recovery-controller\" is terminated: Completed: \nNodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready"
10:58:44 | openshift-kube-scheduler | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_6f379deb-077a-405d-908e-faef946bd9d2 became leader
10:58:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:58:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
10:58:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
10:58:48 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
10:58:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Available changed from False to True ("OAuthServerDeploymentAvailable: availableReplicas==2")
10:58:52 | openshift-authentication-operator | cluster-authentication-operator-status-controller-statussyncer_authentication | authentication-operator | OperatorStatusChanged | Status for clusteroperator/authentication changed: Degraded changed from True to False ("")
10:59:05 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 3 to 4 because static pod is ready
10:59:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 4"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 3; 2 nodes are at revision 4" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4"
10:59:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-4 -n openshift-kube-apiserver: cause by changes in data.status
10:59:15 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
10:59:16 | openshift-kube-apiserver | multus | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.48/23]
10:59:18 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
10:59:19 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
10:59:19 | openshift-kube-apiserver | kubelet | revision-pruner-4-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
11:00:45 | openshift-marketplace | multus | redhat-operators-fllzs | AddedInterface | Add eth0 [10.129.2.11/23]
11:00:45 | openshift-marketplace | kubelet | redhat-operators-fllzs | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
11:00:46 | openshift-marketplace | kubelet | redhat-operators-fllzs | Created | Created container registry-server
11:00:46 | openshift-marketplace | kubelet | redhat-operators-fllzs | Started | Started container registry-server
11:00:46 | openshift-marketplace | kubelet | redhat-operators-fllzs | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 482.125219ms
11:00:55 (x2) | openshift-marketplace | kubelet | redhat-operators-fllzs | Killing | Stopping container registry-server
11:01:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObserveInternalRegistryHostnameChanged | Internal registry hostname changed to "image-registry.openshift-image-registry.svc:5000"
11:01:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-config-observer-configobserver | kube-apiserver-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ ... // 2 identical entries "authConfig": map[string]interface{}{"oauthMetadataFile": string("/etc/kubernetes/static-pod-resources/configmaps/oauth-metadata/oauthMetadata")}, "corsAllowedOrigins": []interface{}{string(`//127\.0\.0\.1(:|$)`), string("//localhost(:|$)")}, + "imagePolicyConfig": map[string]interface{}{ + "internalRegistryHostname": string("image-registry.openshift-image-registry.svc:5000"), + }, "servicesSubnet": string("172.30.0.0/16"), "servingInfo": map[string]interface{}{"bindAddress": string("0.0.0.0:6443"), "bindNetwork": string("tcp4"), "cipherSuites": []interface{}{string("TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"), string("TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"), string("TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"), string("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"), string("TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"), string("TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256")}, "minTLSVersion": string("VersionTLS12"), "namedCertificates": []interface{}{map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key")}, map[string]interface{}{"certFile": string("/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt"), "keyFile": string("/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key")}}}, }
11:01:09 | openshift-image-registry | deployment-controller | image-registry | ScalingReplicaSet | Scaled up replica set image-registry-7b5479fc47 to 1
11:01:09 | openshift-apiserver-operator | openshift-apiserver-operator-config-observer-configobserver | openshift-apiserver-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ "apiServerArguments": map[string]interface{}{"audit-policy-file": []interface{}{string("/var/run/configmaps/audit/secure-oauth-storage-default.yaml")}}, + "imagePolicyConfig": map[string]interface{}{ + "internalRegistryHostname": string("image-registry.openshift-image-registry.svc:5000"), + }, "projectConfig": map[string]interface{}{"projectRequestMessage": string("")}, "routingConfig": map[string]interface{}{"subdomain": string("apps.case03116926.lab.upshift.rdu2.redhat.com")}, ... // 2 identical entries }
11:01:09 | openshift-image-registry | kubelet | image-registry-7b5479fc47-m64ps | FailedMount | MountVolume.SetUp failed for volume "registry-tls" : secret "image-registry-tls" not found
11:01:09 | openshift-controller-manager-operator | openshift-controller-manager-operator-config-observer-configobserver | openshift-controller-manager-operator | ObservedConfigChanged | Writing updated observed config: map[string]interface{}{ "build": map[string]interface{}{"buildDefaults": map[string]interface{}{"resources": map[string]interface{}{}}, "imageTemplateFormat": map[string]interface{}{"format": string("quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17d408aaf7b8a00c66895a1e50b3c8529be76a342cfa84c478fa2af72576f67f")}}, "deployer": map[string]interface{}{"imageTemplateFormat": map[string]interface{}{"format": string("quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31644151defbf36a8f70424531a0d83fa238a0fe2bea2fbcd1a1db527e1b78b2")}}, + "dockerPullSecret": map[string]interface{}{ + "internalRegistryHostname": string("image-registry.openshift-image-registry.svc:5000"), + }, "ingress": map[string]interface{}{"ingressIPNetworkCIDR": string("")}, }
11:01:09 | openshift-image-registry | deployment-controller | image-registry | ScalingReplicaSet | Scaled up replica set image-registry-6cb956b4c8 to 1
11:01:09 | openshift-image-registry | replicaset-controller | image-registry-6cb956b4c8 | SuccessfulCreate | Created pod: image-registry-6cb956b4c8-nfhv5
11:01:09 | openshift-image-registry | replicaset-controller | image-registry-7b5479fc47 | SuccessfulCreate | Created pod: image-registry-7b5479fc47-m64ps
11:01:10 | openshift-image-registry | replicaset-controller | image-registry-7958686d94 | SuccessfulCreate | Created pod: image-registry-7958686d94-5n487
11:01:10 | openshift-image-registry | deployment-controller | image-registry | ScalingReplicaSet | Scaled down replica set image-registry-6cb956b4c8 to 0
11:01:10 | openshift-image-registry | deployment-controller | image-registry | ScalingReplicaSet | Scaled up replica set image-registry-7958686d94 to 1
11:01:10 | openshift-image-registry | replicaset-controller | image-registry-6cb956b4c8 | SuccessfulDelete | Deleted pod: image-registry-6cb956b4c8-nfhv5
11:01:11 | openshift-image-registry | multus | image-registry-6cb956b4c8-nfhv5 | AddedInterface | Add eth0 [10.129.2.12/23]
11:01:11 | openshift-image-registry | multus | image-registry-7958686d94-5n487 | AddedInterface | Add eth0 [10.129.2.14/23]
11:01:12 | openshift-image-registry | kubelet | image-registry-7958686d94-5n487 | Started | Started container registry
11:01:12 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-apiserver: cause by changes in data.config.yaml
11:01:12 | openshift-image-registry | kubelet | image-registry-7958686d94-5n487 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" already present on machine
11:01:12 | openshift-image-registry | kubelet | image-registry-7958686d94-5n487 | Created | Created container registry
11:01:12 | openshift-image-registry | kubelet | image-registry-6cb956b4c8-nfhv5 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" already present on machine
11:01:12 | openshift-image-registry | kubelet | image-registry-7b5479fc47-m64ps | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf3d24ef4f1e8350c5b5aaef2360121a262adf1cc088cc1c6268d8ccfe8d60d5" already present on machine
11:01:12 | openshift-image-registry | multus | image-registry-7b5479fc47-m64ps | AddedInterface | Add eth0 [10.129.2.13/23]
11:01:12 | openshift-image-registry | kubelet | image-registry-6cb956b4c8-nfhv5 | Failed | Error: cannot find volume "registry-storage" to mount into container "registry"
11:01:13 | openshift-image-registry | kubelet | image-registry-7b5479fc47-m64ps | Created | Created container registry
11:01:13 | openshift-image-registry | kubelet | image-registry-7b5479fc47-m64ps | Started | Started container registry
11:01:17 | openshift-image-registry | replicaset-controller | image-registry-7b5479fc47 | SuccessfulDelete | Deleted pod: image-registry-7b5479fc47-m64ps
11:01:17 | openshift-image-registry | kubelet | image-registry-7b5479fc47-m64ps | Killing | Stopping container registry
11:01:17 | openshift-image-registry | deployment-controller | image-registry | ScalingReplicaSet | Scaled down replica set image-registry-7b5479fc47 to 0
11:01:17 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/image-import-ca -n openshift-apiserver: cause by changes in data.image-registry.openshift-image-registry.svc..5000,data.image-registry.openshift-image-registry.svc.cluster.local..5000
11:01:21 | openshift-monitoring | kubelet | prometheus-k8s-1 | Killing | Stopping container prometheus
11:01:21 | openshift-monitoring | kubelet | prometheus-k8s-1 | Killing | Stopping container thanos-sidecar
11:01:21 | openshift-monitoring | kubelet | prometheus-k8s-1 | Killing | Stopping container kube-rbac-proxy
11:01:21 | openshift-monitoring | kubelet | prometheus-k8s-1 | Killing | Stopping container prom-label-proxy
11:01:21 | openshift-monitoring | statefulset-controller | prometheus-k8s | SuccessfulDelete | delete Pod prometheus-k8s-1 in StatefulSet prometheus-k8s successful
11:01:21 | openshift-monitoring | kubelet | prometheus-k8s-1 | Killing | Stopping container prometheus-proxy
11:01:23 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-target-config-controller-targetconfigcontroller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-kube-apiserver: cause by changes in data.config.yaml
11:01:24 | openshift-apiserver-operator | openshift-apiserver-operator | openshift-apiserver-operator | DeploymentUpdated | Updated Deployment.apps/apiserver -n openshift-apiserver because it changed
11:01:25 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Killing | Stopping container openshift-apiserver
11:01:25 | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Killing | Stopping container openshift-apiserver-check-endpoints
11:01:25 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled down replica set apiserver-686bc75759 to 2
11:01:25 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled up replica set apiserver-5bdfbb9984 to 1
11:01:25 | openshift-apiserver | replicaset-controller | apiserver-686bc75759 | SuccessfulDelete | Deleted pod: apiserver-686bc75759-v9g2f
11:01:25 | default | apiserver | kube-system | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
11:01:25 | openshift-apiserver | replicaset-controller | apiserver-5bdfbb9984 | SuccessfulCreate | Created pod: apiserver-5bdfbb9984-2hx9z
11:01:25 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/revision-status-5 -n openshift-kube-apiserver because it was missing
11:01:25 | default | apiserver | kube-system | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
11:01:25 | default | apiserver | kube-system | TerminationStoppedServing | Server has stopped listening
11:01:25 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Progressing changed from False to True ("OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 2, desired generation is 3.")
11:01:25 | default | apiserver | kube-system | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 0s finished
11:01:25 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Progressing message changed from "OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 2, desired generation is 3." to "OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 2, desired generation is 3.\nAPIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 1, desired generation is 2."
11:01:26 | openshift-monitoring | statefulset-controller | prometheus-k8s | SuccessfulCreate | create Pod prometheus-k8s-1 in StatefulSet prometheus-k8s successful
11:01:27 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-kzgff
11:01:27 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-pod-5 -n openshift-kube-apiserver because it was missing
11:01:27 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | DaemonSetUpdated | Updated DaemonSet.apps/controller-manager -n openshift-controller-manager because it changed
11:01:27 | openshift-controller-manager | kubelet | controller-manager-kzgff | Killing | Stopping container controller-manager
11:01:27 | openshift-controller-manager | kubelet | controller-manager-n7fv8 | Killing | Stopping container controller-manager
11:01:27 | openshift-controller-manager | kubelet | controller-manager-zcv97 | Killing | Stopping container controller-manager
11:01:27 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing changed from False to True ("Progressing: daemonset/controller-manager: observed generation is 10, desired generation is 11.\nProgressing: openshiftcontrollermanagers.operator.openshift.io/cluster: observed generation is 2, desired generation is 3.")
11:01:27 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-zcv97
11:01:27 | openshift-controller-manager-operator | openshift-controller-manager-operator | openshift-controller-manager-operator | ConfigMapUpdated | Updated ConfigMap/config -n openshift-controller-manager: cause by changes in data.config.yaml
11:01:27 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulDelete | Deleted pod: controller-manager-n7fv8
11:01:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/config-5 -n openshift-kube-apiserver because it was missing
11:01:29 | openshift-monitoring | multus | prometheus-k8s-1 | AddedInterface | Add eth0 [10.129.2.15/23]
11:01:29 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e"
11:01:30 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-cert-syncer-kubeconfig-5 -n openshift-kube-apiserver because it was missing
11:01:32 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" in 2.656052858s
11:01:32 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4"
11:01:32 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/oauth-metadata-5 -n openshift-kube-apiserver because it was missing
11:01:33 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container config-reloader
11:01:33 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container config-reloader
11:01:33 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" in 978.638222ms
11:01:33 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5"
11:01:36 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/bound-sa-token-signing-certs-5 -n openshift-kube-apiserver because it was missing
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5" in 2.172003395s
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container thanos-sidecar
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container prometheus-proxy
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container prometheus-proxy
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container kube-rbac-proxy
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container kube-rbac-proxy
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3"
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
11:01:36 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container thanos-sidecar
11:01:37 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" in 1.079634938s
11:01:38 (x2) | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container prometheus
11:01:38 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Progressing message changed from "OperatorConfigProgressing: openshiftapiserveroperatorconfigs/instance: observed generation is 2, desired generation is 3.\nAPIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 1, desired generation is 2." to "APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 1, desired generation is 2."
11:01:38 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/etcd-serving-ca-5 -n openshift-kube-apiserver because it was missing
11:01:38 | openshift-monitoring | kubelet | prometheus-k8s-1 | Created | Created container prom-label-proxy
11:01:38 (x2) | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()",Progressing changed from True to False ("")
11:01:38 | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container prom-label-proxy
11:01:38 (x2) | openshift-monitoring | kubelet | prometheus-k8s-1 | Started | Started container prometheus
11:01:38 | openshift-monitoring | kubelet | prometheus-k8s-1 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" already present on machine
11:01:40 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-server-ca-5 -n openshift-kube-apiserver because it was missing
11:01:41 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kubelet-serving-ca-5 -n openshift-kube-apiserver because it was missing
11:01:43 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/sa-token-signing-certs-5 -n openshift-kube-apiserver because it was missing
11:01:44 | openshift-monitoring | kubelet | prometheus-k8s-0 | Killing | Stopping container kube-rbac-proxy
11:01:44 | openshift-monitoring | kubelet | prometheus-k8s-0 | Killing | Stopping container prom-label-proxy
11:01:44 | openshift-monitoring | kubelet | prometheus-k8s-0 | Killing | Stopping container prometheus
11:01:44 | openshift-monitoring | statefulset-controller | prometheus-k8s | SuccessfulDelete | delete Pod prometheus-k8s-0 in StatefulSet prometheus-k8s successful
11:01:45 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | ConfigMapCreated | Created ConfigMap/kube-apiserver-audit-policies-5 -n openshift-kube-apiserver because it was missing
11:01:46 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing message changed from "Progressing: daemonset/controller-manager: observed generation is 10, desired generation is 11.\nProgressing: openshiftcontrollermanagers.operator.openshift.io/cluster: observed generation is 2, desired generation is 3." to ""
11:01:47 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/etcd-client-5 -n openshift-kube-apiserver because it was missing
11:01:49 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/kubelet-client-5 -n openshift-kube-apiserver because it was missing
11:01:50 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-serving-certkey-5 -n openshift-kube-apiserver because it was missing
11:01:52 (x2) | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionTriggered | new revision 5 triggered by "configmap/config has changed"
11:01:52 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | RevisionCreate | Revision 4 created because configmap/config has changed
11:01:52 | openshift-kube-apiserver-operator | kube-apiserver-operator-revisioncontroller | kube-apiserver-operator | SecretCreated | Created Secret/localhost-recovery-client-token-5 -n openshift-kube-apiserver because it was missing
11:01:54 (x3) | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Unhealthy | Liveness probe failed: Get "https://10.129.0.13:8443/healthz": dial tcp 10.129.0.13:8443: connect: connection refused
11:01:55 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 5" to "NodeControllerDegraded: All master nodes are ready"
11:01:55 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready\nRevisionControllerDegraded: conflicting latestAvailableRevision 5"
11:01:57 | openshift-monitoring | statefulset-controller | prometheus-k8s | SuccessfulCreate | create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s successful
11:01:59 | openshift-monitoring | multus | prometheus-k8s-0 | AddedInterface | Add eth0 [10.128.2.7/23]
11:02:00 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c8a467fbd0f848e30a51503acb8986507ebc276ca853848312e2d63817320b4" already present on machine
11:02:00 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-fz2vw
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container kube-rbac-proxy
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container thanos-sidecar
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container thanos-sidecar
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container config-reloader
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd0255e32f1829d23920e44fa3c772269c5fc013cbdbb7e4a5817c7400c6cefd" already present on machine
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container prometheus-proxy
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container prometheus-proxy
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:533973e0b3b8e64b0fd672a78ebabc5accc5f9a969c6a610ce14aa73d9685874" already present on machine
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container config-reloader
11:02:01 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14771b4fe9cbb53bc2215f99fe15a117c212446c1895cfc55c946ba0ab77eab5" already present on machine
11:02:02 | openshift-controller-manager | multus | controller-manager-fz2vw | AddedInterface | Add eth0 [10.130.0.49/23]
11:02:02 (x2) | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2fb18d00f4da6a1f4f00186b793155f231a275dd3af0e54e924a687bb16ec6e" already present on machine
11:02:02 | openshift-monitoring | kubelet | prometheus-k8s-0 | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2417c6691b2a9e14e5f682cad05931631f259fedf040468acb73db604a4dfa3" already present on machine
11:02:02 | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container prom-label-proxy
11:02:02 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-ndhqq
11:02:02 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container kube-rbac-proxy
11:02:02 | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container prom-label-proxy
11:02:03 | openshift-controller-manager | kubelet | controller-manager-fz2vw | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" already present on machine
11:02:03 (x2) | openshift-monitoring | kubelet | prometheus-k8s-0 | Started | Started container prometheus
11:02:03 | openshift-controller-manager | kubelet | controller-manager-fz2vw | Created | Created container controller-manager
11:02:03 | openshift-controller-manager | kubelet | controller-manager-fz2vw | Started | Started container controller-manager
11:02:03 (x2) | openshift-monitoring | kubelet | prometheus-k8s-0 | Created | Created container prometheus
11:02:04 | openshift-controller-manager | daemonset-controller | controller-manager | SuccessfulCreate | Created pod: controller-manager-pcvtn
11:02:05 | openshift-controller-manager | kubelet | controller-manager-ndhqq | Created | Created container controller-manager
11:02:05 | openshift-controller-manager | multus | controller-manager-ndhqq | AddedInterface | Add eth0 [10.129.0.34/23]
11:02:05 | openshift-controller-manager | kubelet | controller-manager-ndhqq | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" already present on machine
11:02:05 | openshift-controller-manager | kubelet | controller-manager-ndhqq | Started | Started container controller-manager
11:02:06 | openshift-controller-manager | multus | controller-manager-pcvtn | AddedInterface | Add eth0 [10.128.0.49/23]
11:02:06 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 4 to 5 because node master-1.case03116926.lab.upshift.rdu2.redhat.com with revision 4 is the oldest
11:02:06 | openshift-controller-manager | kubelet | controller-manager-pcvtn | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3196e390c4a47d32aa01b8aeef7bbab1e2b8dafeb5e9b13c20bf24eda2c5a697" already present on machine
11:02:07 | openshift-controller-manager | kubelet | controller-manager-pcvtn | Started | Started container controller-manager
11:02:07 | openshift-controller-manager-operator | openshift-controller-manager-operator-status-controller-statussyncer_openshift-controller-manager | openshift-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/openshift-controller-manager changed: Progressing changed from True to False ("")
11:02:07 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing changed from False to True ("NodeInstallerProgressing: 3 nodes are at revision 4; 0 nodes have achieved new revision 5"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4; 0 nodes have achieved new revision 5"
11:02:07 | openshift-controller-manager | kubelet | controller-manager-pcvtn | Created | Created container controller-manager
11:02:11 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
11:02:14 | openshift-kube-apiserver | multus | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.50/23]
11:02:15 | openshift-kube-apiserver | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:02:16 | openshift-kube-apiserver | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
11:02:16 | openshift-kube-apiserver | kubelet | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
11:02:18 (x6) | openshift-apiserver | kubelet | apiserver-686bc75759-v9g2f | Unhealthy | Readiness probe failed: Get "https://10.129.0.13:8443/healthz": dial tcp 10.129.0.13:8443: connect: connection refused
11:02:23 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
11:02:23 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
11:02:23 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
11:02:23 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
11:02:23 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
11:02:23 | openshift-kube-apiserver | static-pod-installer | installer-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 5
11:02:25 | default | apiserver | kube-system | TerminationGracefulTerminationFinished | All pending requests processed
11:02:37 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
11:02:37 | openshift-apiserver | multus | apiserver-5bdfbb9984-2hx9z | AddedInterface | Add eth0 [10.129.0.35/23]
11:02:37 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (3 containers are waiting in pending apiserver-5bdfbb9984-2hx9z pod)"
11:02:38 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Created | Created container fix-audit-permissions
11:02:38 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Started | Started container fix-audit-permissions
11:02:39 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:02:39 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Created | Created container openshift-apiserver-check-endpoints
11:02:39 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Started | Started container openshift-apiserver-check-endpoints
11:02:39 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Created | Created container openshift-apiserver
11:02:39 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Started | Started container openshift-apiserver
11:02:39 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-2hx9z | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
11:02:47 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled down replica set apiserver-686bc75759 to 1
11:02:47 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Killing | Stopping container openshift-apiserver-check-endpoints
11:02:47 | default | apiserver | kube-system | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
11:02:47 | default | apiserver | kube-system | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
11:02:47 | openshift-apiserver | replicaset-controller | apiserver-686bc75759 | SuccessfulDelete | Deleted pod: apiserver-686bc75759-77rhg
11:02:47 | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Killing | Stopping container openshift-apiserver
11:02:48 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled up replica set apiserver-5bdfbb9984 to 2
11:02:48 | openshift-apiserver | replicaset-controller | apiserver-5bdfbb9984 | SuccessfulCreate | Created pod: apiserver-5bdfbb9984-k7b4h
11:02:48 | default | apiserver | kube-system | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 0s finished
11:02:48 | default | apiserver | kube-system | TerminationStoppedServing | Server has stopped listening
11:02:49 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (3 containers are waiting in pending apiserver-5bdfbb9984-2hx9z pod)" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()"
11:03:06 | openshift-controller-manager | openshift-controller-manager | openshift-master-controllers | LeaderElection | controller-manager-fz2vw became leader
11:03:11 (x3) | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Unhealthy | Liveness probe failed: Get "https://10.130.0.25:8443/healthz": dial tcp 10.130.0.25:8443: connect: connection refused
11:03:38 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded changed from False to True ("APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()")
11:03:38 | openshift-marketplace | kubelet | redhat-marketplace-vsvkh | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
11:03:38 | openshift-marketplace | multus | redhat-marketplace-vsvkh | AddedInterface | Add eth0 [10.129.2.16/23]
11:03:39 | openshift-marketplace | kubelet | redhat-marketplace-vsvkh | Started | Started container registry-server
11:03:39 | openshift-marketplace | kubelet | redhat-marketplace-vsvkh | Created | Created container registry-server
11:03:39 | openshift-marketplace | kubelet | redhat-marketplace-vsvkh | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 442.691093ms
11:03:47 (x6) | openshift-apiserver | kubelet | apiserver-686bc75759-77rhg | Unhealthy | Readiness probe failed: Get "https://10.130.0.25:8443/healthz": dial tcp 10.130.0.25:8443: connect: connection refused
11:03:48 | default | apiserver | kube-system | TerminationGracefulTerminationFinished | All pending requests processed
11:03:51 (x2) | openshift-marketplace | kubelet | redhat-marketplace-vsvkh | Killing | Stopping container registry-server
11:03:57 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
11:03:57 | openshift-apiserver | multus | apiserver-5bdfbb9984-k7b4h | AddedInterface | Add eth0 [10.130.0.50/23]
11:03:58 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
11:03:58 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Created | Created container fix-audit-permissions
11:03:58 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Started | Started container fix-audit-permissions
11:03:59 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Started | Started container openshift-apiserver
11:03:59 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Started | Started container openshift-apiserver-check-endpoints
11:03:59 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Created | Created container openshift-apiserver
11:03:59 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Created | Created container openshift-apiserver-check-endpoints
11:03:59 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:04:00 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (container is not ready in apiserver-5bdfbb9984-k7b4h pod)"
11:04:01 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-k7b4h | Unhealthy | Readiness probe failed: Get "https://10.130.0.50:8443/healthz": context deadline exceeded
11:04:10 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled down replica set apiserver-686bc75759 to 0
11:04:11 | default | apiserver | kube-system | TerminationStart | Received signal to terminate, becoming unready, but keeping serving
11:04:11 | default | apiserver | kube-system | TerminationPreShutdownHooksFinished | All pre-shutdown hooks have been finished
11:04:11 | default | apiserver | kube-system | TerminationMinimalShutdownDurationFinished | The minimal shutdown duration of 0s finished
11:04:11 | openshift-apiserver | deployment-controller | apiserver | ScalingReplicaSet | Scaled up replica set apiserver-5bdfbb9984 to 3
11:04:11 | openshift-apiserver | replicaset-controller | apiserver-5bdfbb9984 | SuccessfulCreate | Created pod: apiserver-5bdfbb9984-6rjfg
11:04:11 | default | apiserver | kube-system | TerminationStoppedServing | Server has stopped listening
11:04:11 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Killing | Stopping container openshift-apiserver-check-endpoints
11:04:11 | openshift-apiserver | replicaset-controller | apiserver-686bc75759 | SuccessfulDelete | Deleted pod: apiserver-686bc75759-tbpkl
11:04:11 | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Killing | Stopping container openshift-apiserver
11:04:13 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (container is not ready in apiserver-5bdfbb9984-k7b4h pod)" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()"
11:04:18 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
11:04:18 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
11:04:18 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
11:04:35 (x3) | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Unhealthy | Liveness probe failed: Get "https://10.128.0.32:8443/healthz": dial tcp 10.128.0.32:8443: connect: connection refused
11:04:41 | openshift-kube-controller-manager | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-2.case03116926.lab.upshift.rdu2.redhat.com_5fbc95ca-4f05-4ca1-9ce9-9e306a921d2a became leader
11:04:44 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:04:45 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
11:04:46 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
11:04:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
11:04:47 | openshift-kube-apiserver | kubelet | kube-apiserver-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
11:05:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 3 nodes are at revision 4; 0 nodes have achieved new revision 5" to "NodeInstallerProgressing: 2 nodes are at revision 4; 1 nodes are at revision 5",Available message changed from "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 4; 0 nodes have achieved new revision 5" to "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 4; 1 nodes are at revision 5"
11:05:04 (x6) | openshift-apiserver | kubelet | apiserver-686bc75759-tbpkl | Unhealthy | Readiness probe failed: Get "https://10.128.0.32:8443/healthz": dial tcp 10.128.0.32:8443: connect: connection refused
11:05:04 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-1.case03116926.lab.upshift.rdu2.redhat.com" from revision 4 to 5 because static pod is ready
11:05:08 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
11:05:10 | openshift-kube-apiserver | multus | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.128.0.51/23]
11:05:11 | default | apiserver | kube-system | TerminationGracefulTerminationFinished | All pending requests processed
11:05:12 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:05:13 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
11:05:13 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-1.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
11:05:23 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 4 to 5 because node master-2.case03116926.lab.upshift.rdu2.redhat.com with revision 4 is the oldest
11:05:23 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (2 containers are waiting in apiserver-686bc75759-tbpkl pod)"
11:05:28 | openshift-apiserver | multus | apiserver-5bdfbb9984-6rjfg | AddedInterface | Add eth0 [10.128.0.52/23]
11:05:28 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
11:05:28 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
11:05:28 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Started | Started container fix-audit-permissions
11:05:28 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Created | Created container fix-audit-permissions
11:05:29 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24334ae0354b28b6c3dc830871fe6ded2c257ecaa5794f4dca075f86b6504333" already present on machine
11:05:29 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:05:29 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Created | Created container openshift-apiserver
11:05:29 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Started | Started container openshift-apiserver
11:05:30 | openshift-kube-apiserver | multus | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.36/23]
11:05:30 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Created | Created container openshift-apiserver-check-endpoints
11:05:30 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Started | Started container openshift-apiserver-check-endpoints
11:05:31 | openshift-apiserver | kubelet | apiserver-5bdfbb9984-6rjfg | Unhealthy | Readiness probe failed: Get "https://10.128.0.52:8443/healthz": context deadline exceeded
11:05:32 | openshift-kube-apiserver | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:05:33 | openshift-kube-apiserver | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
11:05:33 | openshift-kube-apiserver | kubelet | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
11:05:35 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (2 containers are waiting in apiserver-686bc75759-tbpkl pod)" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver (container is not ready in apiserver-5bdfbb9984-6rjfg pod)"
11:05:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
11:05:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
11:05:40 | openshift-kube-apiserver | static-pod-installer | installer-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 5
11:05:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
11:05:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
11:05:40 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
11:05:42 | openshift-kube-apiserver | cert-regeneration-controller | cert-regeneration-controller-lock | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_86923d67-579f-46fd-a216-ded3d63eba4d became leader
11:05:47 | openshift-apiserver-operator | openshift-apiserver-operator-status-controller-statussyncer_openshift-apiserver | openshift-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/openshift-apiserver changed: Degraded changed from True to False ("")
11:07:27 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
11:07:27 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
11:07:27 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pod/kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is terminated: Completed: \nNodeControllerDegraded: All master nodes are ready"
11:07:27 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
11:07:30 | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pod/kube-controller-manager-master-2.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is terminated: Completed: \nNodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready"
11:07:31 (x2) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-scheduler-recovery-controller
11:07:31 (x2) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-scheduler-recovery-controller
11:07:31 (x2) | openshift-kube-scheduler | kubelet | openshift-kube-scheduler-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:47ec92433a059a3adf5abf3216a3ffc3e76b45056355dc2a06d3e5fd13cf8369" already present on machine
11:07:47 | openshift-kube-controller-manager | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-0.case03116926.lab.upshift.rdu2.redhat.com_5a3eda3d-ed64-42c5-a50f-3b4683d2ccea became leader
11:07:52 | openshift-kube-scheduler | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_8962c510-2df8-4077-bd3b-7084160de189 became leader
11:07:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
11:07:54 | openshift-marketplace | multus | community-operators-p6njw | AddedInterface | Add eth0 [10.129.2.17/23]
11:07:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
11:07:54 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
11:07:54 | openshift-marketplace | multus | certified-operators-cbbm2 | AddedInterface | Add eth0 [10.129.2.18/23]
11:07:55 | openshift-marketplace | kubelet | community-operators-p6njw | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
11:07:55 | openshift-marketplace | kubelet | certified-operators-cbbm2 | Created | Created container registry-server
11:07:55 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
11:07:55 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
11:07:55 | openshift-marketplace | kubelet | certified-operators-cbbm2 | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 810.32615ms
11:07:55 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:07:55 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
11:07:55 | openshift-marketplace | kubelet | certified-operators-cbbm2 | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
11:07:56 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
11:07:56 | openshift-marketplace | kubelet | certified-operators-cbbm2 | Started | Started container registry-server
11:07:56 | openshift-marketplace | kubelet | community-operators-p6njw | Created | Created container registry-server
11:07:56 | openshift-marketplace | kubelet | community-operators-p6njw | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 798.040229ms
11:07:56 | openshift-marketplace | kubelet | community-operators-p6njw | Started | Started container registry-server
11:07:56 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
11:07:56 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
11:07:56 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
11:07:56 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:07:56 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:07:57 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
11:07:57 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
11:07:57 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:07:58 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
11:07:58 | openshift-kube-apiserver | kubelet | kube-apiserver-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
11:08:07 (x2) | openshift-marketplace | kubelet | community-operators-p6njw | Killing | Stopping container registry-server
11:08:10 (x2) | openshift-marketplace | kubelet | certified-operators-cbbm2 | Killing | Stopping container registry-server
11:08:22 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing message changed from "NodeInstallerProgressing: 2 nodes are at revision 4; 1 nodes are at revision 5" to "NodeInstallerProgressing: 1 nodes are at revision 4; 2 nodes are at revision 5",Available message changed from "StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 4; 1 nodes are at revision 5" to "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 4; 2 nodes are at revision 5"
11:08:22 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-2.case03116926.lab.upshift.rdu2.redhat.com" from revision 4 to 5 because static pod is ready
11:08:29 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
11:08:31 | openshift-kube-apiserver | multus | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.129.0.37/23]
11:08:33 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:08:34 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
11:08:34 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-2.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
11:08:35 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeTargetRevisionChanged | Updating node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 4 to 5 because node master-0.case03116926.lab.upshift.rdu2.redhat.com with revision 4 is the oldest
11:08:51 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | PodCreated | Created Pod/installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
11:08:53 | openshift-kube-apiserver | multus | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.51/23]
11:08:55 | openshift-kube-apiserver | kubelet | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container installer
11:08:55 | openshift-kube-apiserver | kubelet | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:08:55 | openshift-kube-apiserver | kubelet | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container installer
11:09:03 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-regeneration-controller
11:09:03 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-cert-syncer
11:09:03 | openshift-kube-apiserver | static-pod-installer | installer-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | StaticPodInstallerCompleted | Successfully installed revision 5
11:09:03 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver
11:09:03 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-insecure-readyz
11:09:03 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Killing | Stopping container kube-apiserver-check-endpoints
11:10:53 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a14d07ddcc0c0dfbbda41e18087dcaef4c36de4af4f62df526d878b79bd8b8a" already present on machine
11:10:54 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-controller-manager-recovery-controller
11:10:54 (x2) | openshift-kube-controller-manager | kubelet | kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-controller-manager-recovery-controller
11:10:55 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "NodeControllerDegraded: All master nodes are ready" to "StaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is terminated: Completed: \nNodeControllerDegraded: All master nodes are ready"
11:10:58 (x2) | openshift-kube-controller-manager-operator | kube-controller-manager-operator-status-controller-statussyncer_kube-controller-manager | kube-controller-manager-operator | OperatorStatusChanged | Status for clusteroperator/kube-controller-manager changed: Degraded message changed from "StaticPodsDegraded: pod/kube-controller-manager-master-0.case03116926.lab.upshift.rdu2.redhat.com container \"kube-controller-manager-recovery-controller\" is terminated: Completed: \nNodeControllerDegraded: All master nodes are ready" to "NodeControllerDegraded: All master nodes are ready"
11:11:14 (x3) | openshift-kube-apiserver | endpoint-controller | apiserver | FailedToUpdateEndpoint | Failed to update endpoint openshift-kube-apiserver/apiserver: Operation cannot be fulfilled on endpoints "apiserver": the object has been modified; please apply your changes to the latest version and try again
11:11:14 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
11:11:15 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container setup
11:11:15 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container setup
11:11:15 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7cd85d223d9b8249e4773998db865454043be8231589a59a1f21ea8e9e66b1fe" already present on machine
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:11:16 | openshift-marketplace | kubelet | redhat-operators-z4lwg | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-regeneration-controller
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-regeneration-controller
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:11:16 | openshift-marketplace | multus | redhat-operators-z4lwg | AddedInterface | Add eth0 [10.129.2.19/23]
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-cert-syncer
11:11:16 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-cert-syncer
11:11:17 | openshift-marketplace | kubelet | redhat-operators-z4lwg | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 475.793548ms
11:11:17 | openshift-marketplace | kubelet | redhat-operators-z4lwg | Created | Created container registry-server
11:11:17 | openshift-marketplace | kubelet | redhat-operators-z4lwg | Started | Started container registry-server
11:11:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-insecure-readyz
11:11:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-insecure-readyz
11:11:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:11:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container kube-apiserver-check-endpoints
11:11:17 | openshift-kube-apiserver | kubelet | kube-apiserver-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container kube-apiserver-check-endpoints
11:11:20 | openshift-kube-controller-manager | cert-recovery-controller | cert-recovery-controller-lock | LeaderElection | master-1.case03116926.lab.upshift.rdu2.redhat.com_bf06ed15-60b6-4ddf-abf1-6bde7cab0edc became leader
11:11:30 (x2) | openshift-marketplace | kubelet | redhat-operators-z4lwg | Killing | Stopping container registry-server
11:11:40 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | NodeCurrentRevisionChanged | Updated node "master-0.case03116926.lab.upshift.rdu2.redhat.com" from revision 4 to 5 because static pod is ready
11:11:40 | openshift-kube-apiserver-operator | kube-apiserver-operator-status-controller-statussyncer_kube-apiserver | kube-apiserver-operator | OperatorStatusChanged | Status for clusteroperator/kube-apiserver changed: Progressing changed from True to False ("NodeInstallerProgressing: 3 nodes are at revision 5"),Available message changed from "StaticPodsAvailable: 3 nodes are active; 1 nodes are at revision 4; 2 nodes are at revision 5" to "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 5"
11:11:42 | openshift-kube-apiserver-operator | kube-apiserver-operator-installer-controller | kube-apiserver-operator | ConfigMapUpdated | Updated ConfigMap/revision-status-5 -n openshift-kube-apiserver: cause by changes in data.status
11:11:49 | openshift-kube-apiserver-operator | kube-apiserver-operator-prunecontroller | kube-apiserver-operator | PodCreated | Created Pod/revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com -n openshift-kube-apiserver because it was missing
11:11:51 | openshift-kube-apiserver | multus | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | AddedInterface | Add eth0 [10.130.0.52/23]
11:11:53 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38b0061ea15ace519955e223ba3be004d9e8c71c95bd3c9155b33f028fbef2c5" already present on machine
11:11:54 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Started | Started container pruner
11:11:54 | openshift-kube-apiserver | kubelet | revision-pruner-5-master-0.case03116926.lab.upshift.rdu2.redhat.com | Created | Created container pruner
11:14:47 | openshift-marketplace | job-controller | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9ae165da | SuccessfulCreate | Created pod: 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn
11:14:50 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082"
11:14:50 | openshift-marketplace | multus | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | AddedInterface | Add eth0 [10.129.2.20/23]
11:14:55 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082" in 4.826651129s
11:14:55 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Started | Started container util
11:14:55 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Created | Created container util
11:14:56 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Pulling | Pulling image "registry.redhat.io/fuse7/fuse-online-rhel8-operator-bundle@sha256:f17f79a069e4c85ac9645336cecaeb3ed90dcac7a4a5d65da403bbfbf89d8d34"
11:14:57 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Pulled | Successfully pulled image "registry.redhat.io/fuse7/fuse-online-rhel8-operator-bundle@sha256:f17f79a069e4c85ac9645336cecaeb3ed90dcac7a4a5d65da403bbfbf89d8d34" in 1.171251203s
11:14:57 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Started | Started container pull
11:14:57 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Created | Created container pull
11:14:58 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Pulling | Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f46a215914a6a4746aaecba62dc1e37984e2abc1e091fe6d9ddfc2237e6d24cf"
11:15:06 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Pulled | Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f46a215914a6a4746aaecba62dc1e37984e2abc1e091fe6d9ddfc2237e6d24cf" in 8.315515343s
11:15:07 | openshift-marketplace | job-controller | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9ae165da | Completed | Job completed
11:15:07 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Created | Created container extract
11:15:07 | openshift-marketplace | kubelet | 773bb0bf6441295ddf0539ac7c6b3a1ef9b58824556371a30fec3af9aerllcn | Started | Started container extract
11:15:08 | openshift-marketplace | multus | redhat-marketplace-kw79d | AddedInterface | Add eth0 [10.129.2.21/23]
11:15:09 | openshift-marketplace | kubelet | redhat-marketplace-kw79d | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 660.162947ms
11:15:09 | openshift-marketplace | kubelet | redhat-marketplace-kw79d | Created | Created container registry-server
11:15:09 | openshift-marketplace | kubelet | redhat-marketplace-kw79d | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
11:15:10 | openshift-marketplace | kubelet | redhat-marketplace-kw79d | Started | Started container registry-server
11:15:14 (x2) | openshift-marketplace | kubelet | redhat-marketplace-kw79d | Killing | Stopping container registry-server
11:16:19 | openshift-marketplace | job-controller | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9ff135 | SuccessfulCreate | Created pod: d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q
11:16:21 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0a87f0d70052675bad4a3b3753e801fdf71e0ea329d9ca84b5d107f649fe082" already present on machine
11:16:21 | openshift-marketplace | multus | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | AddedInterface | Add eth0 [10.129.2.33/23]
11:16:22 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Started | Started container util
11:16:22 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Pulling | Pulling image "registry.redhat.io/rhosdt/jaeger-operator-bundle@sha256:6d2fba302f7dc06d00d85fdea0a68105a4495111b59a3fad95d4bdd28bf24ee1"
11:16:22 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Created | Created container util
11:16:25 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Started | Started container pull
11:16:25 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Pulled | Successfully pulled image "registry.redhat.io/rhosdt/jaeger-operator-bundle@sha256:6d2fba302f7dc06d00d85fdea0a68105a4495111b59a3fad95d4bdd28bf24ee1" in 2.352343675s
11:16:25 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Created | Created container pull
11:16:31 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f46a215914a6a4746aaecba62dc1e37984e2abc1e091fe6d9ddfc2237e6d24cf" already present on machine
11:16:43 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Started | Started container extract
11:16:43 | openshift-marketplace | kubelet | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9pgm4q | Created | Created container extract
11:16:44 | openshift-marketplace | job-controller | d38844bcf28e738b69391b139dfe6048008f0d2b2d91d437d57445bbb9ff135 | Completed | Job completed
11:18:37 | openshift-marketplace | kubelet | community-operators-wmcq5 | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
11:18:37 | openshift-marketplace | multus | community-operators-wmcq5 | AddedInterface | Add eth0 [10.129.2.35/23]
11:18:38 | openshift-marketplace | kubelet | community-operators-wmcq5 | Started | Started container registry-server
11:18:38 | openshift-marketplace | kubelet | community-operators-wmcq5 | Created | Created container registry-server
11:18:38 | openshift-marketplace | kubelet | community-operators-wmcq5 | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 754.220379ms
11:18:44 | openshift-marketplace | multus | certified-operators-vgmxf | AddedInterface | Add eth0 [10.129.2.36/23]
11:18:44 | openshift-marketplace | kubelet | certified-operators-vgmxf | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
11:18:45 | openshift-marketplace | kubelet | certified-operators-vgmxf | Started | Started container registry-server
11:18:45 | openshift-marketplace | kubelet | certified-operators-vgmxf | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 737.335576ms
11:18:45 | openshift-marketplace | kubelet | certified-operators-vgmxf | Created | Created container registry-server
11:18:48 (x2) | openshift-marketplace | kubelet | community-operators-wmcq5 | Killing | Stopping container registry-server
11:18:59 (x2) | openshift-marketplace | kubelet | certified-operators-vgmxf | Killing | Stopping container registry-server
11:22:23 | openshift-marketplace | kubelet | redhat-operators-zrw9j | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
11:22:23 | openshift-marketplace | multus | redhat-operators-zrw9j | AddedInterface | Add eth0 [10.129.2.37/23]
11:22:24 | openshift-marketplace | kubelet | redhat-operators-zrw9j | Created | Created container registry-server
11:22:24 | openshift-marketplace | kubelet | redhat-operators-zrw9j | Started | Started container registry-server
11:22:24 | openshift-marketplace | kubelet | redhat-operators-zrw9j | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 709.480206ms
11:22:32 (x2) | openshift-marketplace | kubelet | redhat-operators-zrw9j | Killing | Stopping container registry-server
11:26:55 | openshift-marketplace | multus | redhat-marketplace-qs42c | AddedInterface | Add eth0 [10.129.2.38/23]
11:26:55 | openshift-marketplace | kubelet | redhat-marketplace-qs42c | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
11:26:56 | openshift-marketplace | kubelet | redhat-marketplace-qs42c | Started | Started container registry-server
11:26:56 | openshift-marketplace | kubelet | redhat-marketplace-qs42c | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 808.6548ms
11:26:56 | openshift-marketplace | kubelet | redhat-marketplace-qs42c | Created | Created container registry-server
11:27:02 (x2) | openshift-marketplace | kubelet | redhat-marketplace-qs42c | Killing | Stopping container registry-server
11:28:55 | openshift-marketplace | kubelet | community-operators-svx4n | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
11:28:55 | openshift-marketplace | multus | community-operators-svx4n | AddedInterface | Add eth0 [10.129.2.39/23]
11:28:56 | openshift-marketplace | kubelet | community-operators-svx4n | Started | Started container registry-server
11:28:56 | openshift-marketplace | kubelet | certified-operators-slfsz | Started | Started container registry-server
11:28:56 | openshift-marketplace | kubelet | community-operators-svx4n | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 444.693874ms
11:28:56 | openshift-marketplace | kubelet | certified-operators-slfsz | Created | Created container registry-server
11:28:56 | openshift-marketplace | multus | certified-operators-slfsz | AddedInterface | Add eth0 [10.129.2.40/23]
11:28:56 | openshift-marketplace | kubelet | certified-operators-slfsz | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
11:28:56 | openshift-marketplace | kubelet | certified-operators-slfsz | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 458.1738ms
11:28:56 | openshift-marketplace | kubelet | community-operators-svx4n | Created | Created container registry-server
11:29:02 (x2) | openshift-marketplace | kubelet | certified-operators-slfsz | Killing | Stopping container registry-server
11:29:04 (x2) | openshift-marketplace | kubelet | community-operators-svx4n | Killing | Stopping container registry-server
11:32:49 | openshift-marketplace | multus | redhat-operators-tn8wx | AddedInterface | Add eth0 [10.129.2.41/23]
11:32:49 | openshift-marketplace | kubelet | redhat-operators-tn8wx | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
11:32:51 | openshift-marketplace | kubelet | redhat-operators-tn8wx | Started | Started container registry-server
11:32:51 | openshift-marketplace | kubelet | redhat-operators-tn8wx | Created | Created container registry-server
11:32:51 | openshift-marketplace | kubelet | redhat-operators-tn8wx | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 1.561651245s
11:33:02 (x2) | openshift-marketplace | kubelet | redhat-operators-tn8wx | Killing | Stopping container registry-server
11:37:13 | openshift-marketplace | multus | redhat-marketplace-2g7kf | AddedInterface | Add eth0 [10.129.2.42/23]
11:37:13 | openshift-marketplace | kubelet | redhat-marketplace-2g7kf | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
11:37:14 | openshift-marketplace | kubelet | redhat-marketplace-2g7kf | Started | Started container registry-server
11:37:14 | openshift-marketplace | kubelet | redhat-marketplace-2g7kf | Created | Created container registry-server
11:37:14 | openshift-marketplace | kubelet | redhat-marketplace-2g7kf | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 850.415572ms
11:37:26 (x2) | openshift-marketplace | kubelet | redhat-marketplace-2g7kf | Killing | Stopping container registry-server
11:40:00 | openshift-marketplace | kubelet | certified-operators-9zcz9 | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 439.804401ms
11:40:00 | openshift-marketplace | multus | certified-operators-9zcz9 | AddedInterface | Add eth0 [10.129.2.43/23]
11:40:00 | openshift-marketplace | kubelet | certified-operators-9zcz9 | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
11:40:01 | openshift-marketplace | kubelet | certified-operators-9zcz9 | Started | Started container registry-server
11:40:01 | openshift-marketplace | kubelet | certified-operators-9zcz9 | Created | Created container registry-server
11:40:10 (x2) | openshift-marketplace | kubelet | certified-operators-9zcz9 | Killing | Stopping container registry-server
11:40:19 | openshift-marketplace | multus | community-operators-pqzhm | AddedInterface | Add eth0 [10.129.2.44/23]
11:40:20 | openshift-marketplace | kubelet | community-operators-pqzhm | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
11:40:21 | openshift-marketplace | kubelet | community-operators-pqzhm | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 1.165427114s
11:40:21 | openshift-marketplace | kubelet | community-operators-pqzhm | Created | Created container registry-server
11:40:21 | openshift-marketplace | kubelet | community-operators-pqzhm | Started | Started container registry-server
11:40:30 (x2) | openshift-marketplace | kubelet | community-operators-pqzhm | Killing | Stopping container registry-server
11:46:34 | openshift-marketplace | multus | redhat-operators-dth47 | AddedInterface | Add eth0 [10.129.2.45/23]
11:46:34 | openshift-marketplace | kubelet | redhat-operators-dth47 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
11:46:35 | openshift-marketplace | kubelet | redhat-operators-dth47 | Created | Created container registry-server
11:46:35 | openshift-marketplace | kubelet | redhat-operators-dth47 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 478.039861ms
11:46:35 | openshift-marketplace | kubelet | redhat-operators-dth47 | Started | Started container registry-server
11:46:49 (x2) | openshift-marketplace | kubelet | redhat-operators-dth47 | Killing | Stopping container registry-server
11:50:27 | openshift-marketplace | multus | redhat-marketplace-r25h2 | AddedInterface | Add eth0 [10.129.2.48/23]
11:50:28 | openshift-marketplace | multus | certified-operators-498r2 | AddedInterface | Add eth0 [10.129.2.46/23]
11:50:28 | openshift-marketplace | kubelet | redhat-marketplace-r25h2 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
11:50:28 | openshift-marketplace | multus | community-operators-cr2bj | AddedInterface | Add eth0 [10.129.2.47/23]
11:50:28 | openshift-marketplace | kubelet | certified-operators-498r2 | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
11:50:28 | openshift-marketplace | kubelet | community-operators-cr2bj | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
11:50:29 | openshift-marketplace | kubelet | community-operators-cr2bj | Started | Started container registry-server
11:50:29 | openshift-marketplace | kubelet | community-operators-cr2bj | Created | Created container registry-server
11:50:29 | openshift-marketplace | kubelet | redhat-marketplace-r25h2 | Started | Started container registry-server
11:50:29 | openshift-marketplace | kubelet | redhat-marketplace-r25h2 | Created | Created container registry-server
11:50:29 | openshift-marketplace | kubelet | certified-operators-498r2 | Started | Started container registry-server
11:50:29 | openshift-marketplace | kubelet | certified-operators-498r2 | Created | Created container registry-server
11:50:29 | openshift-marketplace | kubelet | certified-operators-498r2 | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 759.256919ms
11:50:29 | openshift-marketplace | kubelet | community-operators-cr2bj | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 840.546199ms
11:50:29 | openshift-marketplace | kubelet | redhat-marketplace-r25h2 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 830.726545ms
11:50:41 (x2) | openshift-marketplace | kubelet | certified-operators-498r2 | Killing | Stopping container registry-server
11:50:42 (x2) | openshift-marketplace | kubelet | redhat-marketplace-r25h2 | Killing | Stopping container registry-server
11:50:43 (x2) | openshift-marketplace | kubelet | community-operators-cr2bj | Killing | Stopping container registry-server
11:56:37 | openshift-marketplace | multus | redhat-operators-wsvbq | AddedInterface | Add eth0 [10.129.2.49/23]
11:56:37 | openshift-marketplace | kubelet | redhat-operators-wsvbq | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
11:56:38 | openshift-marketplace | kubelet | redhat-operators-wsvbq | Created | Created container registry-server
11:56:38 | openshift-marketplace | kubelet | redhat-operators-wsvbq | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 560.296661ms
11:56:38 | openshift-marketplace | kubelet | redhat-operators-wsvbq | Started | Started container registry-server
11:56:44 (x2) | openshift-marketplace | kubelet | redhat-operators-wsvbq | Killing | Stopping container registry-server
12:01:02 | openshift-marketplace | kubelet | redhat-marketplace-6xsjw | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
12:01:02 | openshift-marketplace | multus | redhat-marketplace-6xsjw | AddedInterface | Add eth0 [10.129.2.50/23]
12:01:03 | openshift-marketplace | kubelet | redhat-marketplace-6xsjw | Created | Created container registry-server
12:01:03 | openshift-marketplace | kubelet | redhat-marketplace-6xsjw | Started | Started container registry-server
12:01:03 | openshift-marketplace | kubelet | redhat-marketplace-6xsjw | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 872.143752ms
12:01:10 (x2) | openshift-marketplace | kubelet | redhat-marketplace-6xsjw | Killing | Stopping container registry-server
12:04:12 | openshift-marketplace | multus | certified-operators-vp4mv | AddedInterface | Add eth0 [10.129.2.51/23]
12:04:13 | openshift-marketplace | multus | community-operators-9f85q | AddedInterface | Add eth0 [10.129.2.52/23]
12:04:13 | openshift-marketplace | kubelet | certified-operators-vp4mv | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
12:04:13 | openshift-marketplace | kubelet | community-operators-9f85q | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
12:04:14 | openshift-marketplace | kubelet | certified-operators-vp4mv | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 818.462735ms
12:04:14 | openshift-marketplace | kubelet | community-operators-9f85q | Started | Started container registry-server
12:04:14 | openshift-marketplace | kubelet | certified-operators-vp4mv | Started | Started container registry-server
12:04:14 | openshift-marketplace | kubelet | community-operators-9f85q | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 700.078171ms
12:04:14 | openshift-marketplace | kubelet | community-operators-9f85q | Created | Created container registry-server
12:04:14 | openshift-marketplace | kubelet | certified-operators-vp4mv | Created | Created container registry-server
12:04:23 (x2) | openshift-marketplace | kubelet | certified-operators-vp4mv | Killing | Stopping container registry-server
12:04:28 (x2) | openshift-marketplace | kubelet | community-operators-9f85q | Killing | Stopping container registry-server
12:07:25 | openshift-marketplace | multus | redhat-operators-9fsfr | AddedInterface | Add eth0 [10.129.2.53/23]
12:07:26 | openshift-marketplace | kubelet | redhat-operators-9fsfr | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
12:07:29 | openshift-marketplace | kubelet | redhat-operators-9fsfr | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 3.759347933s
12:07:30 | openshift-marketplace | kubelet | redhat-operators-9fsfr | Created | Created container registry-server
12:07:30 | openshift-marketplace | kubelet | redhat-operators-9fsfr | Started | Started container registry-server
12:07:44 (x2) | openshift-marketplace | kubelet | redhat-operators-hr4dq | Killing | Stopping container registry-server
12:11:51 | openshift-marketplace | kubelet | redhat-marketplace-27l28 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
12:11:51 | openshift-marketplace | multus | redhat-marketplace-27l28 | AddedInterface | Add eth0 [10.129.2.54/23]
12:11:52 | openshift-marketplace | kubelet | redhat-marketplace-27l28 | Created | Created container registry-server
12:11:52 | openshift-marketplace | kubelet | redhat-marketplace-27l28 | Started | Started container registry-server
12:11:52 | openshift-marketplace | kubelet | redhat-marketplace-27l28 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 433.127552ms
12:11:58 (x2) | openshift-marketplace | kubelet | redhat-marketplace-27l28 | Killing | Stopping container registry-server
12:15:01 | openshift-marketplace | multus | community-operators-zqhdc | AddedInterface | Add eth0 [10.129.2.55/23]
12:15:02 | openshift-marketplace | kubelet | community-operators-zqhdc | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
12:15:02 | openshift-marketplace | kubelet | community-operators-zqhdc | Created | Created container registry-server
12:15:02 | openshift-marketplace | kubelet | community-operators-zqhdc | Started | Started container registry-server
12:15:02 | openshift-marketplace | kubelet | community-operators-zqhdc | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 427.029915ms
12:15:12 (x2) | openshift-marketplace | kubelet | community-operators-zqhdc | Killing | Stopping container registry-server
12:15:36 | openshift-marketplace | multus | certified-operators-8wkgf | AddedInterface | Add eth0 [10.129.2.56/23]
12:15:36 | openshift-marketplace | kubelet | certified-operators-8wkgf | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
12:15:37 | openshift-marketplace | kubelet | certified-operators-8wkgf | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 497.618891ms
12:15:37 | openshift-marketplace | kubelet | certified-operators-8wkgf | Started | Started container registry-server
12:15:37 | openshift-marketplace | kubelet | certified-operators-8wkgf | Created | Created container registry-server
12:15:50 (x2) | openshift-marketplace | kubelet | certified-operators-8wkgf | Killing | Stopping container registry-server
12:21:51 | openshift-marketplace | kubelet | redhat-marketplace-7wcsb | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
12:21:51 | openshift-marketplace | multus | redhat-marketplace-7wcsb | AddedInterface | Add eth0 [10.129.2.58/23]
12:21:52 | openshift-marketplace | multus | redhat-operators-grgl5 | AddedInterface | Add eth0 [10.129.2.57/23]
12:21:52 | openshift-marketplace | kubelet | redhat-operators-grgl5 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
12:21:53 | openshift-marketplace | kubelet | redhat-operators-grgl5 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 926.836622ms
12:21:53 | openshift-marketplace | kubelet | redhat-operators-grgl5 | Started | Started container registry-server
12:21:53 | openshift-marketplace | kubelet | redhat-operators-grgl5 | Created | Created container registry-server
12:21:54 | openshift-marketplace | kubelet | redhat-marketplace-7wcsb | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 3.204559265s
12:21:54 | openshift-marketplace | kubelet | redhat-marketplace-7wcsb | Created | Created container registry-server
12:21:54 | openshift-marketplace | kubelet | redhat-marketplace-7wcsb | Started | Started container registry-server
12:22:03 | openshift-marketplace | endpoint-controller | redhat-marketplace | FailedToUpdateEndpoint | Failed to update endpoint openshift-marketplace/redhat-marketplace: Operation cannot be fulfilled on endpoints "redhat-marketplace": the object has been modified; please apply your changes to the latest version and try again
12:22:03 (x2) | openshift-marketplace | kubelet | redhat-marketplace-xm8bs | Killing | Stopping container registry-server
12:22:07 (x2) | openshift-marketplace | kubelet | redhat-operators-grgl5 | Killing | Stopping container registry-server
12:25:45 | openshift-marketplace | kubelet | certified-operators-wkgpr | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
12:25:45 | openshift-marketplace | multus | community-operators-bk9gr | AddedInterface | Add eth0 [10.129.2.60/23]
12:25:45 | openshift-marketplace | kubelet | community-operators-bk9gr | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
12:25:45 | openshift-marketplace | multus | certified-operators-wkgpr | AddedInterface | Add eth0 [10.129.2.59/23]
12:25:46 | openshift-marketplace | kubelet | community-operators-bk9gr | Started | Started container registry-server
12:25:46 | openshift-marketplace | kubelet | certified-operators-wkgpr | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 776.790061ms
12:25:46 | openshift-marketplace | kubelet | community-operators-bk9gr | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 925.256403ms
12:25:46 | openshift-marketplace | kubelet | community-operators-bk9gr | Created | Created container registry-server
12:25:46 | openshift-marketplace | kubelet | certified-operators-wkgpr | Created | Created container registry-server
12:25:47 | openshift-marketplace | kubelet | certified-operators-wkgpr | Started | Started container registry-server
12:25:55 (x2) | openshift-marketplace | kubelet | community-operators-bk9gr | Killing | Stopping container registry-server
12:25:59 (x2) | openshift-marketplace | kubelet | certified-operators-wkgpr | Killing | Stopping container registry-server
12:32:17 | openshift-marketplace | kubelet | redhat-marketplace-98mxp | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
12:32:17 | openshift-marketplace | multus | redhat-marketplace-98mxp | AddedInterface | Add eth0 [10.129.2.61/23]
12:32:18 | openshift-marketplace | kubelet | redhat-marketplace-98mxp | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 771.843133ms
12:32:18 | openshift-marketplace | kubelet | redhat-marketplace-98mxp | Created | Created container registry-server
12:32:19 | openshift-marketplace | kubelet | redhat-marketplace-98mxp | Started | Started container registry-server
12:32:20 | openshift-marketplace | multus | redhat-operators-4brnp | AddedInterface | Add eth0 [10.129.2.62/23]
12:32:21 | openshift-marketplace | kubelet | redhat-operators-4brnp | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
12:32:22 | openshift-marketplace | kubelet | redhat-operators-4brnp | Created | Created container registry-server
12:32:22 | openshift-marketplace | kubelet | redhat-operators-4brnp | Started | Started container registry-server
12:32:22 | openshift-marketplace | kubelet | redhat-operators-4brnp | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 1.050876245s
12:32:24 (x2) | openshift-marketplace | kubelet | redhat-marketplace-98mxp | Killing | Stopping container registry-server
12:32:36 (x2) | openshift-marketplace | kubelet | redhat-operators-4brnp | Killing | Stopping container registry-server
12:35:46 | openshift-marketplace | kubelet | community-operators-sl84r | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
12:35:46 | openshift-marketplace | multus | community-operators-sl84r | AddedInterface | Add eth0 [10.129.2.63/23]
12:35:47 | openshift-marketplace | kubelet | community-operators-sl84r | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 467.047392ms
12:35:47 | openshift-marketplace | kubelet | community-operators-sl84r | Created | Created container registry-server
12:35:47 | openshift-marketplace | kubelet | community-operators-sl84r | Started | Started container registry-server
12:35:54 (x2) | openshift-marketplace | kubelet | community-operators-sl84r | Killing | Stopping container registry-server
12:36:11 | openshift-marketplace | multus | certified-operators-j54nl | AddedInterface | Add eth0 [10.129.2.64/23]
12:36:11 | openshift-marketplace | kubelet | certified-operators-j54nl | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
12:36:12 | openshift-marketplace | kubelet | certified-operators-j54nl | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 483.476686ms
12:36:12 | openshift-marketplace | kubelet | certified-operators-j54nl | Started | Started container registry-server
12:36:12 | openshift-marketplace | kubelet | certified-operators-j54nl | Created | Created container registry-server
12:36:17 (x2) | openshift-marketplace | kubelet | certified-operators-j54nl | Killing | Stopping container registry-server
12:42:23 | openshift-marketplace | multus | redhat-operators-dft94 | AddedInterface | Add eth0 [10.129.2.65/23]
12:42:24 | openshift-marketplace | kubelet | redhat-operators-dft94 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 859.714362ms
12:42:24 | openshift-marketplace | kubelet | redhat-operators-dft94 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
12:42:25 | openshift-marketplace | kubelet | redhat-operators-dft94 | Started | Started container registry-server
12:42:25 | openshift-marketplace | kubelet | redhat-operators-dft94 | Created | Created container registry-server
12:42:37 (x2) | openshift-marketplace | kubelet | redhat-operators-dft94 | Killing | Stopping container registry-server
12:43:23 | openshift-marketplace | kubelet | redhat-marketplace-wp2vw | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
12:43:23 | openshift-marketplace | multus | redhat-marketplace-wp2vw | AddedInterface | Add eth0 [10.129.2.66/23]
12:43:24 | openshift-marketplace | kubelet | redhat-marketplace-wp2vw | Started | Started container registry-server
12:43:24 | openshift-marketplace | kubelet | redhat-marketplace-wp2vw | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 799.124066ms
12:43:24 | openshift-marketplace | kubelet | redhat-marketplace-wp2vw | Created | Created container registry-server
12:43:33 (x2) | openshift-marketplace | kubelet | redhat-marketplace-wp2vw | Killing | Stopping container registry-server
12:46:27 | openshift-marketplace | multus | community-operators-mpl78 | AddedInterface | Add eth0 [10.129.2.67/23]
12:46:28 | openshift-marketplace | kubelet | community-operators-mpl78 | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
12:46:29 | openshift-marketplace | kubelet | community-operators-mpl78 | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 818.03413ms
12:46:29 | openshift-marketplace | kubelet | community-operators-mpl78 | Started | Started container registry-server
12:46:29 | openshift-marketplace | kubelet | community-operators-mpl78 | Created | Created container registry-server
12:46:38 (x2) | openshift-marketplace | kubelet | community-operators-mpl78 | Killing | Stopping container registry-server
12:53:10 | openshift-marketplace | kubelet | redhat-operators-8t585 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
12:53:10 | openshift-marketplace | kubelet | redhat-operators-8t585 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 455.827186ms
12:53:10 | openshift-marketplace | multus | redhat-operators-8t585 | AddedInterface | Add eth0 [10.129.2.68/23]
12:53:11 | openshift-marketplace | kubelet | redhat-operators-8t585 | Started | Started container registry-server
12:53:11 | openshift-marketplace | kubelet | redhat-operators-8t585 | Created | Created container registry-server
12:53:20 (x2) | openshift-marketplace | kubelet | redhat-operators-8t585 | Killing | Stopping container registry-server
12:53:54 | openshift-marketplace | kubelet | redhat-marketplace-sj6bn | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
12:53:54 | openshift-marketplace | multus | redhat-marketplace-sj6bn | AddedInterface | Add eth0 [10.129.2.69/23]
12:53:55 | openshift-marketplace | kubelet | redhat-marketplace-sj6bn | Created | Created container registry-server
12:53:55 | openshift-marketplace | kubelet | redhat-marketplace-sj6bn | Started | Started container registry-server
12:53:55 | openshift-marketplace | kubelet | redhat-marketplace-sj6bn | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 447.70798ms
12:54:07 (x2) | openshift-marketplace | kubelet | redhat-marketplace-sj6bn | Killing | Stopping container registry-server
12:56:06 | openshift-marketplace | multus | certified-operators-tk85r | AddedInterface | Add eth0 [10.129.2.70/23]
12:56:07 | openshift-marketplace | kubelet | certified-operators-tk85r | Created | Created container registry-server
12:56:07 | openshift-marketplace | kubelet | certified-operators-tk85r | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
12:56:07 | openshift-marketplace | kubelet | certified-operators-tk85r | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 474.275766ms
12:56:07 | openshift-marketplace | kubelet | certified-operators-tk85r | Started | Started container registry-server
12:56:17 (x2) | openshift-marketplace | kubelet | certified-operators-tk85r | Killing | Stopping container registry-server
12:57:08 | openshift-marketplace | kubelet | community-operators-vkbm4 | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
12:57:08 | openshift-marketplace | multus | community-operators-vkbm4 | AddedInterface | Add eth0 [10.129.2.71/23]
12:57:08 | openshift-marketplace | kubelet | community-operators-vkbm4 | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 416.877123ms
12:57:09 | openshift-marketplace | kubelet | community-operators-vkbm4 | Created | Created container registry-server
12:57:09 | openshift-marketplace | kubelet | community-operators-vkbm4 | Started | Started container registry-server
12:57:19 (x2) | openshift-marketplace | kubelet | community-operators-vkbm4 | Killing | Stopping container registry-server
13:03:59 | openshift-marketplace | kubelet | redhat-operators-6hkhj | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
13:03:59 | openshift-marketplace | multus | redhat-operators-6hkhj | AddedInterface | Add eth0 [10.129.2.72/23]
13:04:00 | openshift-marketplace | kubelet | redhat-operators-6hkhj | Created | Created container registry-server
13:04:00 | openshift-marketplace | kubelet | redhat-operators-6hkhj | Started | Started container registry-server
13:04:00 | openshift-marketplace | kubelet | redhat-operators-6hkhj | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 475.044508ms
13:04:08 (x2) | openshift-marketplace | kubelet | redhat-operators-6hkhj | Killing | Stopping container registry-server
13:07:23 | openshift-marketplace | multus | community-operators-trkt6 | AddedInterface | Add eth0 [10.129.2.73/23]
13:07:24 | openshift-marketplace | kubelet | community-operators-trkt6 | Started | Started container registry-server
13:07:24 | openshift-marketplace | kubelet | community-operators-trkt6 | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
13:07:24 | openshift-marketplace | kubelet | community-operators-trkt6 | Created | Created container registry-server
13:07:24 | openshift-marketplace | kubelet | community-operators-trkt6 | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 746.078029ms
13:07:33 (x2) | openshift-marketplace | kubelet | community-operators-trkt6 | Killing | Stopping container registry-server
13:14:12 | openshift-marketplace | kubelet | redhat-marketplace-cs46k | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
13:14:12 | openshift-marketplace | multus | redhat-marketplace-cs46k | AddedInterface | Add eth0 [10.129.2.74/23]
13:14:13 | openshift-marketplace | kubelet | redhat-marketplace-cs46k | Created | Created container registry-server
13:14:13 | openshift-marketplace | kubelet | redhat-marketplace-cs46k | Started | Started container registry-server
13:14:13 | openshift-marketplace | kubelet | redhat-marketplace-cs46k | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 1.011146584s
13:14:20 (x2) | openshift-marketplace | kubelet | redhat-marketplace-cs46k | Killing | Stopping container registry-server
13:14:46 | openshift-marketplace | multus | redhat-operators-ljxld | AddedInterface | Add eth0 [10.129.2.75/23]
13:14:47 | openshift-marketplace | kubelet | redhat-operators-ljxld | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
13:14:47 | openshift-marketplace | multus | certified-operators-nh9n8 | AddedInterface | Add eth0 [10.129.2.76/23]
13:14:47 | openshift-marketplace | kubelet | redhat-operators-ljxld | Created | Created container registry-server
13:14:47 | openshift-marketplace | kubelet | redhat-operators-ljxld | Started | Started container registry-server
13:14:47 | openshift-marketplace | kubelet | certified-operators-nh9n8 | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
13:14:47 | openshift-marketplace | kubelet | redhat-operators-ljxld | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 483.306129ms
13:14:48 | openshift-marketplace | kubelet | certified-operators-nh9n8 | Created | Created container registry-server
13:14:48 | openshift-marketplace | kubelet | certified-operators-nh9n8 | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 479.507195ms
13:14:48 | openshift-marketplace | kubelet | certified-operators-nh9n8 | Started | Started container registry-server
13:15:01 (x2) | openshift-marketplace | kubelet | redhat-operators-ljxld | Killing | Stopping container registry-server
13:15:01 (x2) | openshift-marketplace | kubelet | certified-operators-nh9n8 | Killing | Stopping container registry-server
13:18:41 | openshift-marketplace | multus | community-operators-gwd9c | AddedInterface | Add eth0 [10.129.2.77/23]
13:18:41 | openshift-marketplace | kubelet | community-operators-gwd9c | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
13:18:42 | openshift-marketplace | kubelet | community-operators-gwd9c | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 452.013866ms
13:18:42 | openshift-marketplace | kubelet | community-operators-gwd9c | Created | Created container registry-server
13:18:42 | openshift-marketplace | kubelet | community-operators-gwd9c | Started | Started container registry-server
13:18:51 (x2) | openshift-marketplace | kubelet | community-operators-gwd9c | Killing | Stopping container registry-server
13:24:57 | openshift-marketplace | multus | certified-operators-fdrn2 | AddedInterface | Add eth0 [10.129.2.78/23]
13:24:57 | openshift-marketplace | kubelet | certified-operators-fdrn2 | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
13:24:58 | openshift-marketplace | kubelet | certified-operators-fdrn2 | Started | Started container registry-server
13:24:58 | openshift-marketplace | kubelet | certified-operators-fdrn2 | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 519.617286ms
13:24:58 | openshift-marketplace | kubelet | certified-operators-fdrn2 | Created | Created container registry-server
13:25:06 (x2) | openshift-marketplace | kubelet | certified-operators-fdrn2 | Killing | Stopping container registry-server
13:25:55 | openshift-marketplace | multus | redhat-operators-zc6ts | AddedInterface | Add eth0 [10.129.2.79/23]
13:25:56 | openshift-marketplace | kubelet | redhat-operators-zc6ts | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
13:25:57 | openshift-marketplace | kubelet | redhat-operators-zc6ts | Started | Started container registry-server
13:25:57 | openshift-marketplace | kubelet | redhat-operators-zc6ts | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 856.318837ms
13:25:57 | openshift-marketplace | kubelet | redhat-operators-zc6ts | Created | Created container registry-server
13:26:06 (x2) | openshift-marketplace | kubelet | redhat-operators-zc6ts | Killing | Stopping container registry-server
13:29:00 | openshift-marketplace | multus | community-operators-4l9vk | AddedInterface | Add eth0 [10.129.2.80/23]
13:29:01 | openshift-marketplace | kubelet | community-operators-4l9vk | Pulling | Pulling image "registry.redhat.io/redhat/community-operator-index:v4.6"
13:29:02 | openshift-marketplace | kubelet | community-operators-4l9vk | Started | Started container registry-server
13:29:02 | openshift-marketplace | kubelet | community-operators-4l9vk | Pulled | Successfully pulled image "registry.redhat.io/redhat/community-operator-index:v4.6" in 765.726415ms
13:29:02 | openshift-marketplace | kubelet | community-operators-4l9vk | Created | Created container registry-server
13:29:09 (x2) | openshift-marketplace | kubelet | community-operators-4l9vk | Killing | Stopping container registry-server
13:31:56 | openshift-marketplace | multus | redhat-marketplace-cfbxv | AddedInterface | Add eth0 [10.129.2.81/23]
13:31:56 | openshift-marketplace | kubelet | redhat-marketplace-cfbxv | Pulling | Pulling image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6"
13:31:57 | openshift-marketplace | kubelet | redhat-marketplace-cfbxv | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-marketplace-index:v4.6" in 843.203511ms
13:31:58 | openshift-marketplace | kubelet | redhat-marketplace-cfbxv | Created | Created container registry-server
13:31:58 | openshift-marketplace | kubelet | redhat-marketplace-cfbxv | Started | Started container registry-server
13:32:05 (x2) | openshift-marketplace | kubelet | redhat-marketplace-cfbxv | Killing | Stopping container registry-server
13:36:19 | openshift-marketplace | multus | redhat-operators-7q9g9 | AddedInterface | Add eth0 [10.129.2.83/23]
13:36:19 | openshift-marketplace | multus | certified-operators-g44p8 | AddedInterface | Add eth0 [10.129.2.82/23]
13:36:19 | openshift-marketplace | kubelet | certified-operators-g44p8 | Pulling | Pulling image "registry.redhat.io/redhat/certified-operator-index:v4.6"
13:36:19 | openshift-marketplace | kubelet | redhat-operators-7q9g9 | Pulling | Pulling image "registry.redhat.io/redhat/redhat-operator-index:v4.6"
13:36:19 | openshift-marketplace | kubelet | certified-operators-g44p8 | Pulled | Successfully pulled image "registry.redhat.io/redhat/certified-operator-index:v4.6" in 452.680649ms
13:36:20 | openshift-marketplace | kubelet | redhat-operators-7q9g9 | Created | Created container registry-server
13:36:20 | openshift-marketplace | kubelet | certified-operators-g44p8 | Started | Started container registry-server
13:36:20 | openshift-marketplace | kubelet | redhat-operators-7q9g9 | Pulled | Successfully pulled image "registry.redhat.io/redhat/redhat-operator-index:v4.6" in 480.765951ms
13:36:20 | openshift-marketplace | kubelet | redhat-operators-7q9g9 | Started | Started container registry-server
13:36:20 | openshift-marketplace | kubelet | certified-operators-g44p8 | Created | Created container registry-server
13:36:26 (x2) | openshift-marketplace | kubelet | redhat-operators-7q9g9 | Killing | Stopping container registry-server
13:36:28 (x2) | openshift-marketplace | kubelet | certified-operators-g44p8 | Killing | Stopping container registry-server