Uploaded image for project: 'Maistra'
  1. Maistra
  2. MAISTRA-52

The istio-installer occasionally starts a second job which keeps failing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Do
    • Major
    • None
    • None
    • None
    • None

    Description

      // Some comments here
      public String getFoo()
      {
          return foo;
      }
      

      Following the steps for Service Mesh installation (https://docs.openshift.com/container-platform/3.10/servicemesh-install/servicemesh-install.html) it might happen that the istio-installer job spins up a second instance which keeps failing. It took quite a while to start all the pods in the istio-system namespace in this case. So I'm wondering if this is some kind of timeout error. Anyway, it seems all the pods and services are successfully running, just the second instance of the job keeps failing.

      The openshift-ansible-istio-installer-job-x46k8 terminates with exit code 2. The logs are attached. Among other things the log says "configmap \"install.istio.inventory\" deleted"

      Later, the openshift-ansible-istio-installer-job-vws72 start and keeps hanging in state "Container Creating". The events show this:

      Unable to mount volumes for pod "openshift-ansible-istio-installer-job-vws72_istio-system(9a09fc9a-c58b-11e8-9117-fa163eb24e1a)": timeout expired waiting for volumes to attach or mount for pod "istio-system"/"openshift-ansible-istio-installer-job-vws72". list of unmounted volumes=[configmap]. list of unattached volumes=[configdir configmap openshift-ansible-token-88dtp]
      MountVolume.SetUp failed for volume "configmap" : configmaps "install.istio.inventory" not found
      

      Other parts of the system:

      [root@mmaster master]# kubectl get all -n istio-system
      NAME                                              READY     STATUS              RESTARTS   AGE
      pod/istio-citadel-6f9c778bb6-g8wgk                1/1       Running             0          17h
      pod/istio-egressgateway-97d67464f-lb9qb           1/1       Running             0          17h
      pod/istio-galley-c47f5dffc-jzcfn                  1/1       Running             2          17h
      pod/istio-ingressgateway-5cd78cf4fd-pkk5q         1/1       Running             0          17h
      pod/istio-pilot-5774d68898-l8dp9                  2/2       Running             0          17h
      pod/istio-policy-5d78b8fb9b-g8fjj                 2/2       Running             0          17h
      pod/istio-sidecar-injector-57466d9bb-lkpzt        1/1       Running             1          17h
      pod/istio-statsd-prom-bridge-7f44bb5ddb-jcjlq     1/1       Running             0          17h
      pod/istio-telemetry-7cd5f59d57-cdpn7              2/2       Running             0          17h
      pod/knative-ingressgateway-6fc979cc4d-nzvkt       1/1       Running             0          16h
      pod/openshift-ansible-istio-installer-job-vws72   0/1       ContainerCreating   0          17h
      pod/openshift-ansible-istio-installer-job-x46k8   0/1       Error               0          17h
      pod/prometheus-84bd4b9796-5b7xs                   1/1       Running             0          17h
      
      NAME                               TYPE           CLUSTER-IP       EXTERNAL-IP                     PORT(S)                                                                                                                   AGE
      service/istio-citadel              ClusterIP      172.22.136.97    <none>                          8060/TCP,9093/TCP                                                                                                         17h
      service/istio-egressgateway        ClusterIP      172.22.157.17    <none>                          80/TCP,443/TCP                                                                                                            17h
      service/istio-galley               ClusterIP      172.22.47.6      <none>                          443/TCP,9093/TCP                                                                                                          17h
      service/istio-ingressgateway       LoadBalancer   172.22.55.23     172.29.251.108,172.29.251.108   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31065/TCP,8060:32491/TCP,15030:30287/TCP,15031:31753/TCP                 17h
      service/istio-pilot                ClusterIP      172.22.170.208   <none>                          15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                                     17h
      service/istio-policy               ClusterIP      172.22.206.133   <none>                          9091/TCP,15004/TCP,9093/TCP                                                                                               17h
      service/istio-sidecar-injector     ClusterIP      172.22.134.45    <none>                          443/TCP                                                                                                                   17h
      service/istio-statsd-prom-bridge   ClusterIP      172.22.189.112   <none>                          9102/TCP,9125/UDP                                                                                                         17h
      service/istio-telemetry            ClusterIP      172.22.93.231    <none>                          9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                                     17h
      service/knative-ingressgateway     LoadBalancer   172.22.245.251   172.29.25.194,172.29.25.194     80:32380/TCP,443:32390/TCP,31400:32400/TCP,15011:30896/TCP,8060:30426/TCP,853:32620/TCP,15030:30380/TCP,15031:31090/TCP   16h
      service/prometheus                 ClusterIP      172.22.230.27    <none>                          9090/TCP                                                                                                                  17h
      
      NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      deployment.apps/istio-citadel              1         1         1            1           17h
      deployment.apps/istio-egressgateway        1         1         1            1           17h
      deployment.apps/istio-galley               1         1         1            1           17h
      deployment.apps/istio-ingressgateway       1         1         1            1           17h
      deployment.apps/istio-pilot                1         1         1            1           17h
      deployment.apps/istio-policy               1         1         1            1           17h
      deployment.apps/istio-sidecar-injector     1         1         1            1           17h
      deployment.apps/istio-statsd-prom-bridge   1         1         1            1           17h
      deployment.apps/istio-telemetry            1         1         1            1           17h
      deployment.apps/knative-ingressgateway     1         1         1            1           16h
      deployment.apps/prometheus                 1         1         1            1           17h
      
      NAME                                                  DESIRED   CURRENT   READY     AGE
      replicaset.apps/istio-citadel-6f9c778bb6              1         1         1         17h
      replicaset.apps/istio-egressgateway-97d67464f         1         1         1         17h
      replicaset.apps/istio-galley-c47f5dffc                1         1         1         17h
      replicaset.apps/istio-ingressgateway-5cd78cf4fd       1         1         1         17h
      replicaset.apps/istio-pilot-5774d68898                1         1         1         17h
      replicaset.apps/istio-policy-5d78b8fb9b               1         1         1         17h
      replicaset.apps/istio-sidecar-injector-57466d9bb      1         1         1         17h
      replicaset.apps/istio-statsd-prom-bridge-7f44bb5ddb   1         1         1         17h
      replicaset.apps/istio-telemetry-7cd5f59d57            1         1         1         17h
      replicaset.apps/knative-ingressgateway-6fc979cc4d     1         1         1         16h
      replicaset.apps/prometheus-84bd4b9796                 1         1         1         17h
      
      NAME                                                         REFERENCE                           TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
      horizontalpodautoscaler.autoscaling/knative-ingressgateway   Deployment/knative-ingressgateway   <unknown>/60%   1         10        1          16h
      
      NAME                                              DESIRED   SUCCESSFUL   AGE
      job.batch/openshift-ansible-istio-installer-job   1         0            17h
      
      NAME                             HOST/PORT                                                                   PATH      SERVICES                 PORT      TERMINATION   WILDCARD
      route.route.openshift.io/myapp   myapp-istio-system.ose.ocfqeopenshift.osoos.centralci.eng.rdu2.redhat.com             knative-ingressgateway   http2                   None
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mgencur Martin Gencur
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: