Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-110

Pods are continuously crash looping

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • 2.0 ER4
    • Backend, Gateway, Infra, Memcached, System
    • None

    Description

      Issue Description:
      When running OpenShift in a resource constrained virtual machine, the 3scale pods end up in a crash loop until the containers fail to start. This mainly seems to impact the apicast-production and system-app deployments.

      Reason:
      The crash loop occurs due to the health checks in place for the apicast-production and system-app deployments. More specifically, the liveness and readiness probes [1] do not allocate sufficient time for the containers to fully start - and therefore mistakenly assume that they failed to start properly. This results in a redeployment of the pod, and a loop that eventually triggers a full 3scale deployment failure.

      Solution:
      Assuming that virtual machine resource constraints cannot be remedied, one can increase the value of the initialDelaySeconds for each deployment config that is affected to allow for additional time for the containers to start properly; the change would need need to be applied to both the readiness and liveness probes for the container.

      Additional details regarding health checks and probes can be found at the link below:
      [1] https://docs.openshift.com/enterprise/3.1/dev_guide/application_health.html

      Attachments

        Activity

          People

            Unassigned Unassigned
            vramosp Vanessa Ramos (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: