Uploaded image for project: 'RH-SSO'
  1. RH-SSO
  2. RHSSO-2744

Liveness probe failing when connection pool is full

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Optional
    • None
    • RH-SSO-7.6.4
    • None
    • False
    • None
    • False

    Description

      Infra:

      OCP 4.12
      JBoss 7.4.11
      RH-SSO 7.6.4
      liveness probe
      external Azure database.

      Description
      Liveness Probe is failing when the connection pool is becoming full, and no more connection available.

      Liveness probe failure triggers a pod restart, although the pod was still fully healthy.

      Expected behaviour
      This can be very annoying in case of long running transaction, as the long running will get cancelled in the middle.

      There is no need in fact to restart the pod, when there is more more connection available from the connection pool.

      Other Remark (RHBK)

      It seems that this issue has been fixed with RHBK. RHBK is now using the Qarkus executor pool, and no longer Vert.X

      See Keycloak under load
      https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/load-behavior

      ~~~~
      Starting with Keycloak 20.0.0 and running Quarkus, all Keycloak requests are dispatched to the executor pool of Quarkus (see keycloak#15193).

      ...

      ...

      Before that change, regular Keycloak requests were handled on the Vert.x worker pool and could therefore exhaust it, the liveness probe could fail, and Keycloak pods would restart under a high load

      ~~~

       

      Other Investigation (EAPSUP_

      A EAPSUP has been created against the EAP team at first to determine if this issue was due to EAP/JDBC.

      https://issues.redhat.com/browse/EAPSUP-1280

       

      EAPSUP team has indicated that 

      The liveness probe for EAP 7.4.11 does not check connection status to determine its liveness.

       

       

      ===> It means that this issue an with OCP livenesss probe and SSO connection pool becoming full.

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            rhn-engineering-shawkins Steven Hawkins
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: