Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29373

http/2 connection coalescing component routing broken with single certificate

XMLWordPrintable

    • Critical
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: Using the same certificate for multiple routes signals to Web browsers that they can do HTTP/2 connection coalescing. This means that a browser could open a connection for one route and then re-use this connection to connect to a second route, which could cause the client's request to go to the wrong backend server. OpenShift router tries to prevent this client behavior by advertising HTTP/2 in TLS ALPN only for routes that have unique certificates. OpenShift router assumed that if a route specified a custom certificate, then it was unique. However, it is actually possible to specify the same custom certificate on multiple routes or specify the default certificate as a custom certificate on a route.

      Consequence: If multiple routes specified the same certificate (or if a route specified the default certificate as a custom certificate), and HTTP/2 was enabled on the router, then HTTP/2 clients could perform connection coalescing, re-using connections and potentially connecting to the wrong backend server.

      Fix: A check was added to detect when the same certificate is specified on more than one route or when a route specifies the default certificate as a custom certificate. When either one of these conditions is detected, OpenShift router configures HAProxy not to allow HTTP/2 client connections to the routes that use the certificate.

      Result: Clients are less likely to connect to the wrong backend server due to HTTP/2 connection coalescing.
      Show
      Cause: Using the same certificate for multiple routes signals to Web browsers that they can do HTTP/2 connection coalescing. This means that a browser could open a connection for one route and then re-use this connection to connect to a second route, which could cause the client's request to go to the wrong backend server. OpenShift router tries to prevent this client behavior by advertising HTTP/2 in TLS ALPN only for routes that have unique certificates. OpenShift router assumed that if a route specified a custom certificate, then it was unique. However, it is actually possible to specify the same custom certificate on multiple routes or specify the default certificate as a custom certificate on a route. Consequence: If multiple routes specified the same certificate (or if a route specified the default certificate as a custom certificate), and HTTP/2 was enabled on the router, then HTTP/2 clients could perform connection coalescing, re-using connections and potentially connecting to the wrong backend server. Fix: A check was added to detect when the same certificate is specified on more than one route or when a route specifies the default certificate as a custom certificate. When either one of these conditions is detected, OpenShift router configures HAProxy not to allow HTTP/2 client connections to the routes that use the certificate. Result: Clients are less likely to connect to the wrong backend server due to HTTP/2 connection coalescing.

      Description of problem:

      Due to HTTP/2 Connection Coalescing (https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/), routes which use the same certificate can present unexplained 503 errors when attempting to access an HTTP/2 enabled ingress controller.

      It appears that HAProxy supports the ability to force HTTP 1.1 on a route-by-route basis, but our Ingress Controller does not expose that option.

      This is especially problematic for component routes because generally speaking, customers use a wildcard or SAN to deploy custom component routes (console, OAuth, downloads), but with HTTP/2, this does not work properly.

      To address this issue, we're proposing the creation of an annotation haproxy.router.openshift.io/http2-disable, which will allow the disabling of HTTP/2 on a route-by-route basis, or smarter logic built into our Ingress operator to handle this situation.  

      Version-Release number of selected component (if applicable):

       OpenShift 4.14

      How reproducible:

      Serve routes to applications in Openshift.
Observe the routes through a HTTP/2 enabled client.
Notice that http/2 client connections are broken (returns 503 on second connection when using same certificates across a mix of re-encrypt and passthrough routes)

      Steps to Reproduce:

      (see above notes)

      Actual results:

      503 error

      Expected results:

      no error

      Additional info:

            mmasters1@redhat.com Miciah Masters
            rh-ee-adejong Aaren de Jong
            Shudi Li Shudi Li
            Votes:
            4 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: