Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-608

backend-listener fails to connect backend-redis

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 2.1 GA
    • Backend
    • -
    • Hide

      A useful workaround that can help in "some" situations is to set the APICAST_BACKEND_CACHE_HANDLER env var to resilient so at least perviously authorised keys will not fail. This will not help if the key was not already in the gateway cache though.

      Show
      A useful workaround that can help in "some" situations is to set the APICAST_BACKEND_CACHE_HANDLER env var to resilient so at least perviously authorised keys will not fail. This will not help if the key was not already in the gateway cache though.
    • Hide

      Authorise a few requests for one application through APIcast then kill the backend-redis container:

      docker ps
docker kill backend-redis-xyz

      Authorise the same application through APIcast, the first request will succeed as the key exists in the cache but the backend-listener logs will show the following error:

      Exception `Redis::ConnectionError' at /opt/ruby/3scale_backend-2.77.1.1/lib/3scale/backend/logger/middleware.rb:35 - Connection lost (ECONNRESET)
2018-01-03 17:11:41 +0000: Rack app error: #<Redis::ConnectionError: Connection lost (ECONNRESET)>

      The second request with the same application through APIcast will return a 403 to the client as the cache will have been deleted based on the previous response and the same error as above can be seen in the logs.

      Show
      Authorise a few requests for one application through APIcast then kill the backend-redis container: docker ps docker kill backend-redis-xyz Authorise the same application through APIcast, the first request will succeed as the key exists in the cache but the backend-listener logs will show the following error: Exception `Redis::ConnectionError' at /opt/ruby/3scale_backend-2.77.1.1/lib/3scale/backend/logger/middleware.rb:35 - Connection lost (ECONNRESET) 2018-01-03 17:11:41 +0000: Rack app error: #<Redis::ConnectionError: Connection lost (ECONNRESET)> The second request with the same application through APIcast will return a 403 to the client as the cache will have been deleted based on the previous response and the same error as above can be seen in the logs.

    Description

      When the backend-redis container is killed or scaled down it is then respawned based on the number of replicas in the dc. The backend-listener component should be able to reconnect to the new Redis container to authorise and report new traffic but it always fails on the first 2 attempts to connect.

      Most likely this is happening because the previous connections are cached in the listener and it first fails connecting to the data db and then fails connecting to the resque db.

      Unfortunately this results in a failed request for previously authorised clients (only one client request will fail) or if the first 2 requests processed by the listener are from clients that are not in the APIcast cache they will both fail.

      There is already an issue in GH/backend tracking the reason why we are unable to address this currently. we cannot simply just set a retry due to the fact there is a risk in double reporting traffic involved with this.

      Attachments

        Issue Links

          relates to

          Feature Request - Feature requests from customers and/or users THREESCALE-2168 Redis connection retry

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              Unassigned Unassigned
              rhn-support-keprice Kevin Price
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: