Uploaded image for project: 'Hawkular Metrics'
  1. Hawkular Metrics
  2. HWKMETRICS-542

BusyPoolException under heavy load - no available connection and the queue has reached its max size 256

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None

      Description

      Performance test is POSTing data to h-metrics and it's failing with following exception:
      2016-11-11 05:56:05,469 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-30) HAWKMETRICS200010: Failed to process request: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/127.0.0.1] Pool is busy (no available connection and the queue has reached its max size 256)))

      This issue was visible after commit https://github.com/hawkular/hawkular-metrics/commit/ac55bcfaa862607e438604af760f49566faf8080

      This issue is visible only for bigger msgs. When sending just 1 or 10 datapoints per request it's not happening. It's happening for 100,500 datapoints per request.

      Attempt to use older c* driver version didn't help - https://github.com/hawkular/hawkular-metrics/pull/692 the issue is still visible.

      Complete server.log system.log are attached.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  john.sanda John Sanda
                  Reporter:
                  fbrychta Filip Brychta
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: