Uploaded image for project: 'Hawkular Metrics'
  1. Hawkular Metrics
  2. HWKMETRICS-564

Compression job is stuck after 5 days

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 0.21.5
    • Fix Version/s: 0.23.1, 0.24.0
    • Component/s: None
    • Labels:
      None
    • Environment:
    • Steps to Reproduce:
      Hide

      Not clear, following steps were used:
      1 - start h-services with 1 external cassandra node
      2 - start generating constant load 5000 metrics/sec
      3 - stop the load after 5 days
      4 - keep the instance up without load and check if the compression job runs

      Show
      Not clear, following steps were used: 1 - start h-services with 1 external cassandra node 2 - start generating constant load 5000 metrics/sec 3 - stop the load after 5 days 4 - keep the instance up without load and check if the compression job runs

      Description

      Compression job was running in regular intervals for 5 days but then it got stuck.
      Last INFO msg from server.log was:
      2016-12-30 10:00:36,869 INFO [org.hawkular.metrics.core.jobs.CompressData] (RxComputationScheduler-1) Starting execution
      but there is no
      "Finished compressing data"
      msg even after 4 days.

      The job scheduler is stuck in sort of a loop trying to execute the following over and over:
      <jsanda> 2017-01-03 13:21:36,838 DEBUG [org.hawkular.metrics.scheduler.impl.SchedulerImpl] (RxComputationScheduler-2) Starting execution for JobDetails{jobId=f6c065d2-1eb4-4f6d-a9a9-6f9022d0fbaf, jobType=COMPRESS_DATA, jobName=COMPRESS_DATA, parameters={}, trigger=RepeatingTrigger

      {triggerTime=1483002000000, interval=7200000, delay=60000}

      , status=FINISHED} in time slice [Thu Dec 29 04:05:00 EST 2016]

      All logs are attached, debug level was enabled on Jan 3th.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  john.sanda John Sanda
                  Reporter:
                  fbrychta Filip Brychta
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: