Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-1915

When StatefulSet is scaled to zero all PV are lost

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • container image
    • None
    • Hide

      1. Deploy StatefulSet together with drain controller
      2. Produce persistent N messages
      3. Scale down to 0
      4. List PVC oc get pvc
      5. Scale up to 1
      6. No messages to be consumed

      Show
      1. Deploy StatefulSet together with drain controller 2. Produce persistent N messages 3. Scale down to 0 4. List PVC oc get pvc 5. Scale up to 1 6. No messages to be consumed
    • AMQ Broker 1833

      Drain controller seems to be ignoring any error code returned by drain.sh script. Therefor it's possible to scale down from 2 to 0 replica and delete all PersistenVolumes with messages. Even though the script is complaining about missing endpoint to resolve IP.

      Expected: No persistent message is lost upon scale to 0.

      Starting the drainer
      { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "endpoints \"broker-amq-tcp\" not found", "reason": "NotFound", "details": { "name": "broker-amq-tcp", "kind": "endpoints" }, "code": 404 }
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      KeyError: 'subsets'
      Can't find ip to scale down to tried 0 ips
      
      
      W0813 14:18:48.263601       1 client_config.go:553] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
      I0813 14:18:48.327999       1 controller.go:65] Configured to only operate on StatefulSets in namespace dsimansk
      I0813 14:18:48.328474       1 controller.go:125] Setting up event handlers
      I0813 14:18:48.328537       1 controller.go:166] Starting StatefulSet scaledown cleanup controller
      I0813 14:18:48.328557       1 controller.go:169] Waiting for informer caches to sync
      I0813 14:18:48.629560       1 controller.go:174] Starting workers
      I0813 14:18:48.629617       1 controller.go:179] Started workers
      I0813 14:20:00.732706       1 controller.go:341] Found orphaned PVC(s) for ordinal '1'. Creating drain pod 'broker-1'.
      I0813 14:20:13.464590       1 controller.go:410] Drain pod 'broker-1' finished.
      I0813 14:20:13.464631       1 controller.go:417] Deleting PVC broker-vol-broker-1
      I0813 14:20:13.475379       1 controller.go:430] Deleting drain pod broker-1
      I0813 14:20:22.003402       1 controller.go:341] Found orphaned PVC(s) for ordinal '0'. Creating drain pod 'broker-0'.
      I0813 14:20:32.978874       1 controller.go:410] Drain pod 'broker-0' finished.
      I0813 14:20:32.978927       1 controller.go:417] Deleting PVC broker-vol-broker-0
      I0813 14:20:32.988569       1 controller.go:430] Deleting drain pod broker-0
      

            rhn-support-rkieley Roderick Kieley
            rhn-support-rkieley Roderick Kieley
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: