Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-8401

Operator crashes when backend CR is deleted

XMLWordPrintable

    • False
    • None
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Hide
      1. I've created backend via CRD
      2. connect it to one of products via CRD
      3. delete backend usage from product and backend via UI
      4. operator crashes:
         
      E0427 18:26:08.111870 1 runtime.go:76] Observed a panic: Backend SystemName backend4 not found in backend index
      goroutine 409 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic(0x17a8ea0, 0xc000d32ba0)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:74 +0xa3
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:48 +0x82
      panic(0x17a8ea0, 0xc000d32ba0)
      /opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).syncBackendUsage(0xc000e6eac0, 0x0, 0x0, 0xb6868e, 0xbc8add175c44)
      /remote-source/app/controllers/capabilities/backend_usages.go:93 +0x164c
      github.com/3scale/3scale-operator/pkg/helper.(*taskRunnerImpl).Run(0xc000e6fd00, 0x1a5c53c, 0x15)
      /remote-source/app/pkg/helper/task_runner.go:41 +0xda
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).Reconcile(0xc000e6eac0, 0xc001ba6000, 0xc000274c90, 0xc0018ee840)
      /remote-source/app/controllers/capabilities/product_threescale_reconciler.go:56 +0x411
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).reconcile(0xc00000ee38, 0xc001ba6000, 0xc000b07e00, 0x1cfdc00, 0xc000866360)
      /remote-source/app/controllers/capabilities/product_controller.go:227 +0x3ea
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).Reconcile(0xc00000ee38, 0xc0009bbad0, 0xc, 0xc0009bbab8, 0x8, 0xc000b07cc0, 0xc000842098, 0x1cbee60, 0xc000842090)
      /remote-source/app/controllers/capabilities/product_controller.go:149 +0x9ed
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000833dd0, 0x18930e0, 0xc0008c8660, 0xc000696d00)
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244 +0x27d
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000833dd0, 0x1b11000)
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218 +0xcb
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000833dd0)
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197 +0x2b
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000cbd2a0)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5e
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000cbd2a0, 0x1cabc20, 0xc000b84d50, 0x1, 0xc00065c060)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xa3
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000cbd2a0, 0x3b9aca00, 0x0, 0x1, 0xc00065c060)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0xaa
      k8s.io/apimachinery/pkg/util/wait.Until(0xc000cbd2a0, 0x3b9aca00, 0xc00065c060)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:179 +0x44d
      panic: Backend SystemName backend4 not found in backend index [recovered]
      panic: Backend SystemName backend4 not found in backend index
      
      goroutine 409 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:55 +0x105
      panic(0x17a8ea0, 0xc000d32ba0)
      /opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).syncBackendUsage(0xc000e6eac0, 0x0, 0x0, 0xb6868e, 0xbc8add175c44)
      /remote-source/app/controllers/capabilities/backend_usages.go:93 +0x164c
      github.com/3scale/3scale-operator/pkg/helper.(*taskRunnerImpl).Run(0xc000e6fd00, 0x1a5c53c, 0x15)
      /remote-source/app/pkg/helper/task_runner.go:41 +0xda
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).Reconcile(0xc000e6eac0, 0xc001ba6000, 0xc000274c90, 0xc0018ee840)
      /remote-source/app/controllers/capabilities/product_threescale_reconciler.go:56 +0x411
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).reconcile(0xc00000ee38, 0xc001ba6000, 0xc000b07e00, 0x1cfdc00, 0xc000866360)
      /remote-source/app/controllers/capabilities/product_controller.go:227 +0x3ea
      github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).Reconcile(0xc00000ee38, 0xc0009bbad0, 0xc, 0xc0009bbab8, 0x8, 0xc000b07cc0, 0xc000842098, 0x1cbee60, 0xc000842090)
      /remote-source/app/controllers/capabilities/product_controller.go:149 +0x9ed
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000833dd0, 0x18930e0, 0xc0008c8660, 0xc000696d00)
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244 +0x27d
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000833dd0, 0x1b11000)
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218 +0xcb
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000833dd0)
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197 +0x2b
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000cbd2a0)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5e
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000cbd2a0, 0x1cabc20, 0xc000b84d50, 0x1, 0xc00065c060)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xa3
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000cbd2a0, 0x3b9aca00, 0x0, 0x1, 0xc00065c060)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0xaa
      k8s.io/apimachinery/pkg/util/wait.Until(0xc000cbd2a0, 0x3b9aca00, 0xc00065c060)
      /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
      /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:179 +0x44d
      
      Show
      I've created backend via CRD connect it to one of products via CRD delete backend usage from product and backend via UI operator crashes:   E0427 18:26:08.111870 1 runtime.go:76] Observed a panic: Backend SystemName backend4 not found in backend index goroutine 409 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x17a8ea0, 0xc000d32ba0) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:74 +0xa3 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:48 +0x82 panic(0x17a8ea0, 0xc000d32ba0) /opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2 github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).syncBackendUsage(0xc000e6eac0, 0x0, 0x0, 0xb6868e, 0xbc8add175c44) /remote-source/app/controllers/capabilities/backend_usages.go:93 +0x164c github.com/3scale/3scale-operator/pkg/helper.(*taskRunnerImpl).Run(0xc000e6fd00, 0x1a5c53c, 0x15) /remote-source/app/pkg/helper/task_runner.go:41 +0xda github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).Reconcile(0xc000e6eac0, 0xc001ba6000, 0xc000274c90, 0xc0018ee840) /remote-source/app/controllers/capabilities/product_threescale_reconciler.go:56 +0x411 github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).reconcile(0xc00000ee38, 0xc001ba6000, 0xc000b07e00, 0x1cfdc00, 0xc000866360) /remote-source/app/controllers/capabilities/product_controller.go:227 +0x3ea github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).Reconcile(0xc00000ee38, 0xc0009bbad0, 0xc, 0xc0009bbab8, 0x8, 0xc000b07cc0, 0xc000842098, 0x1cbee60, 0xc000842090) /remote-source/app/controllers/capabilities/product_controller.go:149 +0x9ed sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000833dd0, 0x18930e0, 0xc0008c8660, 0xc000696d00) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244 +0x27d sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000833dd0, 0x1b11000) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218 +0xcb sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000833dd0) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197 +0x2b k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000cbd2a0) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5e k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000cbd2a0, 0x1cabc20, 0xc000b84d50, 0x1, 0xc00065c060) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xa3 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000cbd2a0, 0x3b9aca00, 0x0, 0x1, 0xc00065c060) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0xaa k8s.io/apimachinery/pkg/util/wait.Until(0xc000cbd2a0, 0x3b9aca00, 0xc00065c060) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:179 +0x44d panic: Backend SystemName backend4 not found in backend index [recovered] panic: Backend SystemName backend4 not found in backend index goroutine 409 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:55 +0x105 panic(0x17a8ea0, 0xc000d32ba0) /opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2 github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).syncBackendUsage(0xc000e6eac0, 0x0, 0x0, 0xb6868e, 0xbc8add175c44) /remote-source/app/controllers/capabilities/backend_usages.go:93 +0x164c github.com/3scale/3scale-operator/pkg/helper.(*taskRunnerImpl).Run(0xc000e6fd00, 0x1a5c53c, 0x15) /remote-source/app/pkg/helper/task_runner.go:41 +0xda github.com/3scale/3scale-operator/controllers/capabilities.(*ProductThreescaleReconciler).Reconcile(0xc000e6eac0, 0xc001ba6000, 0xc000274c90, 0xc0018ee840) /remote-source/app/controllers/capabilities/product_threescale_reconciler.go:56 +0x411 github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).reconcile(0xc00000ee38, 0xc001ba6000, 0xc000b07e00, 0x1cfdc00, 0xc000866360) /remote-source/app/controllers/capabilities/product_controller.go:227 +0x3ea github.com/3scale/3scale-operator/controllers/capabilities.(*ProductReconciler).Reconcile(0xc00000ee38, 0xc0009bbad0, 0xc, 0xc0009bbab8, 0x8, 0xc000b07cc0, 0xc000842098, 0x1cbee60, 0xc000842090) /remote-source/app/controllers/capabilities/product_controller.go:149 +0x9ed sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000833dd0, 0x18930e0, 0xc0008c8660, 0xc000696d00) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244 +0x27d sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000833dd0, 0x1b11000) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218 +0xcb sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000833dd0) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197 +0x2b k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000cbd2a0) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5e k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000cbd2a0, 0x1cabc20, 0xc000b84d50, 0x1, 0xc00065c060) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xa3 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000cbd2a0, 0x3b9aca00, 0x0, 0x1, 0xc00065c060) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0xaa k8s.io/apimachinery/pkg/util/wait.Until(0xc000cbd2a0, 0x3b9aca00, 0xc00065c060) /remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:179 +0x44d

      I know that it is recommended to not modify or delete objects created by CRs from UI, API or Toolbox but user can still do this. So I think that Operator should be able to deal with this situation and do not crash because it manages other more important objects like APIManager object.

       

            Unassigned Unassigned
            mkudlej@redhat.com Martin Kudlej
            Martin Kudlej Martin Kudlej
            Eguzki Astiz Lezaun Eguzki Astiz Lezaun
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: