Uploaded image for project: 'AMQ Streams'
  1. AMQ Streams
  2. ENTMQST-4164

Persistent error on periodic reconciliation of internal topics

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Major Major
    • 2.7.0.GA
    • 2.1.0.GA
    • topic-operator
    • None
    • False
    • None
    • False

      I think this is an other edge case in which the internal store gets corrupted.

      I'm not able to reproduce it, but it looks like the periodic reconciliation is constantly failing on the following internal topics.
      It keeps trying to create them when they already exist.

      • __consumer_offsets
      • __strimzi_store_topic
      • __strimzi-topic-operator-kstreams-topic-store-changelog

      Error example which is going on for about 1 hour in logs I got:

      2022-07-20 09:35:40,67318 ERROR [vert.x-eventloop-thread-1] TopicOperator:1490 - Reconciliation #24957(periodic -__strimzi_store_topic) KafkaTopic(myproject/strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55): Error reconciling KafkaTopic null
      io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.137.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/myproject/kafkatopics. Message: kafkatopics.kafka.strimzi.io "strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkatopics, name=strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkatopics.kafka.strimzi.io "strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:683) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:662) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:613) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:556) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:519) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:306) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[io.fabric8.kubernetes-client-5.12.0.redhat-00001.jar:?]
      	at io.strimzi.operator.topic.K8sImpl.lambda$createResource$0(K8sImpl.java:51) ~[io.strimzi.topic-operator-0.28.0.redhat-00003.jar:0.28.0.redhat-00003]
      	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.5.redhat-00001.jar:4.2.5.redhat-00001]
      	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.5.redhat-00001.jar:4.2.5.redhat-00001]
      	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.5.redhat-00001.jar:4.2.5.redhat-00001]
      	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.2.5.redhat-00001.jar:4.2.5.redhat-00001]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.74.Final-redhat-00001.jar:4.1.74.Final-redhat-00001]
      	at java.lang.Thread.run(Thread.java:829) [?:?]
      

            Unassigned Unassigned
            rhn-support-fvaleri Federico Valeri
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: