Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-2073

[A-MQ, kahadb] NullPointerException raised at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.findNextPage()

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JBoss A-MQ 6.3.x
    • None
    • kahadb
    • None

    Description

      After the broker runs out of disk space (intended in the disaster recovery testing) intermittently the next start results in many "message not found in sequence id index:" and failed to start with a NullPointerException.

      initial "no space on device" :

      2017-03-08 11:02:21,058 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1896 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint done.
2017-03-08 11:02:26,106 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1663 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint started.
2017-03-08 11:02:26,148 | DEBUG | heckpoint Worker | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1896 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | Checkpoint done.
2017-03-08 11:02:30,251 | ERROR | .0.1:45603@61616 | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1117 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | KahaDB failed to store to Journal
java.io.FileNotFoundException: /XXXXX/jboss-a-mq-6.3.0.redhat-250/data/amqdb/kahadb/db-32.log (No space left on device)
	at java.io.RandomAccessFile.open0(Native Method)[:1.8.0_91]
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)[:1.8.0_91]
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)[:1.8.0_91]
	at org.apache.activemq.util.RecoverableRa2017-03-08 11:03:16,121 | INFO  | FelixStartLevel  | fileinstall                      | ?                                   ? | 9 - org.apache.felix.fileinstall - 3.5.8 | Updating configuration from io.fabric8.mq.fabric.server-broker.cfg

      Subsequent restart has many

      2017-03-08 11:03:30,822 | DEBUG | AMQ-1-thread-1   | MessageDatabase                  | emq.store.kahadb.MessageDatabase 1516 | 163 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630250 | message not found in sequence id index: ID:XXXXX:16:1704

      and fails to start with the following NullPointer

      2017-03-08 11:03:34,903 | INFO  | AMQ-1-thread-1   | ActiveMQServiceFactory           | Factory$ClusteredConfiguration$1  502 | 177 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-630250 | Broker amq failed to start.  Will try again in 10 seconds                                                                                                                            
2017-03-08 11:03:34,903 | ERROR | AMQ-1-thread-1   | ActiveMQServiceFactory           | Factory$ClusteredConfiguration$1  503 | 177 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-630250 | Exception on start: java.lang.NullPointerException
java.lang.NullPointerException
	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:118)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.disk.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:129)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$2.execute(KahaDBStore.java:528)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$2.execute(KahaDBStore.java:521)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:802)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.getMessageCount(KahaDBStore.java:521)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:100)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.Queue.initialize(Queue.java:386)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:87)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:541)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:135)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:348)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:199)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:172)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:118)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:177)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:208)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:120)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:119)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:187)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService$6.start(BrokerService.java:2325)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:719)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:705)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:611)[163:org.apache.activemq.activemq-osgi:5.11.0.redhat-630250]
	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.doStart(ActiveMQServiceFactory.java:549)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.access$400(ActiveMQServiceFactory.java:359)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
	at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$1.run(ActiveMQServiceFactory.java:490)[177:io.fabric8.mq.mq-fabric:1.2.0.redhat-630250]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

      Attachments

        Issue Links

          is related to

          Bug - A problem that impairs or prevents the functions of the product. ENTMQ-2112 [A-MQ 6, kahadb] kahaDBIOExceptionHandler cannot be used when using lease locker

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. ENTMQ-2065 [A-MQ 6, kahadb, recovery] partial last write in journal is blocking broker restart

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              gtully@redhat.com Gary Tully
              rhn-support-pfox Patrick Fox (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: