Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-812

java.lang.OutOfMemoryError: GC overhead limit exceeded when there is a problem with NFSv4 KahaDB back-end storage

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • None
    • broker, kahadb, message-store
    • None

    Description

      I have three A-MQ brokers inside Jboss Fuse 6.1 fabric, one master and two slaves. Back-end storage for kahadb is NFSv4. Now we have issues on the storage side and the nodes are loosing the connection. In dmesg:

      NFS: server my.nfs.server.com error: fileid changed
      fsid 0:13: expected fileid 0x32, got 0xc000c

      At some point (usually 1-2 days) the master broker is giving:
      2014-09-25 13:44:52,455 | WARN | 925772421-209484 | nio | tty.io.nio.SelectChannelEndPoint 697 | 92 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.14.v20131031 | handle failed
      java.lang.OutOfMemoryError: GC overhead limit exceeded

      And the slave cannot pick-up because the master don't release the NFS lock. Then I have to manually stop the master container.
      Note that this server has very low load, 50 messages per day or something. And nagios checks to jolokia every few minutes.

      Attachments

        Activity

          People

            Unassigned Unassigned
            pntdevops-vvasilev Vladimir Vasilev
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: