Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: JBoss A-MQ 6.2
    • Fix Version/s: None
    • Component/s: broker, kahadb
    • Labels:
      None
    • Environment:

      RHEL 6, ext4 filesystem

    • Steps to Reproduce:
      Hide

      1. Unzip an original A-MQ 6.2.0 and activate (i.e. uncomment) the admin/admin account in etc/users.properties
      2. Install Patch P2 (v -153) using the patch:add and patch:install commands
      3. Remove the memoryLimit on the queue policy in activemq.xml (to facilitate enqueuing of messages)
      4. Start the broker: bin/start
      5. Take note of your "Store limit" which might depend on the available space in your journal data file system (e.g. look at the tree root "amq" in hawtio GUI, attribute "Store limit")
      6. Fill-in the journal data file system with garbage files, until available space is less than your "Store limit"
      7. Inject "enough" persistent messages i.e. more than what remains available in your journal data file system. E.g. To inject approx. 300MB of messages, use: bin/client "activemq:producer --user admin --password admin --messageSize 100000 --messageCount 3000 --transactionBatchSize 30 --destination testQ"
      -> The injector fails with an exception stating "No space left on device"
      9. Stop the broker: bin/stop
      10. Delete the garbage file(s) you have set at step (6.)
      11. Start the broker again: bin/start
      -> The broker logs show: "Corrupt journal records found...", "Broker amq failed to start" and "Failed to recover data at position:XX:YYY"

      Show
      1. Unzip an original A-MQ 6.2.0 and activate (i.e. uncomment) the admin/admin account in etc/users.properties 2. Install Patch P2 (v -153) using the patch:add and patch:install commands 3. Remove the memoryLimit on the queue policy in activemq.xml (to facilitate enqueuing of messages) 4. Start the broker: bin/start 5. Take note of your "Store limit" which might depend on the available space in your journal data file system (e.g. look at the tree root "amq" in hawtio GUI, attribute "Store limit") 6. Fill-in the journal data file system with garbage files, until available space is less than your "Store limit" 7. Inject "enough" persistent messages i.e. more than what remains available in your journal data file system. E.g. To inject approx. 300MB of messages, use: bin/client "activemq:producer --user admin --password admin --messageSize 100000 --messageCount 3000 --transactionBatchSize 30 --destination testQ" -> The injector fails with an exception stating "No space left on device" 9. Stop the broker: bin/stop 10. Delete the garbage file(s) you have set at step (6.) 11. Start the broker again: bin/start -> The broker logs show: "Corrupt journal records found...", "Broker amq failed to start" and "Failed to recover data at position:XX:YYY"

      Description

      When hitting filesystem full on data directory, if amq is shut down, some space freed up and amq started again, the broker data is corrupted (amq.log shows "Corrupt journal records"), so
      *not only*the indexThe broker would typically not even start (which I guess was my example) if those data are live data.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  gbonocor Giuseppe Bonocore
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: