Loading...

Details

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: JBoss A-MQ 6.3
Affects Version/s: JBoss A-MQ 6.2.1
Component/s: broker, fabric8
Labels:
None

Steps to Reproduce:
Hide

setup three networked broker within fabric

fabric:create --wait-for-provisioning container-create-child root broker 3 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker1 broker1 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker2 broker2 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker3 broker3

run attached testcase

mvn clean test -Dclient.threads=125 -Dmessage.count=1000000 -Ddestination.name="mesh.3.failover.1m" -Dbroker.url="failover:(tcp://host1:42270,tcp://host1:37617,tcp://host1:46291)?randomize=true"

I am able to reproduce the issue only with large number of threads/connections (I can reliably reproduce with 125 consumer and 125 producer threads and 1000000 messages).
Show
setup three networked broker within fabric fabric:create --wait- for -provisioning container-create-child root broker 3 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker1 broker1 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker2 broker2 mq-create --group mesh --kind StandAlone --network mesh --networks-password admin --networks-username admin --assign-container broker3 broker3 run attached testcase mvn clean test -Dclient.threads=125 -Dmessage.count=1000000 -Ddestination.name= "mesh.3.failover.1m" -Dbroker.url= "failover:(tcp: //host1:42270,tcp://host1:37617,tcp://host1:46291)?randomize= true " I am able to reproduce the issue only with large number of threads/connections (I can reliably reproduce with 125 consumer and 125 producer threads and 1000000 messages).

SFDC Cases Counter:
SFDC Cases Links:

Description

There are three child containers in fabric8 with brokers. All brokers are in the same group. The brokers are set up to form network of brokers (brokers are connected to each other).

Then I run test case which attaches producers in consumers in following way:

each consumer/producer has its own connection
each consumer/producer disconnects after receiving 100 messages and connects again (to random broker)
Sessions are not transacted ( AUTO_ACKNOWLEDGE is used)
the producers send large number of messages

The test case counts number of sent and received messages in total. When the test finishes the number of message received is not equal to number of messages sent. The number of messagess received is usually less than number of sent messages

The queues show no messages:

dstat output on the brokers

broker1:
Name                                                Queue Size  Producer #  Consumer #   Enqueue #   Dequeue #   Forward #
ActiveMQ.DLQ                                                 1           0           0           1           0           0
mesh.3.failover.1m                                           0           0           0      362098      362098       17347
broker2:
Name                                                Queue Size  Producer #  Consumer #   Enqueue #   Dequeue #   Forward #
mesh.3.failover.1m                                           0           0           0      383995      383995       16195
broker3:
Name                                                Queue Size  Producer #  Consumer #   Enqueue #   Dequeue #   Forward #
ActiveMQ.DLQ                                                 1           0           0           1           0           0
mesh.3.failover.1m                                           0           0           0      302459      302459       15045

but when I try browse mesh.3.failover.1m queue using hawtio I can see messages there. But these messages cannot be consumed.

The following warning appears in the log of brokers multiple times:

Warning in the logs

2016-03-16 13:02:22,391 | WARN  | .125:42829@35073 | AbstractStoreCursor              | 109 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-621090 | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@22d82c12:mesh.3.failover.1m,batchResetNeeded=false,size=0,cacheEnabled=true,maxBatchSize:1,hasSpace:true,pendingCachedIds.size:553,lastSyncCachedId:ID:soaperf22.mw.lab.eng.bos.redhat.com-60962-1458147522872-1:2283:1:1:49,lastSyncCachedId-seq:4969,lastAsyncCachedId:ID:soaperf22.mw.lab.eng.bos.redhat.com-60962-1458147522872-1:2283:1:1:49,lastAsyncCachedId-seq:4969,store=permits:9446,sd=nextSeq:4970,lastRet:MessageOrderCursor:[def:4968, low:4966, high:4966],pending:0 - cursor got duplicate send ID:soaperf22.mw.lab.eng.bos.redhat.com-60962-1458147522872-1:2529:1:1:99 seq: org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask$InnerFutureTask@1ee69e9a

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

171-failover-less-messages.tar.gz
76 kB
2016/08/24 9:16 AM
broker-1.log
164 kB
2016/03/16 3:29 PM
broker-2.log
148 kB
2016/03/16 3:29 PM
broker-3.log
158 kB
2016/03/16 3:29 PM
mesh-client-reproducer.tar.gz
4 kB
2016/03/16 3:28 PM

Issue Links

is related to

ENTMQ-1822 Broker Unable To Recover After Exceeding File Descriptor Limit

Closed

Some messages cannot be received in network of brokers

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates