Starting Publisher 2020-01-21 14:13:44 DEBUG FailoverProvider:153 - Initiating initial connection attempt task 2020-01-21 14:13:44 TRACE FailoverProvider:1334 - Initial connect attempt will be performed immediately DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Connecting... 2020-01-21 14:13:44 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:44 TRACE FailoverProvider:1143 - Created Failover Task: create -> JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = null } (1) 2020-01-21 14:13:44 TRACE FailoverProvider:1256 - Failover task held until connection recovered: create -> JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = null } (1) 2020-01-21 14:13:45 DEBUG InternalLoggerFactory:43 - Using SLF4J as the default logging framework 2020-01-21 14:13:45 DEBUG PlatformDependent0:395 - -Dio.netty.noUnsafe: false 2020-01-21 14:13:45 DEBUG PlatformDependent0:851 - Java version: 8 2020-01-21 14:13:45 DEBUG PlatformDependent0:120 - sun.misc.Unsafe.theUnsafe: available 2020-01-21 14:13:45 DEBUG PlatformDependent0:144 - sun.misc.Unsafe.copyMemory: available 2020-01-21 14:13:45 DEBUG PlatformDependent0:182 - java.nio.Buffer.address: available 2020-01-21 14:13:45 DEBUG PlatformDependent0:243 - direct buffer constructor: available 2020-01-21 14:13:45 DEBUG PlatformDependent0:313 - java.nio.Bits.unaligned: available, true 2020-01-21 14:13:45 DEBUG PlatformDependent0:378 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9 2020-01-21 14:13:45 DEBUG PlatformDependent0:385 - java.nio.DirectByteBuffer.(long, int): available 2020-01-21 14:13:45 DEBUG PlatformDependent:1028 - sun.misc.Unsafe: available 2020-01-21 14:13:45 DEBUG PlatformDependent:1147 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir) 2020-01-21 14:13:45 DEBUG PlatformDependent:1226 - -Dio.netty.bitMode: 64 (sun.arch.data.model) 2020-01-21 14:13:45 DEBUG PlatformDependent:173 - -Dio.netty.maxDirectMemory: 7337410560 bytes 2020-01-21 14:13:45 DEBUG PlatformDependent:180 - -Dio.netty.uninitializedArrayAllocationThreshold: -1 2020-01-21 14:13:45 DEBUG CleanerJava6:92 - java.nio.ByteBuffer.cleaner(): available 2020-01-21 14:13:45 DEBUG PlatformDependent:200 - -Dio.netty.noPreferDirect: false 2020-01-21 14:13:45 DEBUG NativeLibraryLoader:73 - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir) 2020-01-21 14:13:45 DEBUG NativeLibraryLoader:78 - -Dio.netty.native.deleteLibAfterLoading: true 2020-01-21 14:13:45 DEBUG NativeLibraryLoader:82 - -Dio.netty.native.tryPatchShadedId: true 2020-01-21 14:13:45 DEBUG NativeLibraryLoader:344 - Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism. java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:186) at io.netty.channel.epoll.Native.(Native.java:57) at io.netty.channel.epoll.Epoll.(Epoll.java:39) at org.apache.qpid.jms.transports.netty.EpollSupport.isAvailable(EpollSupport.java:37) at org.apache.qpid.jms.transports.netty.NettyTcpTransport.connect(NettyTcpTransport.java:137) at org.apache.qpid.jms.provider.amqp.AmqpProvider.connect(AmqpProvider.java:214) at org.apache.qpid.jms.provider.failover.FailoverProvider$14.run(FailoverProvider.java:747) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:13:45 DEBUG NativeLibraryLoader:140 - netty_transport_native_epoll_x86_64 cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349) at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:186) at io.netty.channel.epoll.Native.(Native.java:57) at io.netty.channel.epoll.Epoll.(Epoll.java:39) at org.apache.qpid.jms.transports.netty.EpollSupport.isAvailable(EpollSupport.java:37) at org.apache.qpid.jms.transports.netty.NettyTcpTransport.connect(NettyTcpTransport.java:137) at org.apache.qpid.jms.provider.amqp.AmqpProvider.connect(AmqpProvider.java:214) at org.apache.qpid.jms.provider.failover.FailoverProvider$14.run(FailoverProvider.java:747) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) at java.security.AccessController.doPrivileged(Native Method) at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) ... 15 more 2020-01-21 14:13:45 DEBUG NativeLibraryLoader:340 - Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_6462055617064950251.so 2020-01-21 14:13:45 DEBUG NetUtil:139 - -Djava.net.preferIPv4Stack: false 2020-01-21 14:13:45 DEBUG NetUtil:140 - -Djava.net.preferIPv6Addresses: false 2020-01-21 14:13:45 DEBUG NetUtil:224 - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) 2020-01-21 14:13:45 DEBUG NetUtil:271 - /proc/sys/net/core/somaxconn: 4096 2020-01-21 14:13:45 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:45 DEBUG MultithreadEventLoopGroup:44 - -Dio.netty.eventLoopThreads: 16 2020-01-21 14:13:45 DEBUG InternalThreadLocalMap:54 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024 2020-01-21 14:13:45 DEBUG InternalThreadLocalMap:57 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096 2020-01-21 14:13:45 DEBUG PlatformDependent:890 - org.jctools-core.MpscChunkedArrayQueue: available 2020-01-21 14:13:45 DEBUG DefaultChannelId:79 - -Dio.netty.processId: 586504 (auto-detected) 2020-01-21 14:13:45 DEBUG DefaultChannelId:101 - -Dio.netty.machineId: 8c:16:45:ff:fe:c4:c9:f4 (auto-detected) 2020-01-21 14:13:45 DEBUG ResourceLeakDetector:129 - -Dio.netty.leakDetection.level: simple 2020-01-21 14:13:45 DEBUG ResourceLeakDetector:130 - -Dio.netty.leakDetection.targetRecords: 4 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:142 - -Dio.netty.allocator.numHeapArenas: 16 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:143 - -Dio.netty.allocator.numDirectArenas: 16 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:145 - -Dio.netty.allocator.pageSize: 8192 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:150 - -Dio.netty.allocator.maxOrder: 11 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:154 - -Dio.netty.allocator.chunkSize: 16777216 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:155 - -Dio.netty.allocator.tinyCacheSize: 512 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:156 - -Dio.netty.allocator.smallCacheSize: 256 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:157 - -Dio.netty.allocator.normalCacheSize: 64 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:158 - -Dio.netty.allocator.maxCachedBufferCapacity: 32768 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:159 - -Dio.netty.allocator.cacheTrimInterval: 8192 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:160 - -Dio.netty.allocator.cacheTrimIntervalMillis: 0 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:161 - -Dio.netty.allocator.useCacheForAllThreads: true 2020-01-21 14:13:45 DEBUG PooledByteBufAllocator:162 - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023 2020-01-21 14:13:45 DEBUG ByteBufUtil:85 - -Dio.netty.allocator.type: pooled 2020-01-21 14:13:45 DEBUG ByteBufUtil:94 - -Dio.netty.threadLocalDirectBufferSize: 0 2020-01-21 14:13:45 DEBUG ByteBufUtil:97 - -Dio.netty.maxThreadLocalCharBufferSize: 16384 2020-01-21 14:13:45 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0xd32bd1be, L:/10.10.125.235:56036 - R:node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672] 2020-01-21 14:13:45 DEBUG Recycler:96 - -Dio.netty.recycler.maxCapacityPerThread: 4096 2020-01-21 14:13:45 DEBUG Recycler:97 - -Dio.netty.recycler.maxSharedCapacityFactor: 2 2020-01-21 14:13:45 DEBUG Recycler:98 - -Dio.netty.recycler.linkCapacity: 16 2020-01-21 14:13:45 DEBUG Recycler:99 - -Dio.netty.recycler.ratio: 8 2020-01-21 14:13:45 DEBUG AbstractByteBuf:62 - -Dio.netty.buffer.checkAccessible: true 2020-01-21 14:13:45 DEBUG AbstractByteBuf:63 - -Dio.netty.buffer.checkBounds: true 2020-01-21 14:13:45 DEBUG ResourceLeakDetectorFactory:202 - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@618dffe0 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:13:45 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:45 DEBUG FailoverProvider:1159 - Executing Failover Task: create -> JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = null } (1) 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE MetaDataSupport:54 - Problem generating primary version details java.lang.NullPointerException at java.util.regex.Matcher.getTextLength(Matcher.java:1283) at java.util.regex.Matcher.reset(Matcher.java:309) at java.util.regex.Matcher.(Matcher.java:229) at java.util.regex.Pattern.matcher(Pattern.java:1093) at org.apache.qpid.jms.util.MetaDataSupport.(MetaDataSupport.java:47) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.createEndpoint(AmqpConnectionBuilder.java:118) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.createEndpoint(AmqpConnectionBuilder.java:50) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.buildResource(AmqpResourceBuilder.java:76) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.buildResource(AmqpConnectionBuilder.java:60) at org.apache.qpid.jms.provider.amqp.AmqpProvider$2.processConnectionInfo(AmqpProvider.java:451) at org.apache.qpid.jms.meta.JmsConnectionInfo.visit(JmsConnectionInfo.java:434) at org.apache.qpid.jms.provider.amqp.AmqpProvider.lambda$create$4(AmqpProvider.java:403) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 295 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 295, cap: 65536) 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:13:45 TRACE AmqpConnectionProperties:126 - Remote sent Failover Server List. 2020-01-21 14:13:45 TRACE AmqpConnectionProperties:137 - Failover Server List: [amqp://10.0.89.240:61616?amqp.vhost=10.0.89.240] 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:13:45 DEBUG AmqpConnectionBuilder:83 - AmqpConnection { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 } is now open: 2020-01-21 14:13:45 TRACE AmqpProvider:1064 - IdleTimeoutCheck being initiated, initial delay: 15000 2020-01-21 14:13:45 TRACE FailoverProvider:1277 - First connection requst has completed: 2020-01-21 14:13:45 DEBUG FailoverProvider:884 - Processing alternates uris:URI Pool { [amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672, amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672] } with new set: [amqp://10.0.89.240:61616?amqp.vhost=10.0.89.240] 2020-01-21 14:13:45 DEBUG FailoverProvider:913 - Processing alternates done new uris:URI Pool { [amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672, amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672] } 2020-01-21 14:13:45 INFO JmsConnection:1333 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 connected to remote Broker: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Connected... 2020-01-21 14:13:45 TRACE FailoverProvider:1143 - Created Failover Task: create -> JmsSessionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1 } (2) 2020-01-21 14:13:45 DEBUG FailoverProvider:1159 - Executing Failover Task: create -> JmsSessionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1 } (2) 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:13:45 TRACE FailoverProvider:1143 - Created Failover Task: create -> JmsProducerInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1, destination = MyAddress } (3) 2020-01-21 14:13:45 DEBUG FailoverProvider:1159 - Executing Failover Task: create -> JmsProducerInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1, destination = MyAddress } (3) 2020-01-21 14:13:45 DEBUG AmqpProducerBuilder:68 - Creating AmqpFixedProducer for: MyAddress 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 299 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 331 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 331, cap: 65536) 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: LINK_INIT 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: LINK_LOCAL_OPEN 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: LINK_REMOTE_OPEN 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: LINK_FLOW 2020-01-21 14:13:45 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-1, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-1 } (4) 2020-01-21 14:13:45 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-1, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-1 } (4) 2020-01-21 14:13:45 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-1, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-1 } 2020-01-21 14:13:45 TRACE NettyTcpTransport:255 - Attempted write of: 189 bytes 2020-01-21 14:13:45 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:45 TRACE NettyTcpTransport:539 - New data read: 22 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 22, cap: 65536) 2020-01-21 14:13:45 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:45 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2f371adb, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@e7e1538, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(0)... 2020-01-21 14:13:46 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-2, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-2 } (5) 2020-01-21 14:13:46 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-2, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-2 } (5) 2020-01-21 14:13:46 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-2, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-2 } 2020-01-21 14:13:46 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:46 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:46 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:46 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:46 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2f371adb, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@58afa523, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(1)... 2020-01-21 14:13:47 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-3, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-3 } (6) 2020-01-21 14:13:47 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-3, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-3 } (6) 2020-01-21 14:13:47 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-3, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-3 } 2020-01-21 14:13:47 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:47 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:47 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:47 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:47 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2f371adb, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@607519fa, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(2)... 2020-01-21 14:13:48 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-4, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-4 } (7) 2020-01-21 14:13:48 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-4, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-4 } (7) 2020-01-21 14:13:48 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-4, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-4 } 2020-01-21 14:13:48 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:48 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:48 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:48 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:48 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2f371adb, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@770336ed, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(3)... 2020-01-21 14:13:49 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-5, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-5 } (8) 2020-01-21 14:13:49 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-5, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-5 } (8) 2020-01-21 14:13:49 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-5, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-5 } 2020-01-21 14:13:49 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:49 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:49 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:49 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:49 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2f371adb, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@9aa2ae0, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(4)... 2020-01-21 14:13:50 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-6, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-6 } (9) 2020-01-21 14:13:50 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-6, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-6 } (9) 2020-01-21 14:13:50 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-6, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-6 } 2020-01-21 14:13:50 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:50 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:50 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:50 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:50 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2f371adb, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@2c09004b, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(5)... 2020-01-21 14:13:51 TRACE NettyTcpTransport:539 - New data read: 44 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 44, cap: 65536) 2020-01-21 14:13:51 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:13:51 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:13:51 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:51 DEBUG FailoverProvider:855 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:13:51 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:13:51 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:13:51 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_HEAD_CLOSED 2020-01-21 14:13:51 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_CLOSED 2020-01-21 14:13:51 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:13:51 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderConnectionRemotelyClosedException: Received error from remote peer without description [condition = amqp:connection:forced] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:128) at org.apache.qpid.jms.provider.amqp.AmqpConnection.processRemoteClose(AmqpConnection.java:148) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:13:51 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0xd32bd1be, L:/10.10.125.235:56036 ! R:node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672] 2020-01-21 14:13:51 TRACE NettyTcpTransport:365 - Firing onTransportClosed listener 2020-01-21 14:13:51 DEBUG AmqpProvider:889 - Transport connection remotely closed 2020-01-21 14:13:51 DEBUG PoolThreadCache:249 - Freed 6 thread-local buffer(s) from thread: AmqpProvider :(1):[amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:13:51 TRACE FailoverProvider:1349 - Initial Reconnect attempt will be performed immediately 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xb4dc0c56, L:/10.10.125.235:59216] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x65b14e9e, L:/10.10.125.235:56042] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 TRACE FailoverProvider:779 - Connection attempt:[1] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 10 milliseconds 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xd6c3942e, L:/10.10.125.235:59220] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x281fea3c, L:/10.10.125.235:56046] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 TRACE FailoverProvider:779 - Connection attempt:[2] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 20 milliseconds 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7 } (10) 2020-01-21 14:13:51 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7 } (10) 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x07775b24, L:/10.10.125.235:59224] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x54d70ad3, L:/10.10.125.235:56050] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 TRACE FailoverProvider:779 - Connection attempt:[3] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 40 milliseconds 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xc06d7a2f, L:/10.10.125.235:59228] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x04f8d335, L:/10.10.125.235:56054] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 TRACE FailoverProvider:779 - Connection attempt:[4] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 80 milliseconds 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x1223c8f1, L:/10.10.125.235:59232] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x96e37eaf, L:/10.10.125.235:56058] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 TRACE FailoverProvider:779 - Connection attempt:[5] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 160 milliseconds 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x9b05e68e, L:/10.10.125.235:59236] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:51 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:51 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xce6728f4, L:/10.10.125.235:56062] 2020-01-21 14:13:51 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:51 TRACE FailoverProvider:779 - Connection attempt:[6] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:51 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 320 milliseconds 2020-01-21 14:13:52 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:52 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:52 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x0463e6ff, L:/10.10.125.235:59240] 2020-01-21 14:13:52 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:13:52 INFO FailoverProvider:751 - Connection attempt:[7] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:52 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:52 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:52 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x31e45628, L:/10.10.125.235:56066] 2020-01-21 14:13:52 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:52 INFO FailoverProvider:751 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:13:52 TRACE FailoverProvider:779 - Connection attempt:[7] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:13:52 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 640 milliseconds 2020-01-21 14:13:52 DEBUG FailoverProvider:744 - Connection attempt:[8] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:13:52 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:13:52 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0xa9a04dbf, L:/10.10.125.235:59244 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:13:52 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:13:52 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:13:53 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:53 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:13:53 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 185 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 185, cap: 65536) 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:13:53 DEBUG AmqpConnectionBuilder:83 - AmqpConnection { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 } is now open: 2020-01-21 14:13:53 TRACE AmqpProvider:1064 - IdleTimeoutCheck being initiated, initial delay: 15000 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:13:53 DEBUG AmqpProducerBuilder:68 - Creating AmqpFixedProducer for: MyAddress 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 299 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 331 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 331, cap: 65536) 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: LINK_INIT 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: LINK_LOCAL_OPEN 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: LINK_REMOTE_OPEN 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: LINK_FLOW 2020-01-21 14:13:53 DEBUG JmsConnection:1292 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is finalizing recovery. 2020-01-21 14:13:53 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7 } (10) 2020-01-21 14:13:53 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-7 } 2020-01-21 14:13:53 TRACE NettyTcpTransport:255 - Attempted write of: 189 bytes 2020-01-21 14:13:53 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:53 TRACE NettyTcpTransport:539 - New data read: 22 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 22, cap: 65536) 2020-01-21 14:13:53 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:53 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@2f6c2369, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(6)... 2020-01-21 14:13:54 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-8, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-8 } (11) 2020-01-21 14:13:54 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-8, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-8 } (11) 2020-01-21 14:13:54 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-8, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-8 } 2020-01-21 14:13:54 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:54 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:54 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:54 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:54 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@23239ad9, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(7)... 2020-01-21 14:13:55 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-9, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-9 } (12) 2020-01-21 14:13:55 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-9, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-9 } (12) 2020-01-21 14:13:55 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-9, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-9 } 2020-01-21 14:13:55 TRACE NettyTcpTransport:255 - Attempted write of: 190 bytes 2020-01-21 14:13:55 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:55 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:55 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:55 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@7e17857e, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(8)... 2020-01-21 14:13:56 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-10, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-10 } (13) 2020-01-21 14:13:56 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-10, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-10 } (13) 2020-01-21 14:13:56 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-10, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-10 } 2020-01-21 14:13:56 TRACE NettyTcpTransport:255 - Attempted write of: 191 bytes 2020-01-21 14:13:56 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:56 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:56 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:56 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@4954f391, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(9)... 2020-01-21 14:13:57 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-11, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-11 } (14) 2020-01-21 14:13:57 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-11, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-11 } (14) 2020-01-21 14:13:57 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-11, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-11 } 2020-01-21 14:13:57 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:13:57 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:57 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:57 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:57 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@7edf6b9, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(10)... 2020-01-21 14:13:58 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-12, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-12 } (15) 2020-01-21 14:13:58 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-12, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-12 } (15) 2020-01-21 14:13:58 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-12, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-12 } 2020-01-21 14:13:58 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:13:58 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:58 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:58 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:58 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@382751cf, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(11)... 2020-01-21 14:13:58 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-13, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-13 } (16) 2020-01-21 14:13:58 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-13, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-13 } (16) 2020-01-21 14:13:58 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-13, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-13 } 2020-01-21 14:13:58 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:13:58 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:58 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:58 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:58 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@7a591fc0, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(12)... 2020-01-21 14:13:59 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-14, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-14 } (17) 2020-01-21 14:13:59 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-14, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-14 } (17) 2020-01-21 14:13:59 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-14, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-14 } 2020-01-21 14:13:59 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:13:59 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:13:59 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:13:59 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:13:59 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@14ffa361, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(13)... 2020-01-21 14:14:00 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-15, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-15 } (18) 2020-01-21 14:14:00 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-15, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-15 } (18) 2020-01-21 14:14:00 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-15, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-15 } 2020-01-21 14:14:00 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:00 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:00 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:00 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:00 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@7ccfc779, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(14)... 2020-01-21 14:14:01 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-16, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-16 } (19) 2020-01-21 14:14:01 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-16, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-16 } (19) 2020-01-21 14:14:01 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-16, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-16 } 2020-01-21 14:14:01 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:01 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:01 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:01 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:01 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@7e644806, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(15)... 2020-01-21 14:14:02 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-17, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-17 } (20) 2020-01-21 14:14:02 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-17, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-17 } (20) 2020-01-21 14:14:02 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-17, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-17 } 2020-01-21 14:14:02 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:02 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:02 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:02 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:02 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@5b9d9d30, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@4c71ff15, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(16)... 2020-01-21 14:14:03 TRACE NettyTcpTransport:539 - New data read: 44 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 44, cap: 65536) 2020-01-21 14:14:03 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:03 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:03 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:03 DEBUG FailoverProvider:855 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:03 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:03 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:03 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderConnectionRemotelyClosedException: Received error from remote peer without description [condition = amqp:connection:forced] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:128) at org.apache.qpid.jms.provider.amqp.AmqpConnection.processRemoteClose(AmqpConnection.java:148) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:03 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:03 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_HEAD_CLOSED 2020-01-21 14:14:03 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_CLOSED 2020-01-21 14:14:03 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0xa9a04dbf, L:/10.10.125.235:59244 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:03 DEBUG PoolThreadCache:249 - Freed 6 thread-local buffer(s) from thread: AmqpProvider :(16):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:03 TRACE FailoverProvider:1349 - Initial Reconnect attempt will be performed immediately 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xe969d599, L:/10.10.125.235:56074] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x40beec50, L:/10.10.125.235:59252] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 TRACE FailoverProvider:779 - Connection attempt:[1] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 10 milliseconds 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x321643ba, L:/10.10.125.235:56078] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x4724b1f6, L:/10.10.125.235:59256] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 TRACE FailoverProvider:779 - Connection attempt:[2] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 20 milliseconds 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x8d02a376, L:/10.10.125.235:56082] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18 } (21) 2020-01-21 14:14:03 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18 } (21) 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x8d3bb3ad, L:/10.10.125.235:59260] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 TRACE FailoverProvider:779 - Connection attempt:[3] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 40 milliseconds 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xfa0bef87, L:/10.10.125.235:56086] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xb910454f, L:/10.10.125.235:59264] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 TRACE FailoverProvider:779 - Connection attempt:[4] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 80 milliseconds 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xecd4dbaf, L:/10.10.125.235:56090] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:03 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:03 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x39fca152, L:/10.10.125.235:59268] 2020-01-21 14:14:03 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:03 TRACE FailoverProvider:779 - Connection attempt:[5] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:03 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 160 milliseconds 2020-01-21 14:14:04 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:04 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:04 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x85d39d76, L:/10.10.125.235:56094] 2020-01-21 14:14:04 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:04 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:04 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:04 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:04 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x441c288d, L:/10.10.125.235:59272] 2020-01-21 14:14:04 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:04 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:04 TRACE FailoverProvider:779 - Connection attempt:[6] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:04 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 320 milliseconds 2020-01-21 14:14:04 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:04 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:04 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xf359021d, L:/10.10.125.235:56098] 2020-01-21 14:14:04 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:04 INFO FailoverProvider:751 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:04 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:04 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:04 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xd339ec74, L:/10.10.125.235:59276] 2020-01-21 14:14:04 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:04 INFO FailoverProvider:751 - Connection attempt:[7] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:04 TRACE FailoverProvider:779 - Connection attempt:[7] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:04 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 640 milliseconds 2020-01-21 14:14:05 DEBUG FailoverProvider:744 - Connection attempt:[8] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:05 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:05 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0xa8f6315a, L:/10.10.125.235:56102 - R:node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672] 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:05 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:05 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:05 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 186 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 186, cap: 65536) 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:14:05 DEBUG AmqpConnectionBuilder:83 - AmqpConnection { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 } is now open: 2020-01-21 14:14:05 TRACE AmqpProvider:1064 - IdleTimeoutCheck being initiated, initial delay: 15000 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:14:05 DEBUG AmqpProducerBuilder:68 - Creating AmqpFixedProducer for: MyAddress 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 299 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 331 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 331, cap: 65536) 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: LINK_INIT 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: LINK_LOCAL_OPEN 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: LINK_REMOTE_OPEN 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: LINK_FLOW 2020-01-21 14:14:05 DEBUG JmsConnection:1292 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is finalizing recovery. 2020-01-21 14:14:05 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18 } (21) 2020-01-21 14:14:05 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-18 } 2020-01-21 14:14:05 TRACE NettyTcpTransport:255 - Attempted write of: 191 bytes 2020-01-21 14:14:05 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:05 TRACE NettyTcpTransport:539 - New data read: 22 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 22, cap: 65536) 2020-01-21 14:14:05 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:05 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@901ffc, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@5880ad5a, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(17)... 2020-01-21 14:14:06 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-19, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-19 } (22) 2020-01-21 14:14:06 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-19, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-19 } (22) 2020-01-21 14:14:06 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-19, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-19 } 2020-01-21 14:14:06 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:06 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:06 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:06 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:06 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@901ffc, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@7859ecec, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(18)... 2020-01-21 14:14:07 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-20, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-20 } (23) 2020-01-21 14:14:07 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-20, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-20 } (23) 2020-01-21 14:14:07 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-20, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-20 } 2020-01-21 14:14:07 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:07 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:07 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:07 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:07 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@901ffc, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@98cffba, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(19)... 2020-01-21 14:14:08 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-21, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-21 } (24) 2020-01-21 14:14:08 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-21, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-21 } (24) 2020-01-21 14:14:08 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-21, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-21 } 2020-01-21 14:14:08 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:08 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:08 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:08 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:08 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@901ffc, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@9637e88, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(20)... 2020-01-21 14:14:09 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-22, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-22 } (25) 2020-01-21 14:14:09 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-22, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-22 } (25) 2020-01-21 14:14:09 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-22, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-22 } 2020-01-21 14:14:09 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:09 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:09 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:09 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:09 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@901ffc, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@5454e883, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(21)... 2020-01-21 14:14:10 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-23, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-23 } (26) 2020-01-21 14:14:10 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-23, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-23 } (26) 2020-01-21 14:14:10 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-23, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-23 } 2020-01-21 14:14:10 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:10 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:10 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:10 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:10 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@901ffc, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@635d4069, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(22)... 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 44 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 44, cap: 65536) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 DEBUG FailoverProvider:855 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:11 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:11 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderConnectionRemotelyClosedException: Received error from remote peer without description [condition = amqp:connection:forced] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:128) at org.apache.qpid.jms.provider.amqp.AmqpConnection.processRemoteClose(AmqpConnection.java:148) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_HEAD_CLOSED 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_CLOSED 2020-01-21 14:14:11 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0xa8f6315a, L:/10.10.125.235:56102 ! R:node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672] 2020-01-21 14:14:11 DEBUG PoolThreadCache:249 - Freed 6 thread-local buffer(s) from thread: AmqpProvider :(31):[amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:11 TRACE FailoverProvider:1349 - Initial Reconnect attempt will be performed immediately 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xf694e606, L:/10.10.125.235:59280] 2020-01-21 14:14:11 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:11 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xe9739f0a, L:/10.10.125.235:56106] 2020-01-21 14:14:11 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:11 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:11 TRACE FailoverProvider:779 - Connection attempt:[1] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:11 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 10 milliseconds 2020-01-21 14:14:11 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24 } (27) 2020-01-21 14:14:11 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24 } (27) 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xf0bc62e4, L:/10.10.125.235:59284] 2020-01-21 14:14:11 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:11 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x190d2c0c, L:/10.10.125.235:56110] 2020-01-21 14:14:11 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:11 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:11 TRACE FailoverProvider:779 - Connection attempt:[2] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:11 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 20 milliseconds 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xa4e46837, L:/10.10.125.235:59288] 2020-01-21 14:14:11 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:11 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xf2d21497, L:/10.10.125.235:56114] 2020-01-21 14:14:11 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:11 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:11 TRACE FailoverProvider:779 - Connection attempt:[3] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:11 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 40 milliseconds 2020-01-21 14:14:11 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:11 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:11 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x87bacfd4, L:/10.10.125.235:59292 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:11 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:11 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:11 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 185 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 185, cap: 65536) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:14:11 DEBUG AmqpConnectionBuilder:83 - AmqpConnection { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 } is now open: 2020-01-21 14:14:11 TRACE AmqpProvider:1064 - IdleTimeoutCheck being initiated, initial delay: 15000 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:14:11 DEBUG AmqpProducerBuilder:68 - Creating AmqpFixedProducer for: MyAddress 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 299 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 331 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 331, cap: 65536) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: LINK_INIT 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: LINK_LOCAL_OPEN 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: LINK_REMOTE_OPEN 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: LINK_FLOW 2020-01-21 14:14:11 DEBUG JmsConnection:1292 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is finalizing recovery. 2020-01-21 14:14:11 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24 } (27) 2020-01-21 14:14:11 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-24 } 2020-01-21 14:14:11 TRACE NettyTcpTransport:255 - Attempted write of: 191 bytes 2020-01-21 14:14:11 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:11 TRACE NettyTcpTransport:539 - New data read: 22 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 22, cap: 65536) 2020-01-21 14:14:11 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:11 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@76f424aa, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(23)... 2020-01-21 14:14:12 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-25, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-25 } (28) 2020-01-21 14:14:12 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-25, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-25 } (28) 2020-01-21 14:14:12 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-25, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-25 } 2020-01-21 14:14:12 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:12 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:12 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:12 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:12 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@63520012, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(24)... 2020-01-21 14:14:13 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-26, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-26 } (29) 2020-01-21 14:14:13 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-26, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-26 } (29) 2020-01-21 14:14:13 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-26, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-26 } 2020-01-21 14:14:13 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:13 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:13 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:13 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:13 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@19982156, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(25)... 2020-01-21 14:14:14 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-27, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-27 } (30) 2020-01-21 14:14:14 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-27, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-27 } (30) 2020-01-21 14:14:14 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-27, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-27 } 2020-01-21 14:14:14 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:14 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:14 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:14 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:14 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@471c9452, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(26)... 2020-01-21 14:14:15 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-28, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-28 } (31) 2020-01-21 14:14:15 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-28, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-28 } (31) 2020-01-21 14:14:15 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-28, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-28 } 2020-01-21 14:14:15 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:15 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:15 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:15 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:15 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@109a396f, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(27)... 2020-01-21 14:14:16 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-29, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-29 } (32) 2020-01-21 14:14:16 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-29, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-29 } (32) 2020-01-21 14:14:16 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-29, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-29 } 2020-01-21 14:14:16 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:16 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:16 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:16 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:16 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@63bbf9fe, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(28)... 2020-01-21 14:14:17 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-30, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-30 } (33) 2020-01-21 14:14:17 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-30, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-30 } (33) 2020-01-21 14:14:17 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-30, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-30 } 2020-01-21 14:14:17 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:17 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:17 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:17 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:17 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@3bf1f64e, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(29)... 2020-01-21 14:14:18 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-31, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-31 } (34) 2020-01-21 14:14:18 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-31, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-31 } (34) 2020-01-21 14:14:18 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-31, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-31 } 2020-01-21 14:14:18 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:18 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:18 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:18 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:18 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@18ec5b66, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(30)... 2020-01-21 14:14:19 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-32, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-32 } (35) 2020-01-21 14:14:19 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-32, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-32 } (35) 2020-01-21 14:14:19 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-32, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-32 } 2020-01-21 14:14:19 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:19 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:19 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:19 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:19 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@597c4855, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(31)... 2020-01-21 14:14:20 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-33, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-33 } (36) 2020-01-21 14:14:20 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-33, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-33 } (36) 2020-01-21 14:14:20 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-33, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-33 } 2020-01-21 14:14:20 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:20 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:20 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:20 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:20 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@65b42e8d, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(32)... 2020-01-21 14:14:21 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-34, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-34 } (37) 2020-01-21 14:14:21 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-34, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-34 } (37) 2020-01-21 14:14:21 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-34, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-34 } 2020-01-21 14:14:21 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:21 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:21 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:21 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:21 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@51c3e7f3, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(33)... 2020-01-21 14:14:22 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-35, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-35 } (38) 2020-01-21 14:14:22 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-35, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-35 } (38) 2020-01-21 14:14:22 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-35, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-35 } 2020-01-21 14:14:22 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:22 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:22 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:22 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:22 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@2b8eb1c6, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(34)... 2020-01-21 14:14:23 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-36, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-36 } (39) 2020-01-21 14:14:23 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-36, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-36 } (39) 2020-01-21 14:14:23 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-36, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-36 } 2020-01-21 14:14:23 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:23 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:23 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:23 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:23 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@108b6a02, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(35)... 2020-01-21 14:14:23 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-37, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-37 } (40) 2020-01-21 14:14:23 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-37, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-37 } (40) 2020-01-21 14:14:23 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-37, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-37 } 2020-01-21 14:14:23 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:23 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:23 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:23 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:23 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@548feed8, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(36)... 2020-01-21 14:14:24 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-38, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-38 } (41) 2020-01-21 14:14:24 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-38, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-38 } (41) 2020-01-21 14:14:24 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-38, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-38 } 2020-01-21 14:14:24 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:24 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:24 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:24 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:24 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@1b78b45b, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@1a3d77c5, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(37)... 2020-01-21 14:14:25 TRACE NettyTcpTransport:539 - New data read: 44 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 44, cap: 65536) 2020-01-21 14:14:25 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:25 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:25 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:25 DEBUG FailoverProvider:855 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:25 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:25 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:25 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_HEAD_CLOSED 2020-01-21 14:14:25 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_CLOSED 2020-01-21 14:14:25 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:25 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderConnectionRemotelyClosedException: Received error from remote peer without description [condition = amqp:connection:forced] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:128) at org.apache.qpid.jms.provider.amqp.AmqpConnection.processRemoteClose(AmqpConnection.java:148) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:25 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x87bacfd4, L:/10.10.125.235:59292 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:25 TRACE NettyTcpTransport:365 - Firing onTransportClosed listener 2020-01-21 14:14:25 DEBUG AmqpProvider:889 - Transport connection remotely closed 2020-01-21 14:14:25 DEBUG PoolThreadCache:249 - Freed 6 thread-local buffer(s) from thread: AmqpProvider :(38):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:25 TRACE FailoverProvider:1349 - Initial Reconnect attempt will be performed immediately 2020-01-21 14:14:25 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:25 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:25 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xf64e7ab3, L:/10.10.125.235:56120] 2020-01-21 14:14:25 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:25 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:25 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:25 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:25 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39 } (42) 2020-01-21 14:14:25 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39 } (42) 2020-01-21 14:14:25 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x32d9c01d, L:/10.10.125.235:59298] 2020-01-21 14:14:25 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:25 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:25 TRACE FailoverProvider:779 - Connection attempt:[1] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:25 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 10 milliseconds 2020-01-21 14:14:25 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:25 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:25 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xdfd847c2, L:/10.10.125.235:56124] 2020-01-21 14:14:25 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:25 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:25 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:25 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:25 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x96352d78, L:/10.10.125.235:59302] 2020-01-21 14:14:25 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:25 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:25 TRACE FailoverProvider:779 - Connection attempt:[2] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:25 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 20 milliseconds 2020-01-21 14:14:25 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:25 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:25 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x7b65a6d4, L:/10.10.125.235:56128] 2020-01-21 14:14:25 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:25 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:25 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:25 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x1b9e2811, L:/10.10.125.235:59306] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 TRACE FailoverProvider:779 - Connection attempt:[3] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 40 milliseconds 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x103447db, L:/10.10.125.235:56132] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xcf047306, L:/10.10.125.235:59310] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 TRACE FailoverProvider:779 - Connection attempt:[4] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 80 milliseconds 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xd02510db, L:/10.10.125.235:56136] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x766889bb, L:/10.10.125.235:59314] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 TRACE FailoverProvider:779 - Connection attempt:[5] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 160 milliseconds 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x701d3fac, L:/10.10.125.235:56140] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x37994c81, L:/10.10.125.235:59318] 2020-01-21 14:14:26 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:26 TRACE FailoverProvider:779 - Connection attempt:[6] failed error: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:26 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 320 milliseconds 2020-01-21 14:14:26 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:26 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:26 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x45001996, L:/10.10.125.235:56144 - R:node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672] 2020-01-21 14:14:26 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:26 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:27 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:27 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:27 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 186 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 186, cap: 65536) 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:14:27 DEBUG AmqpConnectionBuilder:83 - AmqpConnection { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 } is now open: 2020-01-21 14:14:27 TRACE AmqpProvider:1064 - IdleTimeoutCheck being initiated, initial delay: 15000 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 32 bytes 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 34 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 34, cap: 65536) 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: SESSION_INIT 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: SESSION_LOCAL_OPEN 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: SESSION_REMOTE_OPEN 2020-01-21 14:14:27 DEBUG AmqpProducerBuilder:68 - Creating AmqpFixedProducer for: MyAddress 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 299 bytes 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 331 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 331, cap: 65536) 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: LINK_INIT 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: LINK_LOCAL_OPEN 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: LINK_REMOTE_OPEN 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: LINK_FLOW 2020-01-21 14:14:27 DEBUG JmsConnection:1292 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is finalizing recovery. 2020-01-21 14:14:27 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39 } (42) 2020-01-21 14:14:27 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-39 } 2020-01-21 14:14:27 TRACE NettyTcpTransport:255 - Attempted write of: 191 bytes 2020-01-21 14:14:27 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:27 TRACE NettyTcpTransport:539 - New data read: 22 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 22, cap: 65536) 2020-01-21 14:14:27 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:27 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2c62570f, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@1bf640b6, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(38)... 2020-01-21 14:14:28 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-40, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-40 } (43) 2020-01-21 14:14:28 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-40, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-40 } (43) 2020-01-21 14:14:28 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-40, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-40 } 2020-01-21 14:14:28 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:28 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:28 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:28 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:28 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2c62570f, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@5468c231, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(39)... 2020-01-21 14:14:29 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-41, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-41 } (44) 2020-01-21 14:14:29 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-41, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-41 } (44) 2020-01-21 14:14:29 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-41, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-41 } 2020-01-21 14:14:29 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:29 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:29 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:29 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:29 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2c62570f, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@4a397460, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(40)... 2020-01-21 14:14:30 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-42, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-42 } (45) 2020-01-21 14:14:30 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-42, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-42 } (45) 2020-01-21 14:14:30 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-42, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-42 } 2020-01-21 14:14:30 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:30 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:30 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:30 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:30 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2c62570f, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@745d8807, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(41)... 2020-01-21 14:14:31 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-43, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-43 } (46) 2020-01-21 14:14:31 DEBUG FailoverProvider:1159 - Executing Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-43, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-43 } (46) 2020-01-21 14:14:31 TRACE AmqpFixedProducer:123 - Producer sending message: JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-43, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-43 } 2020-01-21 14:14:31 TRACE NettyTcpTransport:255 - Attempted write of: 192 bytes 2020-01-21 14:14:31 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:31 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:31 TRACE AmqpProvider:932 - New Proton Event: DELIVERY 2020-01-21 14:14:31 TRACE AmqpFixedProducer:220 - Outcome of delivery was accepted: DeliveryImpl [_tag=[0], _link=org.apache.qpid.proton.engine.impl.SenderImpl@2c62570f, _deliveryState=null, _settled=false, _remoteSettled=true, _remoteDeliveryState=Accepted{}, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@5d0795ca, _data Size=0, _complete=true, _updated=true, _done=true] DEBUG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>: Message Sent(42)... 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 44 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 44, cap: 65536) 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 DEBUG FailoverProvider:855 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:32 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:connection:forced] 2020-01-21 14:14:32 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderConnectionRemotelyClosedException: Received error from remote peer without description [condition = amqp:connection:forced] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:128) at org.apache.qpid.jms.provider.amqp.AmqpConnection.processRemoteClose(AmqpConnection.java:148) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_HEAD_CLOSED 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_CLOSED 2020-01-21 14:14:32 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x45001996, L:/10.10.125.235:56144 ! R:node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672] 2020-01-21 14:14:32 DEBUG PoolThreadCache:249 - Freed 6 thread-local buffer(s) from thread: AmqpProvider :(51):[amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:32 TRACE FailoverProvider:1349 - Initial Reconnect attempt will be performed immediately 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xba3df9f3, L:/10.10.125.235:59322] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE FailoverProvider:1143 - Created Failover Task: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:32 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x3e8c7752, L:/10.10.125.235:56148] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[1] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 TRACE FailoverProvider:779 - Connection attempt:[1] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 10 milliseconds 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x57f0be2b, L:/10.10.125.235:59326] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xe68cdbe2, L:/10.10.125.235:56152] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[2] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 TRACE FailoverProvider:779 - Connection attempt:[2] failed error: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 20 milliseconds 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x9491f02a, L:/10.10.125.235:59330 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:32 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:32 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:32 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:32 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:32 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:32 TRACE FailoverProvider:697 - Connection attempt:[3] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:32 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:32 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x9491f02a, L:/10.10.125.235:59330 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(56):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:32 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:32 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 40 milliseconds 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xeb55d906, L:/10.10.125.235:56156] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[4] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x588e328b, L:/10.10.125.235:59334 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:32 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:32 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:32 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:32 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:32 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:32 TRACE FailoverProvider:697 - Connection attempt:[4] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:32 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x588e328b, L:/10.10.125.235:59334 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(58):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:32 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:32 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 80 milliseconds 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x2cc444fc, L:/10.10.125.235:56160] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[5] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x2eeb7915, L:/10.10.125.235:59338 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:32 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:32 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:32 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:32 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:32 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:32 TRACE FailoverProvider:697 - Connection attempt:[5] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:32 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:32 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x2eeb7915, L:/10.10.125.235:59338 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(60):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:32 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:32 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 160 milliseconds 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x78a81069, L:/10.10.125.235:56164] 2020-01-21 14:14:32 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:32 INFO FailoverProvider:751 - Connection attempt:[6] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:32 DEBUG FailoverProvider:744 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:32 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:32 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x1f9ba6ad, L:/10.10.125.235:59342 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:32 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:32 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:32 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:32 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:32 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:33 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:33 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:33 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:33 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:33 TRACE FailoverProvider:697 - Connection attempt:[6] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:33 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:33 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x1f9ba6ad, L:/10.10.125.235:59342 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:33 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(62):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:33 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:33 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 320 milliseconds 2020-01-21 14:14:33 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:33 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:33 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x0c784ebc, L:/10.10.125.235:56168] 2020-01-21 14:14:33 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:33 INFO FailoverProvider:751 - Connection attempt:[7] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:33 DEBUG FailoverProvider:744 - Connection attempt:[7] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:33 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:33 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0xbcde5ef4, L:/10.10.125.235:59346 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:33 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:33 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:33 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:33 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:33 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:33 TRACE FailoverProvider:697 - Connection attempt:[7] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:33 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:33 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:33 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:33 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:33 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:33 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0xbcde5ef4, L:/10.10.125.235:59346 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:33 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(64):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:33 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:33 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 640 milliseconds 2020-01-21 14:14:34 DEBUG FailoverProvider:744 - Connection attempt:[8] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:34 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:34 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x445d863d, L:/10.10.125.235:56172] 2020-01-21 14:14:34 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:34 INFO FailoverProvider:751 - Connection attempt:[8] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:34 DEBUG FailoverProvider:744 - Connection attempt:[8] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:34 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:34 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x92200bfa, L:/10.10.125.235:59350 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:34 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:34 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:34 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:34 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:34 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:34 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:34 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:34 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:34 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:34 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:34 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:34 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:34 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:34 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:34 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:34 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:34 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:34 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:34 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:34 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:34 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:34 TRACE FailoverProvider:697 - Connection attempt:[8] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:34 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:34 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:34 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:34 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:34 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:34 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x92200bfa, L:/10.10.125.235:59350 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:34 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(66):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:34 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:34 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 1280 milliseconds 2020-01-21 14:14:35 DEBUG FailoverProvider:744 - Connection attempt:[9] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:35 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:35 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x89198919, L:/10.10.125.235:56176] 2020-01-21 14:14:35 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:35 INFO FailoverProvider:751 - Connection attempt:[9] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:35 DEBUG FailoverProvider:744 - Connection attempt:[9] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:35 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:35 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x2a841b7c, L:/10.10.125.235:59354 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:35 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:35 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:35 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:35 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:35 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:35 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:35 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:35 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:35 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:35 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:35 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:35 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:35 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:35 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:35 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:35 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:35 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:35 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:35 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:35 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:35 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:35 TRACE FailoverProvider:697 - Connection attempt:[9] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:35 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:35 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:35 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:35 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:35 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:35 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x2a841b7c, L:/10.10.125.235:59354 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:35 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(68):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:35 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:35 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 2560 milliseconds 2020-01-21 14:14:38 DEBUG FailoverProvider:744 - Connection attempt:[10] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:38 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:38 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0x57df3bcb, L:/10.10.125.235:56180] 2020-01-21 14:14:38 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:38 INFO FailoverProvider:751 - Connection attempt:[10] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:38 DEBUG FailoverProvider:744 - Connection attempt:[10] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:38 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:38 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x191d2cf7, L:/10.10.125.235:59358 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:38 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:38 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:38 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:38 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:38 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:38 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:38 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:38 TRACE NettyTcpTransport:539 - New data read: 16 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 16, cap: 65536) 2020-01-21 14:14:38 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:38 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:38 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:38 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:38 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:38 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:38 TRACE NettyTcpTransport:539 - New data read: 8 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 8, cap: 65536) 2020-01-21 14:14:38 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:38 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:38 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:38 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:38 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:38 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:38 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:38 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:38 TRACE FailoverProvider:697 - Connection attempt:[10] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:38 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:38 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:38 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:38 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:38 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x191d2cf7, L:/10.10.125.235:59358 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:38 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(70):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:38 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:38 WARN FailoverProvider:1326 - Failed to connect after: 10 attempt(s) continuing to retry. 2020-01-21 14:14:38 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 5120 milliseconds 2020-01-21 14:14:43 DEBUG FailoverProvider:744 - Connection attempt:[11] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:43 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:43 TRACE NettyTcpTransport:377 - Exception on channel! Channel is [id: 0xa78383b8, L:/10.10.125.235:56186] 2020-01-21 14:14:43 TRACE NettyTcpTransport:399 - Holding error until connect succeeds: finishConnect(..) failed: Connection refused: node-1.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.147:5672 2020-01-21 14:14:43 INFO FailoverProvider:751 - Connection attempt:[11] to: amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:43 DEBUG FailoverProvider:744 - Connection attempt:[11] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 in-progress 2020-01-21 14:14:43 TRACE NettyTcpTransport:143 - Netty Transport using Epoll mode 2020-01-21 14:14:43 TRACE NettyTcpTransport:358 - Channel has become active! Channel is [id: 0x1fd7922b, L:/10.10.125.235:59364 - R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:43 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:43 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:43 TRACE NettyTcpTransport:539 - New data read: 42 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 42, cap: 65536) 2020-01-21 14:14:43 INFO SaslMechanismFinder:106 - Best match for SASL auth was: SASL-PLAIN 2020-01-21 14:14:43 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_INIT 2020-01-21 14:14:43 TRACE NettyTcpTransport:255 - Attempted write of: 78 bytes 2020-01-21 14:14:43 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:43 TRACE NettyTcpTransport:539 - New data read: 24 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 24, cap: 65536) 2020-01-21 14:14:43 DEBUG FailoverProvider:646 - Signalling connection recovery: AmqpProvider: node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 2020-01-21 14:14:43 DEBUG JmsConnection:1264 - Connection ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1 is starting recovery. 2020-01-21 14:14:43 TRACE NettyTcpTransport:255 - Attempted write of: 8 bytes 2020-01-21 14:14:43 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:43 TRACE NettyTcpTransport:255 - Attempted write of: 351 bytes 2020-01-21 14:14:43 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:43 TRACE NettyTcpTransport:539 - New data read: 154 bytes incomsing: PooledUnsafeDirectByteBuf(ridx: 0, widx: 154, cap: 65536) 2020-01-21 14:14:43 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_OPEN 2020-01-21 14:14:43 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_OPEN 2020-01-21 14:14:43 TRACE AmqpConnectionProperties:121 - Remote sent Connection Establishment Failed marker. 2020-01-21 14:14:43 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_REMOTE_CLOSE 2020-01-21 14:14:43 WARN AmqpResourceBuilder:196 - Open of resource:(JmsConnectionInfo { ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1, configuredURI = failover:(amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672,amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672)?failover.amqpOpenServerListAction=IGNORE&failover.startupMaxReconnectAttempts=4, connectedURI = amqp://node-1.qdramq.lab.upshift.rdu2.redhat.com:5672 }) failed: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:43 TRACE AmqpProvider:932 - New Proton Event: TRANSPORT_TAIL_CLOSED 2020-01-21 14:14:43 TRACE FailoverProvider:697 - Connection attempt:[11] to: amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672 failed 2020-01-21 14:14:43 TRACE AmqpProvider:932 - New Proton Event: CONNECTION_LOCAL_CLOSE 2020-01-21 14:14:43 DEBUG FailoverProvider:549 - handling Provider failure: Received error from remote peer without description [condition = amqp:invalid-field] 2020-01-21 14:14:43 TRACE FailoverProvider:550 - stack org.apache.qpid.jms.provider.exceptions.ProviderInvalidClientIDException: Received error from remote peer without description [condition = amqp:invalid-field] at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToConnectionClosedException(AmqpSupport.java:136) at org.apache.qpid.jms.provider.amqp.builders.AmqpConnectionBuilder.getOpenAbortExceptionFromRemote(AmqpConnectionBuilder.java:158) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.handleClosed(AmqpResourceBuilder.java:185) at org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder.processRemoteClose(AmqpResourceBuilder.java:129) at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:940) at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:850) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:542) at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:535) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) 2020-01-21 14:14:43 TRACE NettyTcpTransport:255 - Attempted write of: 12 bytes 2020-01-21 14:14:43 TRACE NettyTcpTransport:269 - Attempted flush of pending writes 2020-01-21 14:14:43 TRACE NettyTcpTransport:363 - Channel has gone inactive! Channel is [id: 0x1fd7922b, L:/10.10.125.235:59364 ! R:node-2.qdramq.lab.upshift.rdu2.redhat.com/10.0.89.240:5672] 2020-01-21 14:14:43 DEBUG PoolThreadCache:249 - Freed 3 thread-local buffer(s) from thread: AmqpProvider :(72):[amqp://node-2.qdramq.lab.upshift.rdu2.redhat.com:5672] 2020-01-21 14:14:43 TRACE FailoverProvider:1256 - Failover task held until connection recovered: send -> JmsOutboundMessageDispatch {dispatchId = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44, MessageID = ID:27281b71-9f0d-4bbc-bb07-6ec25b2365fa:1:1:1-44 } (47) 2020-01-21 14:14:43 TRACE FailoverProvider:1354 - Next reconnect attempt will be in 10240 milliseconds