Currently,

client.invoke(new byte[1024 * 8 - 6]);

works, but

client.invoke(new byte[1024 * 8 - 5]);

hangs.

The problem is that on the sending side,org.jboss.remoting3.remote.RemoteConnectionHandler hard codes a buffer size of 65536:

private final Pool<ByteBuffer> bufferPool = Buffers.createHeapByteBufferAllocator(65536);

so the sender can transmit up to 65535 bytes at a time. But on the receiving side, FramingChannelListener defaults to a buffer size of 4 * 2048:

FramingChannelListener(final OptionMap optionMap, final AbstractMessageHandler messageHandler)

{ this.messageHandler = messageHandler; maxSize = optionMap.get(Options.MAX_INBOUND_MESSAGE_SIZE, 2048); receiveBuffer = ByteBuffer.allocate(Math.min(optionMap.get(RemotingOptions.BUFFER_SIZE, maxSize * 4), 4096)); }

Sending new byte[1024 * 8 - 5] results in a message with a 2 byte length field followed by 6 header bytes plus 8187 content bytes, where the length field holds 8193. On the receiver side,org.jboss.remoting3.remote.FramingChannelListener has a buffer of default size 8192. FramingChannelListener.handleEvent() wants to get 8193 bytes and it sits in a loop calling handleBufferedData(), which keeps returning because the 8192 byte buffer never holds 8193 bytes:

org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123].handleEvent(): receiveBuffer.remaining(): 0
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123].handleBufferedData()
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123] state: BODY
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123] initial == BODY, size == 8193, remaining == 8192
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123] returning because remaining < size
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123].handleEvent(): receiveBuffer.remaining(): 0
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123].handleBufferedData()
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123] state: BODY
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123] initial == BODY, size == 8193, remaining == 8192
org.jboss.remoting3.remote.FramingChannelListener@1e81a197 [[org.jboss.remoting3.remote.RemoteMessageHandler@7a19a37a], remoteConnectionHandler: 589411738, remoteConnection: 1839586123] returning because remaining < size
.
.
.

When the buffer size is changed to 4 * 16384

maxSize = optionMap.get(Options.MAX_INBOUND_MESSAGE_SIZE, 16384);

the problem goes away.

There should be two changes:

1. The default buffer sizes should be compatible.

2. The buffer size in RemoteConnectionHandler should be configurable.