SnF queue limitation in multi-tenant context

In multi-tenant environment when messages for an address X waiting in the internal queue of a broker B2 for redistribution, for example because the address is full in the broker B1, all messages that pass through this internal queue, regardless of their address, also remain on hold.

All these messages are waiting in this queue even their respective address in broker B1 is not full.

SnF behavior:

• If a Store aNd Forward (SNF) queue has pending message to be delivered to broker1 from broker2, these messages will block any subsequent message that get added to the SnF for distribution

Impact:

• The blocked redistribution of a particular address seem to directly affects the redistribution of all addresses of the cluster.

Customer environment:

• Multi-tenant cluster
• One tenant per environment is not a feasible solution
• Queue federation instead of clustering is not feasible solution for customer since many things depends on the actual architecture that we have spend months to refine

Expected behavior:

• Customer understand that the SnF acts as designed (First in First out), and only the consumption of piled up message ahead will free the last blocked message. So changing the way SnF works is not what they expect.
  However, Customer would like Eng to figure out a solution (example message priority inside SnF ... ) to overcome this bottleneck in multi-tenant environment.

Note:
This is a blocker for the customers to move forward and being able to accept clients.