Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-2072

UFC and MFC headers get mixed up

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.6.9
    • Fix Version/s: 3.6.10, 4.0
    • Labels:
      None

      Description

      11:24:19,178 TRACE (OOB-10,NodeD-30623:[]) [MFC] NodeA-6644 used 4969 credits, 396763 remaining
      11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [MFC] sending 603237 credits to NodeA-6644
      11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [UDP] NodeD-30623: sending msg to NodeA-6644, src=NodeD-30623, headers are MFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default]
      11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UDP] NodeA-6644: received [dst: NodeA-6644, src: NodeD-30623 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are UFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default]
      11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UFC] received 603237 credits from NodeD-30623, old credits: 938772, new credits: 1000000
      

      15:20:14,596 TRACE (OOB-44,NodeC-64981:[]) [UFC] NodeA-12977 used 2305 credits, 798175 remaining
      15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UFC] sending 1201825 credits to NodeA-12977
      15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UDP] NodeC-64981: sending msg to NodeA-12977, src=NodeC-64981, headers are UFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default]
      15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UDP] NodeA-12977: received [dst: NodeA-12977, src: NodeC-64981 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are MFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default]
      15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UFC] NodeC-64981 used 8 credits, 1859662 remaining
      15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [MFC] received 1201825 credits from NodeC-64981, new credits for NodeC-64981: 2000000, min_credits=793978
      

      UFC/MFC expect their messages to be reliable, so they don't repeat REPLENISH messages, and the only way to replenish the credits is to send a CREDIT_REQUEST message. But UFC/MFC only send a credit request after all the threads trying to send a request or response to the affected destination block for max_block_time.

      I was able to reproduce the problem with an Infinispan micro-benchmark: a cluster of 4 nodes, a replicated cache, 400 threads writing to the cache. It happens both with JGroups 3.6.9.Final and the latest master.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  belaban Bela Ban
                  Reporter:
                  dan.berindei Dan Berindei
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: