Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-2072

UFC and MFC headers get mixed up

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 3.6.10, 4.0
    • 3.6.9
    • None

    Description

      11:24:19,178 TRACE (OOB-10,NodeD-30623:[]) [MFC] NodeA-6644 used 4969 credits, 396763 remaining
11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [MFC] sending 603237 credits to NodeA-6644
11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [UDP] NodeD-30623: sending msg to NodeA-6644, src=NodeD-30623, headers are MFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default]
11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UDP] NodeA-6644: received [dst: NodeA-6644, src: NodeD-30623 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are UFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default]
11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UFC] received 603237 credits from NodeD-30623, old credits: 938772, new credits: 1000000

      15:20:14,596 TRACE (OOB-44,NodeC-64981:[]) [UFC] NodeA-12977 used 2305 credits, 798175 remaining
15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UFC] sending 1201825 credits to NodeA-12977
15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UDP] NodeC-64981: sending msg to NodeA-12977, src=NodeC-64981, headers are UFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default]
15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UDP] NodeA-12977: received [dst: NodeA-12977, src: NodeC-64981 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are MFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default]
15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UFC] NodeC-64981 used 8 credits, 1859662 remaining
15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [MFC] received 1201825 credits from NodeC-64981, new credits for NodeC-64981: 2000000, min_credits=793978

      UFC/MFC expect their messages to be reliable, so they don't repeat REPLENISH messages, and the only way to replenish the credits is to send a CREDIT_REQUEST message. But UFC/MFC only send a credit request after all the threads trying to send a request or response to the affected destination block for max_block_time.

      I was able to reproduce the problem with an Infinispan micro-benchmark: a cluster of 4 nodes, a replicated cache, 400 threads writing to the cache. It happens both with JGroups 3.6.9.Final and the latest master.

      Attachments

        Issue Links

          is related to

          Bug - A problem that impairs or prevents the functions of the product. JDG-392 Write performance regression in replicated non-transactional mode

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          relates to

          Task - Represents a small unit of work that is not end-user facing. ISPN-6494 Investigate bundler performance

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              rhn-engineering-bban Bela Ban
              dberinde@redhat.com Dan Berindei (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: