Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-284

Chunked rehashing and state transfer

    Details

      Description

      Currently rehashing uses RPC which uses byte array representations of state. This should be chunked based on a configurable buffer size to use multiple RPCs, to prevent large spikes in memory usage.

        Gliffy Diagrams

          Activity

          manik Manik Surtani created issue -
          Hide
          manik Manik Surtani added a comment -

          JGroups' setState() and getState() callbacks could be used, since these use streams.

          Show
          manik Manik Surtani added a comment - JGroups' setState() and getState() callbacks could be used, since these use streams.
          manik Manik Surtani made changes -
          Field Original Value New Value
          Assignee Manik Surtani [ manik ] Vladimir Blagojevic [ vblagojevic ]
          manik Manik Surtani made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          manik Manik Surtani made changes -
          Fix Version/s 5.0.0.BETA1 [ 12313467 ]
          Fix Version/s 5.0.0.Final [ 12313468 ]
          Fix Version/s 4.1.0.BETA1 [ 12313465 ]
          Fix Version/s 4.1.0.Final [ 12313466 ]
          manik Manik Surtani made changes -
          Fix Version/s 5.1.0.BETA1 [ 12313469 ]
          Fix Version/s 5.1.0.Final [ 12313470 ]
          Fix Version/s 5.0.0.BETA1 [ 12313467 ]
          Fix Version/s 5.0.0.Final [ 12313468 ]
          pmuir Pete Muir made changes -
          Workflow jira [ 12405326 ] GIT Pull Request workflow [ 12449833 ]
          manik Manik Surtani made changes -
          Assignee Vladimir Blagojevic [ vblagojevic ] Dan Berindei [ dan.berindei ]
          manik Manik Surtani made changes -
          Summary Streamed rehashing Chunked rehashing and state transfer
          Labels rehash state_transfer
          Priority Critical [ 2 ] Blocker [ 1 ]
          Description Currently rehashing uses RPC which uses byte array representations of state. These could use streams, perhaps using JGroups' STREAMING_STATE_TRANSFER sockets, for greater performance and robustness. Currently rehashing uses RPC which uses byte array representations of state. This should be chunked based on a configurable buffer size to use multiple RPCs, to prevent large spikes in memory usage.
          Component/s State transfer [ 12312431 ]
          galder.zamarreno Galder Zamarreño made changes -
          Fix Version/s 5.1.0.BETA2 [ 12318065 ]
          Fix Version/s 5.1.0.BETA1 [ 12313469 ]
          galder.zamarreno Galder Zamarreño made changes -
          Fix Version/s 5.1.0.BETA3 [ 12318377 ]
          Fix Version/s 5.1.0.BETA2 [ 12318065 ]
          jira-bugzilla-migration RH Bugzilla Integration made changes -
          Hide
          jira-bugzilla-migration RH Bugzilla Integration added a comment -

          Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220

          Added test link to an Infinispan JIRA

          Show
          jira-bugzilla-migration RH Bugzilla Integration added a comment - Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220 Added test link to an Infinispan JIRA
          Hide
          jira-bugzilla-migration RH Bugzilla Integration added a comment -

          Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220

          Added link to a closed Infinispan JIRA

          Show
          jira-bugzilla-migration RH Bugzilla Integration added a comment - Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220 Added link to a closed Infinispan JIRA
          galder.zamarreno Galder Zamarreño made changes -
          Fix Version/s 5.1.0.BETA4 [ 12318437 ]
          Fix Version/s 5.1.0.BETA3 [ 12318377 ]
          Hide
          jira-bugzilla-migration RH Bugzilla Integration added a comment -

          Anne-Louise Tangring <atangrin@redhat.com> changed the Status of bug 749220 from NEW to CLOSED

          Show
          jira-bugzilla-migration RH Bugzilla Integration added a comment - Anne-Louise Tangring <atangrin@redhat.com> changed the Status of bug 749220 from NEW to CLOSED
          Hide
          jira-bugzilla-migration RH Bugzilla Integration added a comment -

          Anne-Louise Tangring <atangrin@redhat.com> made a comment on bug 749220

          This was only opened for educational purposes.

          Show
          jira-bugzilla-migration RH Bugzilla Integration added a comment - Anne-Louise Tangring <atangrin@redhat.com> made a comment on bug 749220 This was only opened for educational purposes.
          galder.zamarreno Galder Zamarreño made changes -
          Fix Version/s 5.1.0.BETA5 [ 12318502 ]
          Fix Version/s 5.1.0.BETA4 [ 12318437 ]
          galder.zamarreno Galder Zamarreño made changes -
          Fix Version/s 5.1.0.CR1 [ 12318066 ]
          Fix Version/s 5.1.0.BETA5 [ 12318502 ]
          NadirX NadirX made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Done [ 1 ]
          Hide
          dan.berindei Dan Berindei added a comment -

          This issue shouldn't have been closed, it was a valid issue. It's fixed partially, but the chunk size is not yet configurable.

          Show
          dan.berindei Dan Berindei added a comment - This issue shouldn't have been closed, it was a valid issue. It's fixed partially, but the chunk size is not yet configurable.
          dan.berindei Dan Berindei made changes -
          Resolution Done [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          dan.berindei Dan Berindei added a comment -

          This pull request has already been integrated, it implements state transfer chunking with a hardcoded chunk size of 10000 cache entries.

          Show
          dan.berindei Dan Berindei added a comment - This pull request has already been integrated, it implements state transfer chunking with a hardcoded chunk size of 10000 cache entries.
          dan.berindei Dan Berindei made changes -
          Status Reopened [ 4 ] Pull Request Sent [ 10011 ]
          Git Pull Request https://github.com/infinispan/infinispan/pull/698
          manik Manik Surtani made changes -
          Status Pull Request Sent [ 10011 ] Resolved [ 5 ]
          Resolution Done [ 1 ]
          Hide
          dan.berindei Dan Berindei added a comment -

          With ISPN-1475 in, the chunk size is now configurable.

          Show
          dan.berindei Dan Berindei added a comment - With ISPN-1475 in, the chunk size is now configurable.
          mircea.markus Mircea Markus made changes -
          Component/s Core API [ 12312421 ]
          Component/s Distributed Cache [ 12312422 ]
          velias Vlastimil Eliáš made changes -
          Workflow GIT Pull Request workflow [ 12449833 ] GIT Pull Request with Triage workflow [ 12613585 ]

            People

            • Assignee:
              dan.berindei Dan Berindei
              Reporter:
              manik Manik Surtani
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development