Infinispan
  1. Infinispan
  2. ISPN-284

Chunked rehashing and state transfer

    Details

    • Estimated Difficulty:
      High
    • Similar Issues:
      Show 10 results 

      Description

      Currently rehashing uses RPC which uses byte array representations of state. This should be chunked based on a configurable buffer size to use multiple RPCs, to prevent large spikes in memory usage.

        Activity

        Hide
        Manik Surtani
        added a comment -

        JGroups' setState() and getState() callbacks could be used, since these use streams.

        Show
        Manik Surtani
        added a comment - JGroups' setState() and getState() callbacks could be used, since these use streams.
        Manik Surtani
        made changes -
        Field Original Value New Value
        Assignee Manik Surtani [ manik ] Vladimir Blagojevic [ vblagojevic ]
        Manik Surtani
        made changes -
        Priority Major [ 3 ] Critical [ 2 ]
        Manik Surtani
        made changes -
        Fix Version/s 5.0.0.BETA1 [ 12313467 ]
        Fix Version/s 5.0.0.Final [ 12313468 ]
        Fix Version/s 4.1.0.BETA1 [ 12313465 ]
        Fix Version/s 4.1.0.Final [ 12313466 ]
        Manik Surtani
        made changes -
        Fix Version/s 5.1.0.BETA1 [ 12313469 ]
        Fix Version/s 5.1.0.Final [ 12313470 ]
        Fix Version/s 5.0.0.BETA1 [ 12313467 ]
        Fix Version/s 5.0.0.Final [ 12313468 ]
        Pete Muir
        made changes -
        Workflow jira [ 12405326 ] GIT Pull Request workflow [ 12449833 ]
        Manik Surtani
        made changes -
        Assignee Vladimir Blagojevic [ vblagojevic ] Dan Berindei [ dan.berindei ]
        Manik Surtani
        made changes -
        Summary Streamed rehashing Chunked rehashing and state transfer
        Labels rehash state_transfer
        Priority Critical [ 2 ] Blocker [ 1 ]
        Description Currently rehashing uses RPC which uses byte array representations of state. These could use streams, perhaps using JGroups' STREAMING_STATE_TRANSFER sockets, for greater performance and robustness. Currently rehashing uses RPC which uses byte array representations of state. This should be chunked based on a configurable buffer size to use multiple RPCs, to prevent large spikes in memory usage.
        Component/s State transfer [ 12312431 ]
        Galder Zamarreño
        made changes -
        Fix Version/s 5.1.0.BETA2 [ 12318065 ]
        Fix Version/s 5.1.0.BETA1 [ 12313469 ]
        Galder Zamarreño
        made changes -
        Fix Version/s 5.1.0.BETA3 [ 12318377 ]
        Fix Version/s 5.1.0.BETA2 [ 12318065 ]
        RH Bugzilla Integration
        made changes -
        Hide
        RH Bugzilla Integration
        added a comment -

        Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220

        Added test link to an Infinispan JIRA

        Show
        RH Bugzilla Integration
        added a comment - Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220 Added test link to an Infinispan JIRA
        Hide
        RH Bugzilla Integration
        added a comment -

        Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220

        Added link to a closed Infinispan JIRA

        Show
        RH Bugzilla Integration
        added a comment - Steve Jacobs <sjacobs@redhat.com> made a comment on bug 749220 Added link to a closed Infinispan JIRA
        Galder Zamarreño
        made changes -
        Fix Version/s 5.1.0.BETA4 [ 12318437 ]
        Fix Version/s 5.1.0.BETA3 [ 12318377 ]
        Hide
        RH Bugzilla Integration
        added a comment -

        Anne-Louise Tangring <atangrin@redhat.com> changed the Status of bug 749220 from NEW to CLOSED

        Show
        RH Bugzilla Integration
        added a comment - Anne-Louise Tangring <atangrin@redhat.com> changed the Status of bug 749220 from NEW to CLOSED
        Hide
        RH Bugzilla Integration
        added a comment -

        Anne-Louise Tangring <atangrin@redhat.com> made a comment on bug 749220

        This was only opened for educational purposes.

        Show
        RH Bugzilla Integration
        added a comment - Anne-Louise Tangring <atangrin@redhat.com> made a comment on bug 749220 This was only opened for educational purposes.
        Galder Zamarreño
        made changes -
        Fix Version/s 5.1.0.BETA5 [ 12318502 ]
        Fix Version/s 5.1.0.BETA4 [ 12318437 ]
        Galder Zamarreño
        made changes -
        Fix Version/s 5.1.0.CR1 [ 12318066 ]
        Fix Version/s 5.1.0.BETA5 [ 12318502 ]
        NadirX
        made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Done [ 1 ]
        Hide
        Dan Berindei
        added a comment -

        This issue shouldn't have been closed, it was a valid issue. It's fixed partially, but the chunk size is not yet configurable.

        Show
        Dan Berindei
        added a comment - This issue shouldn't have been closed, it was a valid issue. It's fixed partially, but the chunk size is not yet configurable.
        Dan Berindei
        made changes -
        Resolution Done [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Hide
        Dan Berindei
        added a comment -

        This pull request has already been integrated, it implements state transfer chunking with a hardcoded chunk size of 10000 cache entries.

        Show
        Dan Berindei
        added a comment - This pull request has already been integrated, it implements state transfer chunking with a hardcoded chunk size of 10000 cache entries.
        Dan Berindei
        made changes -
        Status Reopened [ 4 ] Pull Request Sent [ 10011 ]
        Git Pull Request https://github.com/infinispan/infinispan/pull/698
        Manik Surtani
        made changes -
        Status Pull Request Sent [ 10011 ] Resolved [ 5 ]
        Resolution Done [ 1 ]
        Hide
        Dan Berindei
        added a comment -

        With ISPN-1475 in, the chunk size is now configurable.

        Show
        Dan Berindei
        added a comment - With ISPN-1475 in, the chunk size is now configurable.
        Mircea Markus
        made changes -
        Component/s Core API [ 12312421 ]
        Component/s Distributed Cache [ 12312422 ]

          People

          • Assignee:
            Dan Berindei
            Reporter:
            Manik Surtani
          • Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: