Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-871

Out-of-heap data container implementation

    Details

    • Estimated Difficulty:
      Medium

      Description

      The DataContainer interface could be implemented using a off-heap impl in C, using a wrapper around TBB's concurrent hashmap.

      Cheap and easy way, no memory management needed, at worst case same performance as the Java CHM-like impl of the data container + some JNI overhead. Potential benefit of large data heaps.

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            cotton.ben Ben Cotton added a comment -

            Hi,

            We want to get started w a POC build of doing this using Peter Lawreys HugeHashMap solution (details of the POC build ambition = http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infinispan-embedded-off-heap-cache-td4026102.html). Any success (of course) to be immediately contributed back for ISPN's consideration to use.

            ROAD-BLOCK: Can't build! How do you build ISPN 7 from GIT? Tryng this:

            Assuming all I need is JDK 1.7 anf Maven 3

            1. Fork https://github.com/infinispan/infinispan/ to my GIT repo
            2. Clone from my GIT repo's ISPN origin to my desktop workspace
            3. cd to the infinispan root pom.xml directory
            4. Build from my workspace = `mvn clean compile`

            #this is the ROADBLOCK point - how do we resolve the below unresolved ' infinispan-extended-statistics'?

            See ben@52-b.liq,risk.ny.jpm.com> mvn clean compile

            <snip ....>

            [INFO] ------------------------------------------------------------------------
            [INFO] BUILD FAILURE
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 48.156s
            [INFO] Finished at: Thu Feb 06 00:14:16 EST 2014
            [INFO] Final Memory: 50M/361M
            [INFO] ------------------------------------------------------------------------
            [ERROR] Failed to execute goal on project infinispan-extended-statistics: Could not resolve dependencies for project org.infinispan:infinispan-extended-statistics:jar:7.0.0-SNAPSHOT: Failure to find org.infinispan:infinispan-core:jar:tests:7.0.0-SNAPSHOT in https://repository.jboss.org/nexus/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of jboss-public-repository has elapsed or updates are forced -> [Help 1]
            [ERROR]
            [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
            [ERROR] Re-run Maven using the -X switch to enable full debug logging.
            [ERROR]
            [ERROR] For more information about the errors and possible solutions, please read the following articles:
            [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
            [ERROR]
            [ERROR] After correcting the problems, you can resume the build with the command
            [ERROR] mvn <goals> -rf :infinispan-extended-statistics
            ben@52-b.liq,risk.ny.jpm.com>

            Show
            cotton.ben Ben Cotton added a comment - Hi, We want to get started w a POC build of doing this using Peter Lawreys HugeHashMap solution (details of the POC build ambition = http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infinispan-embedded-off-heap-cache-td4026102.html ). Any success (of course) to be immediately contributed back for ISPN's consideration to use. ROAD-BLOCK: Can't build! How do you build ISPN 7 from GIT? Tryng this: Assuming all I need is JDK 1.7 anf Maven 3 1. Fork https://github.com/infinispan/infinispan/ to my GIT repo 2. Clone from my GIT repo's ISPN origin to my desktop workspace 3. cd to the infinispan root pom.xml directory 4. Build from my workspace = `mvn clean compile` #this is the ROADBLOCK point - how do we resolve the below unresolved ' infinispan-extended-statistics'? See ben@52-b.liq,risk.ny.jpm.com> mvn clean compile <snip ....> [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 48.156s [INFO] Finished at: Thu Feb 06 00:14:16 EST 2014 [INFO] Final Memory: 50M/361M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project infinispan-extended-statistics: Could not resolve dependencies for project org.infinispan:infinispan-extended-statistics:jar:7.0.0-SNAPSHOT: Failure to find org.infinispan:infinispan-core:jar:tests:7.0.0-SNAPSHOT in https://repository.jboss.org/nexus/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of jboss-public-repository has elapsed or updates are forced -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :infinispan-extended-statistics ben@52-b.liq,risk.ny.jpm.com>
            Hide
            cotton.ben Ben Cotton added a comment - - edited

            All build issues are now resolved (we were trying to build within a bank's firewall, which is perfectly futile).

            It is our understandng that there will a OpenHFT HC Alpha Release in Maven Central next weekend. At that time, Dmitry and I will take the OpenHFT dependency tag and proceed to build a branch of Red Hat's ISPN 7 that will treat net.openhft.collections.SharedHashMap as a Red Hat Infinispan 7 default impl of javax.cache.Cache

            A diagram of this build effort can be found here: https://raw.github.com/Cotton-Ben/OpenHFT/master/doc/AdaptingOpenHFT-SHM-as-JCACHE-Impl.jpg

            Also the OpenHFT view of this effort will be tracked here: https://github.com/OpenHFT/HugeCollections/issues/13

            Show
            cotton.ben Ben Cotton added a comment - - edited All build issues are now resolved (we were trying to build within a bank's firewall, which is perfectly futile). It is our understandng that there will a OpenHFT HC Alpha Release in Maven Central next weekend. At that time, Dmitry and I will take the OpenHFT dependency tag and proceed to build a branch of Red Hat's ISPN 7 that will treat net.openhft.collections.SharedHashMap as a Red Hat Infinispan 7 default impl of javax.cache.Cache A diagram of this build effort can be found here: https://raw.github.com/Cotton-Ben/OpenHFT/master/doc/AdaptingOpenHFT-SHM-as-JCACHE-Impl.jpg Also the OpenHFT view of this effort will be tracked here: https://github.com/OpenHFT/HugeCollections/issues/13
            Hide
            cotton.ben Ben Cotton added a comment -

            The build will take place at my Fork of ISPN 7. https://github.com/Cotton-Ben/infinispan

            Show
            cotton.ben Ben Cotton added a comment - The build will take place at my Fork of ISPN 7. https://github.com/Cotton-Ben/infinispan
            Hide
            rajeevrana_77 Rajeev Rana added a comment -

            Any plan for this one on the roadmap please ?

            Show
            rajeevrana_77 Rajeev Rana added a comment - Any plan for this one on the roadmap please ?
            Hide
            cotton.ben Ben Cotton added a comment -


            Hi, Please be advised that on Sep 18/19 all of Ben Cotton, Peter Lawrey and Dmitry Gordeev will be meeting in NYC to specifically discuss the plan to progress development on https://github.com/Cotton-Ben/infinispan ... we will then pass our findings onto Mircea Markus and Adrian Nistor re: next best steps. Please forgive our lack of progress on this JIRA ... we are not RedHat employees and are 100% donating our time as commuinity contributors. That said we are exceedingly excited wmance Opent to adopting the OpenHFT high-performance off-heap capability as an ISPN Cache. Thanks.

            Show
            cotton.ben Ben Cotton added a comment - Hi, Please be advised that on Sep 18/19 all of Ben Cotton, Peter Lawrey and Dmitry Gordeev will be meeting in NYC to specifically discuss the plan to progress development on https://github.com/Cotton-Ben/infinispan ... we will then pass our findings onto Mircea Markus and Adrian Nistor re: next best steps. Please forgive our lack of progress on this JIRA ... we are not RedHat employees and are 100% donating our time as commuinity contributors. That said we are exceedingly excited wmance Opent to adopting the OpenHFT high-performance off-heap capability as an ISPN Cache. Thanks.

              People

              • Assignee:
                NadirX Tristan Tarrant
                Reporter:
                manik Manik Surtani
              • Votes:
                13 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Development