Infinispan
  1. Infinispan
  2. ISPN-1890

Unable to create new native thread when running core/ testsuite

    Details

    • Similar Issues:
      Show 10 results 

      Description

      Suddenly, running core/ testsuite only with -Xmx768m -XX:MaxPermSize=256M, I'm seeing:

      Test suite progress: tests succeeded: 1695, failed: 4, skipped: 0.
      Exception in thread "ConnectionMap.Acceptor,null,null" java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:658)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.start(TCPConnectionMap.java:568)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection.start(TCPConnectionMap.java:411)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$600(TCPConnectionMap.java:354)
      	at org.jgroups.blocks.TCPConnectionMap$ConnectionAcceptor.run(TCPConnectionMap.java:259)
      	at java.lang.Thread.run(Thread.java:680)
      Exception in thread "ConnectionMap.Acceptor,null,null" java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:658)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.start(TCPConnectionMap.java:568)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection.start(TCPConnectionMap.java:411)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$600(TCPConnectionMap.java:354)
      	at org.jgroups.blocks.TCPConnectionMap$ConnectionAcceptor.run(TCPConnectionMap.java:259)
      	at java.lang.Thread.run(Thread.java:680)
      Exception in thread "ConnectionMap.Acceptor,null,null" java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:658)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.start(TCPConnectionMap.java:568)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection.start(TCPConnectionMap.java:411)
      	at org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$600(TCPConnectionMap.java:354)
      	at org.jgroups.blocks.TCPConnectionMap$ConnectionAcceptor.run(TCPConnectionMap.java:259)
      	at java.lang.Thread.run(Thread.java:680)
      [testng-TopologyAwareDistAsyncFuncTest] Test testPutIfAbsentFromNonOwner(org.infinispan.distribution.topologyaware.TopologyAwareDistAsyncFuncTest) succeeded.
      Test suite progress: tests succeeded: 1696, failed: 4, skipped: 0.
      [testng-TopologyAwareDistAsyncFuncTest] Test testRemoveFromNonOwner(org.infinispan.distribution.topologyaware.TopologyAwareDistAsyncFuncTest) succeeded.
      Test suite progress: tests succeeded: 1697, failed: 4, skipped: 0.
      [testng-TopologyAwareDistAsyncFuncTest] Test testReplaceFromNonOwner(org.infinispan.distribution.topologyaware.TopologyAwareDistAsyncFuncTest) succeeded.
      Test suite progress: tests succeeded: 1698, failed: 4, skipped: 0.
      Exception in thread "CacheViewTrigger,TopologyAwareDistAsyncFuncTest-NodeA-38647" java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:658)
      	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
      	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
      	at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewTriggerThread.run(CacheViewsManagerImpl.java:848)

      See attached thread dump.

      1. output.log
        2.44 MB
        Galder Zamarreño

        Activity

        Hide
        Dan Berindei
        added a comment -

        Sanne, I should have mentioned that it's the same thing. But I talked to Scott and he actually didn't have it set, so I guess it's always good to check with ulimit rather than trust that the limits.conf settings have been applied correctly.

        I just wanted to make it clear, in case someone else gets here wondering about this error, that it's usually about the number of processes (on Linux) or the native memory (on 32-bit Windows, since threads will have a bigger stack by default) - and never about the heap settings.

        Show
        Dan Berindei
        added a comment - Sanne, I should have mentioned that it's the same thing. But I talked to Scott and he actually didn't have it set, so I guess it's always good to check with ulimit rather than trust that the limits.conf settings have been applied correctly. I just wanted to make it clear, in case someone else gets here wondering about this error, that it's usually about the number of processes (on Linux) or the native memory (on 32-bit Windows, since threads will have a bigger stack by default) - and never about the heap settings.
        Hide
        Sanne Grinovero
        added a comment -

        I'm having around 3000 threads in my testsuite, most of them are the KeyAffinityService as Dan also pointed out.
        It's the org.infinispan.affinity.KeyAffinityServiceImpl$KeyGeneratorWorker waiting.

        Show
        Sanne Grinovero
        added a comment - I'm having around 3000 threads in my testsuite, most of them are the KeyAffinityService as Dan also pointed out. It's the org.infinispan.affinity.KeyAffinityServiceImpl$KeyGeneratorWorker waiting.
        Hide
        Sanne Grinovero
        added a comment -

        BTW the threadpool used by the KeyGeneratorWorker is not having a nice name assigned, it would be nice to have it's threads properly named too.

        Show
        Sanne Grinovero
        added a comment - BTW the threadpool used by the KeyGeneratorWorker is not having a nice name assigned, it would be nice to have it's threads properly named too.
        Hide
        Mircea Markus
        added a comment -

        The test suite is down to aprox 1.1k threads from 2k+ threads now. Also thread leaks were removed and a log to dump the current system threads was added in case an OOM happens.

        Show
        Mircea Markus
        added a comment - The test suite is down to aprox 1.1k threads from 2k+ threads now. Also thread leaks were removed and a log to dump the current system threads was added in case an OOM happens.
        Hide
        Radoslav Husar
        added a comment -

        BTW I have just updated the doc with info on the security limits: https://docs.jboss.org/author/display/ISPN/Contributing+-+The+test+suite

        Show
        Radoslav Husar
        added a comment - BTW I have just updated the doc with info on the security limits: https://docs.jboss.org/author/display/ISPN/Contributing+-+The+test+suite

          People

          • Assignee:
            Mircea Markus
            Reporter:
            Galder Zamarreño
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: