Major Description
Because of the high level of concurrency in AS7, I'm seeing occasional socket exceptions in FD_SOCK when trying to start multiple channels using a shared transport. It appears that there are some thread safety issues with the port increment logic within FD_SOCK.
Until JGRP-790 is complete, measures need to be taken to ensure that multiple concurrently connected channels do not select the same port on which to open a server socket.
Here's a sample stack trace:
10:01:26,716 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC00001: Failed to start service jboss.infinispan.cluster: org.jboss.msc.service.StartException in service jboss.infinispan.cluster: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1696)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [:1.6.0_22]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [:1.6.0_22]
at java.lang.Thread.run(Thread.java:679) [:1.6.0_22]
Caused by: org.infinispan.CacheException: Unable to invoke method public abstract void org.infinispan.remoting.transport.Transport.start() on object
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:174)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:687)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:589)
at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:137)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:137)
at org.infinispan.CacheDelegate.start(CacheDelegate.java:323)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:439)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:77)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:66)
at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.start(EmbeddedCacheManagerService.java:171)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1675)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_22]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [:1.6.0_22]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [:1.6.0_22]
at java.lang.reflect.Method.invoke(Method.java:616) [:1.6.0_22]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
... 15 more
Caused by: java.lang.IllegalArgumentException: failed to start server socket
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:295)
at org.jgroups.protocols.FD.down(FD.java:310)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:70)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:90)
at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:634)
at org.jgroups.protocols.UNICAST.down(UNICAST.java:443)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:330)
at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:186)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:926)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)
at org.jgroups.protocols.FRAG2.down(FRAG2.java:155)
at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.down(STREAMING_STATE_TRANSFER.java:324)
at org.jgroups.protocols.pbcast.FLUSH.handleConnect(FLUSH.java:302)
at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:263)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1081)
at org.jgroups.JChannel.downcall(JChannel.java:1680)
at org.jgroups.JChannel.connect(JChannel.java:425)
at org.jgroups.JChannel.connect(JChannel.java:388)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:171)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:155)
... 20 more
Caused by: java.net.SocketException: Address already in use
at java.net.PlainSocketImpl.socketListen(Native Method) [:1.6.0_22]
at java.net.AbstractPlainSocketImpl.listen(AbstractPlainSocketImpl.java:365) [:1.6.0_22]
at java.net.ServerSocket.bind(ServerSocket.java:337) [:1.6.0_22]
at java.net.ServerSocket.<init>(ServerSocket.java:202) [:1.6.0_22]
at org.jgroups.util.DefaultSocketFactory.createServerSocket(DefaultSocketFactory.java:49)
at org.jgroups.util.Util.createServerSocket(Util.java:3046)
at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:561)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:292)
... 40 more
