DefaultExecutorService failoverExecution does not rethrow the root cause of the failure

Hi,

in case if Exception is thrown in Callable/DistributedCallable during execution with DistributedExecutor, it often happens that the real root cause of the exception is not shown - rethrown, but the following exception appears (see below).

In this case, the the code maintenance becomes harder, as it is not visible what caused the issue.

The pull request for the reproduction is:
https://github.com/infinispan/infinispan/pull/1398

I'm just thinking, maybe this issue relates to ISPN-1636 - where it is said that "injection should happen again on each node, avoiding serialization of the injected components", but in case of failover - serialization is performed. Just thoughts.

java.util.concurrent.ExecutionException: Failover execution failed
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:839)
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:801)
at org.infinispan.distexec.DistributedExecutorTest.basicInvocation(DistributedExecutorTest.java:86)
at org.infinispan.cdi.test.distexec.DistributedExecutorCDITest.testInvocationException(DistributedExecutorCDITest.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74)
at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:176)
at org.jboss.arquillian.testng.Arquillian$1.invoke(Arquillian.java:111)
at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114)
at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67)
at org.jboss.arquillian.container.test.impl.client.protocol.local.LocalContainerMethodExecutor.invoke(LocalContainerMethodExecutor.java:50)
at org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114)
at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67)
at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:130)
at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134)
at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111)
at org.jboss.arquillian.testng.Arquillian.run(Arquillian.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:189)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:666)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
at org.testng.TestRunner.privateRun(TestRunner.java:749)
at org.testng.TestRunner.run(TestRunner.java:600)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)
at org.testng.SuiteRunner.run(SuiteRunner.java:223)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1039)
at org.testng.TestNG.runSuitesLocally(TestNG.java:964)
at org.testng.TestNG.run(TestNG.java:900)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:77)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:110)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:113)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.Exception: Failover execution failed
... 108 more
Caused by: java.util.concurrent.ExecutionException: Failover execution failed
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:839)
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:801)
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:834)
... 107 more
Caused by: java.lang.Exception: Failover execution failed
... 110 more
Caused by: java.util.concurrent.ExecutionException: Failover execution failed
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:836)
... 109 more
Caused by: java.lang.Exception: Failover execution failed
... 110 more
Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.get(DefaultExecutorService.java:798)
... 108 more
Caused by: org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:532)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:184)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:514)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:176)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:198)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:255)
at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:81)
at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:289)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:278)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:296)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:177)
... 11 more
Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.CacheImpl
Caused by: an exception which occurred:
in field cache
in object org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8
-> toString = org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8
in object org.infinispan.commands.read.DistributedExecuteCommand@1f
-> toString = DistributedExecuteCommand

{cache=Cache 'DistributedExecutorTest-DIST_SYNC'@DistributedExecutorCDITest-NodeA-37398, keys=[], callable=org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8}
in object org.infinispan.commands.remote.SingleRpcCommand@e8fbe1d0
-> toString = SingleRpcCommand{cacheName='DistributedExecutorTest-DIST_SYNC', command=DistributedExecuteCommand{cache=Cache 'DistributedExecutorTest-DIST_SYNC'@DistributedExecutorCDITest-NodeA-37398, keys=[], callable=org.infinispan.cdi.test.distexec.DistributedExecutorCDITest$DistributedCacheCallable@4c824d8}

}