No such channel exception on invoke timeout

Intermittently an exception "No such channel; id=29" is thrown after an INVOKE activity times out. On further investigation there seems to be a timing issue between the INVOKE_CHECK activity and the JBoss WS CXF request socket timeout.

It looks like for each INVOKE activity the CXF read timeout is automatically set to the mextimeout and an INVOKE_CHECK activity is automatically scheduled for execution in the ODE_JOB table for approximately the same time. Looks like the "No such channel" exception is logged only if the CXF ead timeout fires after the INVOKE_CHECK has been processed.

The error message is:

16:39:04,885 ERROR [org.apache.ode.scheduler.simple.SimpleScheduler] Error while processing a persisted job: [JobId: hqejbhcnphr6tfun8xreke,nodeId: 10.40.42.88:1099,scheduled: false,transacted: true,ts: 1323428941838,channel: 29,instaceId : 107,type: INVOKE_RESPONSE,retrycount: 3]
org.apache.ode.bpel.iapi.Scheduler$JobProcessorException: java.lang.RuntimeException: java.lang.IllegalArgumentException: No such channel; id=29
at org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.java:480)
at org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.java:455)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJobCallable$1.call(SimpleScheduler.java:576)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJobCallable$1.call(SimpleScheduler.java:566)
at org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:294)
at org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:251)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJobCallable.processInTransactionContext(SimpleScheduler.java:566)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJobCallable.call(SimpleScheduler.java:546)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJobCallable.call(SimpleScheduler.java:533)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: No such channel; id=29
at org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:464)
at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139)
at org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(BpelRuntimeContextImpl.java:967)
at org.apache.ode.bpel.engine.BpelProcess.handleJobDetails(BpelProcess.java:483)
at org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.java:462)
... 13 more
Caused by: java.lang.IllegalArgumentException: No such channel; id=29
at org.apache.ode.jacob.vpu.ExecutionQueueImpl.findChannelFrame(ExecutionQueueImpl.java:205)
at org.apache.ode.jacob.vpu.ExecutionQueueImpl.consumeExport(ExecutionQueueImpl.java:232)
at org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.importChannel(JacobVPU.java:369)
at org.apache.ode.jacob.JacobObject.importChannel(JacobObject.java:47)
at org.apache.ode.bpel.engine.BpelRuntimeContextImpl$7.run(BpelRuntimeContextImpl.java:1093)
at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451)

Apart from this error message there are some INVOKE_RESPONSE activites that are added to the ODE_JOB table that are never removed. I'm not sure whether that will have any impact to the further execution of that process instance, but whenver these errors are logged the BPEL console shows the corresponding process instance in the RUNNING state even though it should have failed.

If the CXF timeout happens before the INVOKE_CHECK is processed then there are no error messages logged and the process instance goes to the FAILED state. In either case the INVOKE action is retried as expected.