-
Bug
-
Resolution: Done
-
Major
-
3.0.3.Final
-
None
One of our integration tests (arquillian based running in WildFly 12) intermittently throws an NPE at startup. We've been tracking it down to an NPE in BeanIdentifierIndex.getIndex(BeanIdentifier identifier), see stacktrace below.
One of our CDI beans is initialised and in its @PostConstruct-method calls a method on an injected EJB, which then through SerializableContextualHolder.<init>().
It seems the field reverseIndex of BeanIdentifierIndex is `null` when `getIndex(...)` gets called but BeanIdentifierIndex.isBuilt() returns true. This is possible because isBuilt() returns whether or not the field `index` is filled (ignoring `reverseIndex`) but during the `build()` method `index` gets set before `reverseIndex` gets built. See BeanIdentifierIndex.build(...), line 127-133.
I think `isBuilt()` should only return true once the `build(...)` method has finished completely.
We're working on adding enough DEBUG logging to confirm this method-call/NPE happens in between WeldStartup#495 and BeanIdentifierIndex#138.
2018-05-05 12:09:58,485 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 70) WFLYEJB0034: EJB Invocation failed on component AccountDirectoryService for method public void nl.topicus.keyhub.rest.directory.AccountDirectoryService.initDirectories(): javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:186) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:330) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at nl.topicus.keyhub.rest.directory.AccountDirectoryService$$$view66.initDirectories(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:411) at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) at nl.topicus.keyhub.rest.directory.AccountDirectoryService$Proxy$_$$_Weld$EnterpriseProxy$.initDirectories(Unknown Source) at nl.topicus.keyhub.rest.security.oauth2.KeyHubRestOAuth2Realm.postConstruct(KeyHubRestOAuth2Realm.java:32) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:83) at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:66) at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122) at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:172) at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:70) at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:689) at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) at org.jboss.weld.bean.builtin.InstanceImpl.access$400(InstanceImpl.java:73) at org.jboss.weld.bean.builtin.InstanceImpl$InstanceImplIterator.next(InstanceImpl.java:325) at com.google.common.collect.Iterators$ConcatenatedIterator.next(Iterators.java:1371) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:437) at com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:849) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:277) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:234) at nl.topicus.cobra.rest.security.CdiWebEnvironment.init(CdiWebEnvironment.java:48) at nl.topicus.cobra.rest.security.CdiWebEnvironment$Proxy$_$$_WeldClientProxy.init(Unknown Source) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:45) at nl.topicus.cobra.rest.security.CdiEnvironmentLoaderListener.createEnvironment(CdiEnvironmentLoaderListener.java:22) at org.apache.shiro.web.env.EnvironmentLoader.initEnvironment(EnvironmentLoader.java:139) at org.apache.shiro.web.env.EnvironmentLoaderListener.contextInitialized(EnvironmentLoaderListener.java:58) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:215) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:184) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:249) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) at org.jboss.threads.JBossThread.run(JBossThread.java:485) Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163) at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134) at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88) at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:127) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:48) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273) ... 96 more Caused by: javax.ejb.EJBException: java.lang.NullPointerException at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:186) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275) at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:347) at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:74) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) ... 102 more Caused by: java.lang.NullPointerException at org.jboss.weld.serialization.BeanIdentifierIndex.getIndex(BeanIdentifierIndex.java:64) at org.jboss.weld.contexts.SerializableContextualFactory$SerializableContextualHolder.<init>(SerializableContextualFactory.java:99) at org.jboss.weld.contexts.SerializableContextualFactory$AbstractSerializableBean.<init>(SerializableContextualFactory.java:160) at org.jboss.weld.contexts.SerializableContextualFactory$PassivationCapableSerializableBean.<init>(SerializableContextualFactory.java:267) at org.jboss.weld.contexts.SerializableContextualFactory.create(SerializableContextualFactory.java:53) at org.jboss.weld.serialization.ContextualStoreImpl.getSerializableContextual(ContextualStoreImpl.java:140) at org.jboss.weld.contexts.SerializableContextualInstanceImpl.<init>(SerializableContextualInstanceImpl.java:35) at org.jboss.weld.contexts.unbound.DependentContextImpl.addDependentInstance(DependentContextImpl.java:112) at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:72) at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:689) at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:789) at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:335) at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:346) at org.jboss.weld.injection.producer.DefaultInjector$1.proceed(DefaultInjector.java:71) at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) at org.jboss.weld.injection.producer.DefaultInjector.inject(DefaultInjector.java:73) at org.jboss.weld.module.ejb.DynamicInjectionPointInjector.inject(DynamicInjectionPointInjector.java:61) at org.jboss.weld.module.ejb.SessionBeanInjectionTarget.inject(SessionBeanInjectionTarget.java:138) at org.jboss.as.weld.injection.WeldInjectionContext.inject(WeldInjectionContext.java:39) at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:51) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.interceptors.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:105) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273) ... 115 more
- is related to
-
WELD-2532 Backport WELD-2492 to 2.4 branch
-
- Resolved
-
[WELD-2492] NPE in BeanIdentifierIndex.getIndex due to race condition
manovotn Thanks for the fix and for the long wait. Last week we configured our test wildfly docker image to use a custom built (from a version of master including the PR) weld-core-impl 3.0.5 and since then we have seen no more failures due to this NPE. As far as we can tell, the issue is fixed now. 
Sven Haster (Inactive)
added a comment - - edited manovotn Thanks for the fix and for the long wait. Last week we configured our test wildfly docker image to use a custom built (from a version of master including the PR) weld-core-impl 3.0.5 and since then we have seen no more failures due to this NPE. As far as we can tell, the issue is fixed now. sven.haster thanks for more logging info. The PR still looks like it could help.
As for upgrading...
The proper way would be to update WFLY to Weld 3.0.4.Final which was released in the meantime (using the patch and the how-to described in the end of the release article) and then upgrade core to 3.0.5 snapshot.
If you just bump core from 3.0.3.Final to 3.0.5-SNAPSHOT, I think it won't work (there was an API update so core will expect to use a version of API which won't exist).
EDIT: Actually, I just ran some of our tests with WFLY 12 with just core swapped to PR snapshot and they passed. But that might be just me not hitting the "right" parts of code. I would still recommend going through update to 3.0.4.Final.
Matěj Novotný
added a comment - - edited sven.haster thanks for more logging info. The PR still looks like it could help.
As for upgrading...
The proper way would be to update WFLY to Weld 3.0.4.Final which was released in the meantime (using the patch and the how-to described in the end of the release article ) and then upgrade core to 3.0.5 snapshot.
If you just bump core from 3.0.3.Final to 3.0.5-SNAPSHOT, I think it won't work (there was an API update so core will expect to use a version of API which won't exist).
EDIT: Actually, I just ran some of our tests with WFLY 12 with just core swapped to PR snapshot and they passed. But that might be just me not hitting the "right" parts of code. I would still recommend going through update to 3.0.4.Final. manovotn thanks for the PR. It seems like the fix for our problem. We could test this for you but since we test using a docker container wildfly it's a bit more complicated.
We run Weld 3.0.3 right now (Wildfly 12), can we patch just weld-core-impl to 3.0.5-SNAPSHOT (from the PR) or do we need to replace all jars? It will be easier/more doable if we only need to update one jar in wildfly modules.
Sven Haster (Inactive)
added a comment - - edited manovotn thanks for the PR. It seems like the fix for our problem. We could test this for you but since we test using a docker container wildfly it's a bit more complicated.
We run Weld 3.0.3 right now (Wildfly 12), can we patch just weld-core-impl to 3.0.5-SNAPSHOT (from the PR) or do we need to replace all jars? It will be easier/more doable if we only need to update one jar in wildfly modules. Sven Haster (Inactive)
added a comment - Hi guys,
I have retrieved some extra logging from the Bootstrap and BootstrapTracker, see below. We noticed we never got the `beanIdentifierIndexBuilt`-logging and after looking into it we suspect the identifier index gets built for an empty list of beans because WeldStartup only lists SessionScoped and ConversationScoped beans (see `WeldStartup#getBeansForBeanIdentifierIndex`) so the race condition happens in the
if (beans.isEmpty()) { index = new BeanIdentifier[0]; reverseIndex = Collections.emptyMap(); indexHash = 0; return; }
between the setting of the index and the setting of the reverseIndex 
Relevant part of the logging:
2018-05-14 07:46:38,400 INFO [nl.topicus.cobra.liquibase.UpdateNotifier] (ServerService Thread Pool -- 67) Update finished 2018-05-14 07:46:38,422 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-3) START bootstrap > endInitialization 2018-05-14 07:46:38,451 DEBUG [org.jboss.weld.Bootstrap] (MSC service thread 1-3) EE modules: [BeanDeploymentModule [id=deployment.keyhub-ear.ear.keyhub-dao.jar, web=false, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@4111074], BeanDeploymentModule [id=deployment.keyhub-ear.ear, web=false, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@1680a122], BeanDeploymentModule [id=deployment.keyhub-ear.ear.keyhub-rest.war, web=true, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@7e56c4c3], BeanDeploymentModule [id=BeanDeploymentModules.DEFAULT, web=false, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@4f99027]] 2018-05-14 07:46:38,454 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-3) END bootstrap > endInitialization (32 ms) 2018-05-14 07:46:38,454 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-3) END bootstrap (24091 ms) 2018-05-14 07:46:38,457 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 65) WFLYEJB0034: EJB Invocation failed on component AccountDirectoryService for method public void nl.topicus.keyhub.rest.directory.AccountDirectoryService.initDirectories(): javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
That UpdateNotifier is part of an @Startup bean. The exception at the end is where the NPE happens. Nowhere in our log do we get the "Bean identifier index built" message, which is consistent with WeldStartup passing an empty list to `BeanIdentifierIndex#build(...)`.
Sven Haster (Inactive)
added a comment - Hi guys,
I have retrieved some extra logging from the Bootstrap and BootstrapTracker, see below. We noticed we never got the `beanIdentifierIndexBuilt`-logging and after looking into it we suspect the identifier index gets built for an empty list of beans because WeldStartup only lists SessionScoped and ConversationScoped beans (see `WeldStartup#getBeansForBeanIdentifierIndex`) so the race condition happens in the
if (beans.isEmpty()) {
index = new BeanIdentifier[0];
reverseIndex = Collections.emptyMap();
indexHash = 0;
return ;
}
between the setting of the index and the setting of the reverseIndex
Relevant part of the logging:
2018-05-14 07:46:38,400 INFO [nl.topicus.cobra.liquibase.UpdateNotifier] (ServerService Thread Pool -- 67) Update finished
2018-05-14 07:46:38,422 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-3) START bootstrap > endInitialization
2018-05-14 07:46:38,451 DEBUG [org.jboss.weld.Bootstrap] (MSC service thread 1-3) EE modules: [BeanDeploymentModule [id=deployment.keyhub-ear.ear.keyhub-dao.jar, web=false, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@4111074], BeanDeploymentModule [id=deployment.keyhub-ear.ear, web=false, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@1680a122], BeanDeploymentModule [id=deployment.keyhub-ear.ear.keyhub-rest.war, web=true, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@7e56c4c3], BeanDeploymentModule [id=BeanDeploymentModules.DEFAULT, web=false, managers=org.jboss.weld.util.collections.Iterables$$Lambda$837/2129654567@4f99027]]
2018-05-14 07:46:38,454 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-3) END bootstrap > endInitialization (32 ms)
2018-05-14 07:46:38,454 DEBUG [org.jboss.weld.BootstrapTracker] (MSC service thread 1-3) END bootstrap (24091 ms)
2018-05-14 07:46:38,457 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 65) WFLYEJB0034: EJB Invocation failed on component AccountDirectoryService for method public void nl.topicus.keyhub.rest.directory.AccountDirectoryService.initDirectories(): javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
That UpdateNotifier is part of an @Startup bean. The exception at the end is where the NPE happens. Nowhere in our log do we get the "Bean identifier index built" message, which is consistent with WeldStartup passing an empty list to `BeanIdentifierIndex#build(...)`. Matěj Novotný
added a comment - Hi sven.haster,
I have submitted a PR(link) which I think should address this however we have no such testsuite which would demonstrate this problem.
Do you think you could verify it with yours?
If you need, I can explain how to patch Weld version in your WFLY to that in the PR.
Matěj Novotný
added a comment - Hi sven.haster ,
I have submitted a PR( link ) which I think should address this however we have no such testsuite which would demonstrate this problem.
Do you think you could verify it with yours?
If you need, I can explain how to patch Weld version in your WFLY to that in the PR. 
Martin Kouba
added a comment - We use a number of @Startup EJB beans. As far as I can tell, none is on a direct dependency graph with the spot where this NPE happens though.
I'm asking because the index is built before Weld initialization finished. So the only chance to get this NPE is to inject some EE component even earlier, e.g. @Startup singleton.
How would our application still fail if we modified the BeanIdentifierIndex#isBuilt() method? As far as I can tell, the if-check on SerializableContextualHolder#98 would evaluate to false, meaning line 99
You're right, I overlooked the if there 
Would storing the beanidentifier instead of the identifierIndex (SerializableContextualHolder#101-106) result in errors further down the road?
No, I think it should work ok.
Martin Kouba
added a comment - We use a number of @Startup EJB beans. As far as I can tell, none is on a direct dependency graph with the spot where this NPE happens though.
I'm asking because the index is built before Weld initialization finished. So the only chance to get this NPE is to inject some EE component even earlier, e.g. @Startup singleton.
How would our application still fail if we modified the BeanIdentifierIndex#isBuilt() method? As far as I can tell, the if-check on SerializableContextualHolder#98 would evaluate to false, meaning line 99
You're right, I overlooked the if there
Would storing the beanidentifier instead of the identifierIndex (SerializableContextualHolder#101-106) result in errors further down the road?
No, I think it should work ok. mkouba@redhat.com Thanks for the quick response. I'm still waiting on results from the testcases with extra logging, it didn't fail last night for the first night in many, of course
We use a number of @Startup EJB beans. As far as I can tell, none is on a direct dependency graph with the spot where this NPE happens though.
How would our application still fail if we modified the BeanIdentifierIndex#isBuilt() method? As far as I can tell, the if-check on SerializableContextualHolder#98 would evaluate to false, meaning line 99
idx = beanIdentifierIndex.getIndex(beanIdentifier);
wouldn't get called and that would avoid the direct NPE.
Would storing the beanidentifier instead of the identifierIndex (SerializableContextualHolder#101-106) result in errors further down the road?
Sven Haster (Inactive)
added a comment - - edited mkouba@redhat.com Thanks for the quick response. I'm still waiting on results from the testcases with extra logging, it didn't fail last night for the first night in many, of course
We use a number of @Startup EJB beans. As far as I can tell, none is on a direct dependency graph with the spot where this NPE happens though.
How would our application still fail if we modified the BeanIdentifierIndex#isBuilt() method? As far as I can tell, the if-check on SerializableContextualHolder#98 would evaluate to false, meaning line 99
idx = beanIdentifierIndex.getIndex(beanIdentifier);
wouldn't get called and that would avoid the direct NPE.
Would storing the beanidentifier instead of the identifierIndex (SerializableContextualHolder#101-106) result in errors further down the road? Martin Kouba
added a comment - sven.haster Thanks for the report. But even if we modify the BeanIdentifierIndex#checkIsBuilt() your application would still fail. So I'm wondering if we shouldn't change more. Does your test app use a @Startup EJB singleton? It would be great if you could tell us a little bit more about your setup.
Also bean identifier index is just an optimization which could be disabled: http://docs.jboss.org/weld/reference/latest-master/en-US/html/configure.html#_bean_identifier_index_optimization.
Martin Kouba
added a comment - sven.haster Thanks for the report. But even if we modify the BeanIdentifierIndex#checkIsBuilt() your application would still fail. So I'm wondering if we shouldn't change more. Does your test app use a @Startup EJB singleton? It would be great if you could tell us a little bit more about your setup.
Also bean identifier index is just an optimization which could be disabled: http://docs.jboss.org/weld/reference/latest-master/en-US/html/configure.html#_bean_identifier_index_optimization . 
Glad to hear that!
Thanks for reporting back to us.