Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-7280

Intermittently CXFRS service throws java.lang.NoClassDefFoundError: org/apache/cxf/jaxrs/impl/MetadataMap$KeyComparator while returning response.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • fuse-7.0, fuse-6.x-GA
    • jboss-fuse-6.3
    • Camel, CXF
    • None
    • % %

      • After every 3rd or 4th restart, customer is facing error.
        2017-09-14 13:08:03,906 | WARN  | postuploadevents | ServletHandler                   | 162 - org.eclipse.jetty.util - 9.2.19.v20160908 | Error for /cxf/mr/api/1/documentpostuploadevents
        java.lang.NoClassDefFoundError: org/apache/cxf/jaxrs/impl/MetadataMap$KeyComparator
                at org.apache.cxf.jaxrs.impl.MetadataMap.keySet(MetadataMap.java:178)
                at javax.ws.rs.core.Response.fromResponse(Response.java:573)
                at org.apache.camel.component.cxf.jaxrs.SimpleCxfRsBinding.buildResponse(SimpleCxfRsBinding.java:170)
                at org.apache.camel.component.cxf.jaxrs.SimpleCxfRsBinding.populateCxfRsResponseFromExchange(SimpleCxfRsBinding.java:152)
                at org.apache.camel.component.cxf.jaxrs.CxfRsInvoker.returnResponse(CxfRsInvoker.java:184)
                at org.apache.camel.component.cxf.jaxrs.CxfRsInvoker.asyncInvoke(CxfRsInvoker.java:110)
                at org.apache.camel.component.cxf.jaxrs.CxfRsInvoker.performInvocation(CxfRsInvoker.java:68)
                at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
                at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189)
                at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
                at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)[224:org.apache.cxf.cxf-core:3.1.5.redhat-630262]
                at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)[224:org.apache.cxf.cxf-core:3.1.5.redhat-630262]
                at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)[224:org.apache.cxf.cxf-core:3.1.5.redhat-630262]
                at org.apache.cxf.phase.PhaseInterceptorChain.resume(PhaseInterceptorChain.java:278)[224:org.apache.cxf.cxf-core:3.1.5.redhat-630262]
                at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:78)[224:org.apache.cxf.cxf-core:3.1.5.redhat-630262]
                at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:254)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)[58:javax.servlet-api:3.1.0]
                at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)[236:org.apache.cxf.cxf-rt-transports-http:3.1.5.redhat-630262]
                at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)[160:org.eclipse.jetty.servlet:9.2.19.v20160908]
                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)[160:org.eclipse.jetty.servlet:9.2.19.v20160908]
                at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)[175:org.ops4j.pax.web.pax-web-jetty:4.3.0]
                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)[158:org.eclipse.jetty.security:9.2.19.v20160908]
                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:287)[175:org.ops4j.pax.web.pax-web-jetty:4.3.0]
                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)[160:org.eclipse.jetty.servlet:9.2.19.v20160908]
                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)[175:org.ops4j.pax.web.pax-web-jetty:4.3.0]
                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.server.Server.handleAsync(Server.java:545)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[159:org.eclipse.jetty.server:9.2.19.v20160908]
                at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)[151:org.eclipse.jetty.io:9.2.19.v20160908]
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[162:org.eclipse.jetty.util:9.2.19.v20160908]
                at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[162:org.eclipse.jetty.util:9.2.19.v20160908]
                at java.lang.Thread.run(Thread.java:745)[:1.7.0_71]
        
      • It is a very normal CXF rest based service with route.
        	<bean id="documentServiceProcessor" class="com.xchanging.xsb.imr.rest.processor.DocumentServiceProcessor" >
        		<property name="producerTemplate" ref="producerTemplate" />
        		<property name="serviceHelper" ref="serviceHelper" />
        <cxf:rsServer id="rsdocumentPostUploadEvents" address="/mr/api/1/documentpostuploadevents"
        		serviceClass="com.xchanging.xsb.imr.rest.service.DocumentServiceResource">
        	</cxf:rsServer>
        		<route id="rsdocumentPostUploadEvents">
        			<from uri="cxfrs:bean:rsdocumentPostUploadEvents?bindingStyle=SimpleConsumer" />
        			<camel:when>
        				<simple>${in.header.operationName} == 'documentPostUploadEvents'</simple>
        				<to uri="direct:documentPostUploadEvents"/>
        			</camel:when> 
        		</route>
        		
        		<route id="documentPostUploadEvents">
        			<from uri="direct:documentPostUploadEvents" />
        			<onException>
        				<exception>java.lang.Exception</exception>
        				<handled>
        					<constant>true</constant>
        				</handled>
        				<log loggingLevel="ERROR" message="Exception Caught in Document Post Upload Events" />
        				<setHeader headerName="requestErrorFrom">
        					<simple>${exception}</simple>
        				</setHeader>
        				<to uri="direct:errorResponse" />
        			</onException>
        			<process ref="documentServiceProcessor" />
        		</route>
        	</bean>
        
        
      • Customer doesn't observe this in Windows environment. He only observe it in RHEL 6.9 OS with jdk 1.7. It is not reproducible at my end with 6.9 RHEL+ 1.7 jdk or 7.3 RHEL + 1.8 jdk.
      • Looking at logs and exception it is apparent that response is being not returned successfully. I have tried almost everything but not able to reproduce it. I already suggested customer:
        1) To test once with enabling dynamic-import.
        2) There were two scala library which was leading to Warnings in dev:show-tree output of cxf-frontend-jaxrs bundle. I suggested to remove one of the scala library no affect.
        3) Suggested to add header org.apache.cxf.jaxrs.impl in import-package section of pom.xml for this service but no success.
        4) Already built his complete code and deployed in 6.9 RHEL and 7.3 RHEL, but no success.

        1. DocumentServiceProcessor.java
          3 kB
        2. fuse_trace.zip
          181 kB
        3. cxf-rt-frontend-jaxrs-3.1.5.redhat-630262.jar
          599 kB
        4. cxf-rt-frontend-jaxrs-3.1.5.redhat-630262-new.jar
          600 kB
        5. info.zip
          239 kB
        6. noclassdef.log.zip
          85 kB
        7. 20SepLogs.zip
          4.58 MB
        8. org.apache.felix.framework-4.4.1.jar
          514 kB
        9. 21SepLogs.zip
          5.39 MB
        10. fuselog_27Sep.zip
          10.50 MB
        11. list_of_bundles_4Oct.zip
          11 kB
        12. cxf-rt-frontend-jaxrs-3.1.5.redhat-630262a.jar
          600 kB

            ggrzybek Grzegorz Grzybek
            rhn-support-cpandey Chandra Shekhar Pandey (Inactive)
            Viliam Kasala Viliam Kasala
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: