Uploaded image for project: 'JBoss Web Services'
  1. JBoss Web Services
  2. JBWS-364

many parallel requests to http://host:port/web/service/context?wsdl cause exceptions and loss of service in ws4ee

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • jboss-ws4ee-4.0.4RC1
    • jboss-ws4ee-4.0.2, jboss-ws4ee-4.0.3
    • jbossws-native
    • None

      We have tested and this problem exists in JBoss 4.0.3RC2 as well as 4.0.2. See ++ at start of line below for why the initial submission is Priority: Critical

      Here is the problem. A JAXRPC web service client may/must do this during initialisation:

      Service service = factory.createService(..., wsdlURL, ...);

      This call internally retrieves the WSDL from http://host:port/web/service/context?wsdl, the provided URL.

      We had a flawed test harness that created many instances of a JAXRPC Service from which to create client endpoints (ie endpoint = service.getPort(InterfaceName.class)). The test harness was later corrected to use a single Service to create its multiple endpoints, but ...

      An external consumer of our web service triggerred the same problem using two instances of their client running on two separate computers. It was far less likely to happen, but it has happenned once so far.

      Our flawed test harness made it possible to have multiple Threads under JBoss/Tomcat concurrently call upon the ws4ee core servlet to respond with the WSDL for our web service.

      It seems, from some interesting Exceptions and stack traces we caused, that the call made by ws4ee to WSDL4J to serialise/copy the deployment-time published WSDL to the HTTP response as XML text is the root of the problem.

      My initial ideas:

      Either the WSDL4J API is not Thread safe, and the ws4ee code is not yet guarding its calls upon that unsafe API/method/entry point (the method is com.ibm.wsdl.xml.WSDLWriterImpl.getDocument(Unknown Source), see stack trace below)

      OR

      the problem is somehow driven by a bug in WSDL4J not liking our WSDL file contents, in particular the XML Schema/s in the <types>...</types>.

      Sometimes the Exceptions generated cause the WSDL in the HTTP response to be incomplete. We saw a missing complexType in the XML Schema section of the WSDL received by the web service client, on numerous occasions, in correlation with NullPointerException (see below).

      It also seems that WSDL4J may use recursion in its implementation. server.log has been overwritten and I cannot locate a stack trace to show this, and cannot reproduce the stack traces.
      However it is easy to reproduce the StackOverflowError's that ultimately occur, and render the /our/web/service?wsdl non-functional. server.log contains no stack traces for the StackOverflowError's. (Unsurprising?)

      Speculation:
      The parallel Threads traversing an in-memory WSDL representation seem to cause some servlet processing Threads to experience StackOverflowError.

      Once this happens, all HTTP requests to http://host:port/web/service/context?wsdl result in an empty response. (Observed with 'curl' as well as a stand alone JAXRPC web service client.)

      Clients that successfully create their Service before this problem appears can continue to call on the web service through their JAXRPC endpoint instance. Later clients cannot initialise successfully.

      ++ Note: Though in normal operation a well written (set of) web service clients are very unlikely to make this happen, malicious use of a generic http client like curl can render the web service effectively inoperative.

      Here is a jboss 4.0.2 server.log fragment for the case where incomplete WSDL is provided in the HTTP response to the /web/service/context?wsdl request, with /our/web/service substituted for our actual endpoint URL. It makes the point of call from ws4ee to WSDL4J pretty obvious:

      2005-08-03 22:07:35,504 DEBUG [org.jboss.webservice.server.ServiceEndpointServletJSE] doGet: http://support45:8080/our/web/service?wsdl
      2005-08-03 22:07:35,504 DEBUG [org.jboss.webservice.server.ServiceEndpointServletJSE] Set transport.url=/our/web/service
      2005-08-03 22:07:35,504 DEBUG [org.jboss.webservice.server.ServiceEndpointServletJSE] Process wsdl request
      2005-08-03 22:07:35,508 ERROR [org.jboss.webservice.server.InvokerProvider] Cannot process WSDL document
      java.lang.NullPointerException
      at org.apache.xerces.dom.ParentNode.nodeListItem(Unknown Source)
      at org.apache.xerces.dom.ParentNode.item(Unknown Source)
      at com.ibm.wsdl.util.xml.DOM2Writer.print(Unknown Source)
      at com.ibm.wsdl.util.xml.DOM2Writer.print(Unknown Source)
      at com.ibm.wsdl.util.xml.DOM2Writer.print(Unknown Source)
      at com.ibm.wsdl.util.xml.DOM2Writer.serializeAsXML(Unknown Source)
      at javax.wsdl.extensions.UnknownExtensionSerializer.marshall(Unknown Source)
      at com.ibm.wsdl.xml.WSDLWriterImpl.printExtensibilityElements(Unknown Source)
      at com.ibm.wsdl.xml.WSDLWriterImpl.printTypes(Unknown Source)
      at com.ibm.wsdl.xml.WSDLWriterImpl.printDefinition(Unknown Source)
      at com.ibm.wsdl.xml.WSDLWriterImpl.writeWSDL(Unknown Source)
      at com.ibm.wsdl.xml.WSDLWriterImpl.getDocument(Unknown Source)
      at org.jboss.webservice.server.WSDLRequestHandler.getWSDLDocument(WSDLRequestHandler.java:119)
      at org.jboss.webservice.server.WSDLRequestHandler.getDocumentForPath(WSDLRequestHandler.java:73)
      at org.jboss.webservice.server.InvokerProvider.generateWSDL(InvokerProvider.java:177)
      at org.jboss.axis.strategies.WSDLGenStrategy.visit(WSDLGenStrategy.java:74)
      at org.jboss.axis.SimpleChain.doVisiting(SimpleChain.java:160)
      at org.jboss.axis.SimpleChain.generateWSDL(SimpleChain.java:143)
      at org.jboss.axis.handlers.soap.SOAPService.generateWSDL(SOAPService.java:426)
      at org.jboss.axis.server.AxisServer.generateWSDL(AxisServer.java:555)
      at org.jboss.webservice.server.AbstractServlet.processWsdlRequest(AbstractServlet.java:138)
      at org.jboss.webservice.server.ServiceEndpointServlet.doGet(ServiceEndpointServlet.java:108)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
      at org.jboss.axis.transport.http.AxisServletBase.service(AxisServletBase.java:370)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
      at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
      at java.lang.Thread.run(Unknown Source)
      2005-08-03 22:07:35,513 INFO [org.jboss.axis.transport.http.AxisServlet] org.jboss.axis.AxisFault: Cannot get wsdl document for service: /our/web/service

      Other deeper and more interesting stack traces were produced in the server by our original test client code, but the logs were overwritten with our jboss re-starts and the default log4j.xml config, so I cannot present them here. I have tried to reproduce them, but can only observe the above, followed by later StackOverflowError's (with no stack traces) and loss of functionality from ws4ee.

      Speculation:
      Could the problem cause Threads to be blocked/consumed in the JBoss/Tomcat thread pool, and continued ?wsdl accesses lead to complete starvation?

      If I can reproduce the 'missing' stacktraces, I hope I can update this submission. Never used JIRA to submit before

            Unassigned Unassigned
            johnkavs John Kavadias (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: