Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1422

Concurrency issue with HTTP/2.0 and OpenJDK 11

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 2.0.13.Final
    • Core
    • None
    • Hide

      1) start server with OpenJDK 11

      public class Server {
    public static void main(String[] args) {
        Undertow server = Undertow.builder().setServerOption(UndertowOptions.ENABLE_HTTP2, Boolean.TRUE)
                                  .addHttpsListener(8443, "0.0.0.0", new SSLContextBuilder().build()) // load self signed cert to create SSLContext
                                  .setHandler(exchange -> {
                                      System.out.println(exchange.getProtocol());
                                      exchange.getResponseHeaders().put(Headers.CONTENT_TYPE, "text/plain");
                                      exchange.getResponseSender().send("ok");
                                  }).build();
        server.start();
    }
}

      2) run ClientTest

      public class ClientTest {
    public static void main(String[] args) throws InterruptedException, KeyManagementException, NoSuchAlgorithmException {
        String url = "https://localhost:8443/test";
        testJDKClient(url);
        Thread.sleep(600000);
        System.out.println("finished");
    }

    private static void testJDKClient(String uri) throws KeyManagementException, NoSuchAlgorithmException {
        SSLContext sslContext = SSLContext.getInstance("TLS");
        sslContext.init(null, new TrustManager[]{new TrustAllTrustManager()}, new SecureRandom());
        HttpClient client = HttpClient.newBuilder().sslContext(sslContext).build();
        for (int i = 0; i < 5; i++) {
            new Thread(() -> {
                try {
                    HttpResponse<String> response = client.send(HttpRequest.newBuilder().uri(new URI(uri)).GET().build(), HttpResponse.BodyHandlers.ofString());
                    System.out.println(response.statusCode());
                } catch (Exception e) {
                    throw new Error(e);
                }
            }).start();
        }
    }
}

      3) after all requests are executed, create thread dump, there will be many client works result in infinite loop, e.g.

      "HttpClient-1-Worker-0@2120" daemon prio=5 tid=0x15 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at sun.security.ssl.SSLEngineImpl.checkParams(SSLEngineImpl.java:397)
	  at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:133)
	  - locked <merged>(a sun.security.ssl.SSLEngineImpl)
	  at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:116)
	  at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:519)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Writer.wrapBuffers(SSLFlowDelegate.java:821)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Writer.processData(SSLFlowDelegate.java:736)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Writer$WriterDownstreamPusher.run(SSLFlowDelegate.java:645)
	  at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	  at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	  at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	  at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Writer.triggerWrite(SSLFlowDelegate.java:722)
	  at jdk.internal.net.http.common.SSLFlowDelegate.doHandshake(SSLFlowDelegate.java:1024)
	  at jdk.internal.net.http.common.SSLFlowDelegate.doClosure(SSLFlowDelegate.java:1094)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer(SSLFlowDelegate.java:500)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:389)
	  - locked <merged>(a java.lang.Object)
	  at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:263)
	  at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	  - locked <merged>(a java.lang.Object)
	  at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	  at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	  at java.lang.Thread.run(Thread.java:834)
      Show
      1) start server with OpenJDK 11 public class Server { public static void main( String [] args) { Undertow server = Undertow.builder().setServerOption(UndertowOptions.ENABLE_HTTP2, Boolean .TRUE) .addHttpsListener(8443, "0.0.0.0" , new SSLContextBuilder().build()) // load self signed cert to create SSLContext .setHandler(exchange -> { System .out.println(exchange.getProtocol()); exchange.getResponseHeaders().put(Headers.CONTENT_TYPE, "text/plain" ); exchange.getResponseSender().send( "ok" ); }).build(); server.start(); } } 2) run ClientTest public class ClientTest { public static void main( String [] args) throws InterruptedException, KeyManagementException, NoSuchAlgorithmException { String url = "https: //localhost:8443/test" ; testJDKClient(url); Thread .sleep(600000); System .out.println( "finished" ); } private static void testJDKClient( String uri) throws KeyManagementException, NoSuchAlgorithmException { SSLContext sslContext = SSLContext.getInstance( "TLS" ); sslContext.init( null , new TrustManager[]{ new TrustAllTrustManager()}, new SecureRandom()); HttpClient client = HttpClient.newBuilder().sslContext(sslContext).build(); for ( int i = 0; i < 5; i++) { new Thread (() -> { try { HttpResponse< String > response = client.send(HttpRequest.newBuilder().uri( new URI(uri)).GET().build(), HttpResponse.BodyHandlers.ofString()); System .out.println(response.statusCode()); } catch (Exception e) { throw new Error(e); } }).start(); } } } 3) after all requests are executed, create thread dump, there will be many client works result in infinite loop, e.g. "HttpClient-1-Worker-0@2120" daemon prio=5 tid=0x15 nid=NA runnable java.lang.Thread.State: RUNNABLE at sun.security.ssl.SSLEngineImpl.checkParams(SSLEngineImpl.java:397) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:133) - locked <merged>(a sun.security.ssl.SSLEngineImpl) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:116) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:519) at jdk.internal.net.http.common.SSLFlowDelegate$Writer.wrapBuffers(SSLFlowDelegate.java:821) at jdk.internal.net.http.common.SSLFlowDelegate$Writer.processData(SSLFlowDelegate.java:736) at jdk.internal.net.http.common.SSLFlowDelegate$Writer$WriterDownstreamPusher.run(SSLFlowDelegate.java:645) at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271) at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224) at jdk.internal.net.http.common.SSLFlowDelegate$Writer.triggerWrite(SSLFlowDelegate.java:722) at jdk.internal.net.http.common.SSLFlowDelegate.doHandshake(SSLFlowDelegate.java:1024) at jdk.internal.net.http.common.SSLFlowDelegate.doClosure(SSLFlowDelegate.java:1094) at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer(SSLFlowDelegate.java:500) at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:389) - locked <merged>(a java.lang.Object) at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:263) at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175) - locked <merged>(a java.lang.Object) at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834)

    Description

      I was evaluating to upgrade our microservice to openjdk 11 and to utilize JDK httpclient 2.0 with undertow server,
      and discovered concurrency issue can cause httpclient to result in infinite loop state,

      after investigation, I am able to reproduce it locally with simple setup, as following,
      the symptom is after sending multiple http/2.0 requests in small amount of time, it seems the undertow server won't clearly close each http2 channel, and httpclient thinks there was more data to process and keep reading and try to process, (looks like keep trying to handshake/need_wrap)
      and this result in infinite loop on the client side

      some findings
      1) if I use openjdk 8 or 10 to run same example server code, run ClientTest with openjdk 11, there is no issues
      2) if I send request one by one, there won't be any issues,
      3) undertow HTTP/1.1 works under openjdk 11 for both HTTP or HTTPS,
      4) if I use JDK 11 HTTPClient to test other http/2.0 site (e.g. by GoogleCloud LB), there is no issue

      please advise any future step I should take or provide more details.

      Attachments

        Activity

          People

            sdouglas1@redhat.com Stuart Douglas
            neowu_jira Neo Wu (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: