Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-4693

Slow HTTPS Upload provokes High CPU Load

    Details

    • Steps to Reproduce:
      Hide
      • Install Wildfly 9.0.0 CR1
      • Activate HTTPS in config
      • Deploy a sample webappliction (e.g. https://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/ should work too, although no doPost implemented)
      • Adjust the following client code (https address, upload file)
      • Run the code
      • CPU Load will increase to 100% during the sleeps of the HTTPS client
      TestUndertowLoad.java
      package undertowtest;
      
      import java.io.DataOutputStream;
      import java.io.File;
      import java.io.FileNotFoundException;
      import java.io.IOException;
      import java.io.RandomAccessFile;
      import java.net.URL;
      import javax.net.ssl.HttpsURLConnection;
      
      /**
       * Tests the Undertow module, how it behaves in case of a HTTPS client which sends
       * its POST data very slowly.
       */
      public class TestUndertowLoad {
      	/**
      	 * Starter method of the test.
      	 *
      	 * @param args Arguments,  not used
      	 *
      	 * @throws IOException If a connection to webserver occured.
      	 */
      	public static void main(String[] args) throws IOException {
      		String host = "localhost:8443";
      		String uploadFile = "/opt/import/testfile.bin";
      		int chunkSize = 50;
      		int sleepTime = 1000;
      
      		String url = "https://" + host + "/sample/hello";
      		URL obj = new URL(url);
      		HttpsURLConnection con = (HttpsURLConnection)obj.openConnection();
      		con.setRequestMethod("POST");
      
      		byte[] data = readFile(new File(uploadFile));
      		con.setFixedLengthStreamingMode(data.length);
      
      		// send data using outputstream
      		con.setDoOutput(true);
      
      		System.out.println("Sending 'POST' request to URL : " + url+"\n");
      		DataOutputStream wr = new DataOutputStream(con.getOutputStream());
      		int index = 0;
      		while (index < data.length) {
      			// slow upload to provoke https://issues.jboss.org/browse/UNDERTOW-345
      			// CPU load will rise to 100 percent during the sleeps
      			wr.writeByte(data[index]);
      			System.out.print(".");
      			
      			if ((index % chunkSize) == (chunkSize-1)) {
      				try {
      					System.out.println("Chunk sent, sleeping...");
      					Thread.sleep(sleepTime);
      				} catch (InterruptedException e) {
      					e.printStackTrace();
      				}
      			}
      			index++;
      		}
      
      		wr.flush();
      		wr.close();
      
      		int responseCode = con.getResponseCode();
      		System.out.println("Response Code : " + responseCode);
      	}
      
      	private static byte[] readFile(File file) throws IOException {
      		byte[] ret;
      
      		try {
      			RandomAccessFile f = new RandomAccessFile(file, "r");
      
      			try {
      				ret = new byte[(int)f.length()];
      				f.read(ret);
      			} finally {
      				f.close();
      			}
      		} catch (FileNotFoundException e) {
      			ret = null;
      		}
      
      		return ret;
      	}
      }
      
      
      Show
      Install Wildfly 9.0.0 CR1 Activate HTTPS in config Deploy a sample webappliction (e.g. https://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/ should work too, although no doPost implemented) Adjust the following client code (https address, upload file) Run the code CPU Load will increase to 100% during the sleeps of the HTTPS client TestUndertowLoad.java package undertowtest; import java.io.DataOutputStream; import java.io.File; import java.io.FileNotFoundException; import java.io.IOException; import java.io.RandomAccessFile; import java.net.URL; import javax.net.ssl.HttpsURLConnection; /** * Tests the Undertow module, how it behaves in case of a HTTPS client which sends * its POST data very slowly. */ public class TestUndertowLoad { /** * Starter method of the test. * * @param args Arguments, not used * * @ throws IOException If a connection to webserver occured. */ public static void main( String [] args) throws IOException { String host = "localhost:8443" ; String uploadFile = "/opt/ import /testfile.bin" ; int chunkSize = 50; int sleepTime = 1000; String url = "https: //" + host + "/sample/hello" ; URL obj = new URL(url); HttpsURLConnection con = (HttpsURLConnection)obj.openConnection(); con.setRequestMethod( "POST" ); byte [] data = readFile( new File(uploadFile)); con.setFixedLengthStreamingMode(data.length); // send data using outputstream con.setDoOutput( true ); System .out.println( "Sending 'POST' request to URL : " + url+ "\n" ); DataOutputStream wr = new DataOutputStream(con.getOutputStream()); int index = 0; while (index < data.length) { // slow upload to provoke https://issues.jboss.org/browse/UNDERTOW-345 // CPU load will rise to 100 percent during the sleeps wr.writeByte(data[index]); System .out.print( "." ); if ((index % chunkSize) == (chunkSize-1)) { try { System .out.println( "Chunk sent, sleeping..." ); Thread .sleep(sleepTime); } catch (InterruptedException e) { e.printStackTrace(); } } index++; } wr.flush(); wr.close(); int responseCode = con.getResponseCode(); System .out.println( "Response Code : " + responseCode); } private static byte [] readFile(File file) throws IOException { byte [] ret; try { RandomAccessFile f = new RandomAccessFile(file, "r" ); try { ret = new byte [( int )f.length()]; f.read(ret); } finally { f.close(); } } catch (FileNotFoundException e) { ret = null ; } return ret; } }
    • Workaround Description:
      Hide

      This is the recommended solution for people looking to fix the CPU problem. Just set the read-timeout in standalone.xml, and High CPU Load is gone (even if the client sends all bytes within the timeout).

      <https-listener name="https" socket-binding="https" security-realm="UndertowRealm" read-timeout="20000" />

      Show
      This is the recommended solution for people looking to fix the CPU problem. Just set the read-timeout in standalone.xml, and High CPU Load is gone (even if the client sends all bytes within the timeout). <https-listener name="https" socket-binding="https" security-realm="UndertowRealm" read-timeout="20000" />

      Description

      This issue is a reopen of https://issues.jboss.org/browse/UNDERTOW-345.

      This Issue is not fixed with Wildfly 9.0 CR1 when using the standard config without any timeouts set (however, above mentioned ticket indicates issue was fixed in Undertow 1.2.0.Beta6).

      This is a major bug, as just one client with bad connectivity will cause a 100% CPU Load on Wildfly whilst the HTTPS upload is being read. It's also a real-world bug, as every note-worthy productive server will have at least one client with bad connectivity at any given time. Duration of the high CPU load directly correlates to the duration it takes the client to send all his POST data.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                dmlloyd David Lloyd
                Reporter:
                sepadev Serge Mürset
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: