Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-1875

Logstash TCP-Input throws "Bad record MAC" when trying to connect with WildFly over SSL/TLS

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • None
    • Logging
    • Hide
      1. Created self-signed Logstash cert with OpenSSL: 
           openssl req -x509 -newkey rsa:4096 -keyout logstash_ssl.key -out logstash_ssl.crt -nodes -days 365
      2. Configured Logstash TCP input to use generated cert and key: 
        input {
  tcp {
    port => 12202
    codec => "json"
    ssl_enable => true
    ssl_cert => "/path/to/logstash_ssl.crt"
    ssl_key => "/path/to/logstash_ssl.key"
    ssl_verify => false 
  }
}
      3. Imported cert into a new truststore: 
           keytool -import -alias mycert -file mycert.cer -keystore logstashTruststore
      4. Added jboss-logmanager-ext to modules
      5. Configured SocketHandler in standalone.xml as follows (formatter config is omitted): 
            <custom-handler name="LOGSTASH" class="org.jboss.logmanager.ext.handlers.SocketHandler" module="log.logmanager-ext">
	   <level name="DEBUG"/>
	   <formatter>
		  <named-formatter name="LOGSTASH-FORMATTER"/>
	   </formatter>
	    <properties>
                 <property name="hostname" value="192.168.144.101"/>
		 <property name="port" value="12202"/>
		 <property name="protocol" value="SSL_TCP"/>
	      </properties>
      </custom-handler>
      6. Added truststore path as well as password to VM options: 
           -Djavax.net.ssl.trustStore=/path/to/logstashTruststore
   -Djavax.net.ssl.trustStorePassword="mypassword"
      7. related versions:
      • JDK 1.8.0_11 and 1.8.0_101
      • WildFly 10.1.0 and 10.0.0
      • Logstash 2.1.3 (same behavior with latest Logstash 5.0)
      Show
      Created self-signed Logstash cert with OpenSSL: openssl req -x509 -newkey rsa:4096 -keyout logstash_ssl.key -out logstash_ssl.crt -nodes -days 365 Configured Logstash TCP input to use generated cert and key: input { tcp { port => 12202 codec => "json" ssl_enable => true ssl_cert => "/path/to/logstash_ssl.crt" ssl_key => "/path/to/logstash_ssl.key" ssl_verify => false } } Imported cert into a new truststore: keytool - import -alias mycert -file mycert.cer -keystore logstashTruststore Added jboss-logmanager-ext to modules Configured SocketHandler in standalone.xml as follows (formatter config is omitted): <custom-handler name= "LOGSTASH" class= "org.jboss.logmanager.ext.handlers.SocketHandler" module= "log.logmanager-ext" > <level name= "DEBUG" /> <formatter> <named-formatter name= "LOGSTASH-FORMATTER" /> </formatter> <properties> <property name= "hostname" value= "192.168.144.101" /> <property name= "port" value= "12202" /> <property name= "protocol" value= "SSL_TCP" /> </properties> </custom-handler> Added truststore path as well as password to VM options: -Djavax.net.ssl.trustStore=/path/to/logstashTruststore -Djavax.net.ssl.trustStorePassword= "mypassword" related versions: JDK 1.8.0_11 and 1.8.0_101 WildFly 10.1.0 and 10.0.0 Logstash 2.1.3 (same behavior with latest Logstash 5.0)
    • Compatibility/Configuration

    Description

      I use the jboss-logmanger-ext library for transfering log records to Logstash over a secure socket. For that purpose, my Logstash TCP-Input config authenticates with WildFly by means of a self-signed certificate. However, some time after SSL handshake has started, the following exception is thrown:

      LogManager error of type FLUSH_FAILURE: Error on flush
java.net.SocketException: Socket is closed
	at sun.security.ssl.SSLSocketImpl.getOutputStream(SSLSocketImpl.java:2240)
	at org.jboss.logmanager.handlers.TcpOutputStream.flush(TcpOutputStream.java:210)
	at org.jboss.logmanager.handlers.UninterruptibleOutputStream.flush(UninterruptibleOutputStream.java:110)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
	at org.jboss.logmanager.ext.handlers.SocketHandler.safeFlush(SocketHandler.java:340)
	at org.jboss.logmanager.ext.handlers.SocketHandler.flush(SocketHandler.java:169)
	at org.jboss.logmanager.ExtHandler.doPublish(ExtHandler.java:104)
	at org.jboss.logmanager.ext.handlers.SocketHandler.doPublish(SocketHandler.java:159)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:850)
	at org.jboss.logmanager.Logger.log(Logger.java:596)
	at org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71)
	at org.jboss.stdio.WriterOutputStream.finish(WriterOutputStream.java:143)
	at org.jboss.stdio.WriterOutputStream.flush(WriterOutputStream.java:164)
	at java.io.PrintStream.write(PrintStream.java:482)
	at org.jboss.stdio.StdioContext$DelegatingPrintStream.write(StdioContext.java:264)
	at java.io.PrintStream.write(PrintStream.java:480)
	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
	at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
	at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
	at java.io.PrintStream.newLine(PrintStream.java:546)
	at java.io.PrintStream.println(PrintStream.java:696)
	at sun.misc.HexDumpEncoder.encodeLineSuffix(HexDumpEncoder.java:116)
	at sun.misc.CharacterEncoder.encodeBuffer(CharacterEncoder.java:297)
	at sun.security.ssl.CipherBox.encrypt(CipherBox.java:306)
	at sun.security.ssl.OutputRecord.encrypt(OutputRecord.java:264)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:859)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:847)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at org.jboss.logmanager.handlers.TcpOutputStream.write(TcpOutputStream.java:182)
	at org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(UninterruptibleOutputStream.java:84)
	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
	at org.jboss.logmanager.ext.handlers.SocketHandler.safeFlush(SocketHandler.java:340)
	at org.jboss.logmanager.ext.handlers.SocketHandler.flush(SocketHandler.java:169)
	at org.jboss.logmanager.ExtHandler.doPublish(ExtHandler.java:104)
	at org.jboss.logmanager.ext.handlers.SocketHandler.doPublish(SocketHandler.java:159)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:850)
	at org.jboss.logmanager.Logger.log(Logger.java:802)
	at org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.java:53)
	at org.jboss.logging.Logger.logf(Logger.java:2398)
	at org.jboss.msc.service.ServiceLogger_$logger.greeting(ServiceLogger_$logger.java:65)
	at org.jboss.msc.service.ServiceContainerImpl.<clinit>(ServiceContainerImpl.java:93)
	at org.jboss.msc.service.ServiceContainer$Factory.create(ServiceContainer.java:258)
	at org.jboss.as.server.BootstrapImpl$ShutdownHook.register(BootstrapImpl.java:214)

      On the Logstash side, the following error message appears in the logs:

      :message=>"An error occurred. Closing connection", :exception=>#<IOError: bad record MAC>

      Afterwards, WildFly hangs forever without deploying my webapp or doing anything else. Before that happens, the handshake goes through these phases:

      • *** ClientHello, TLSv1.2
      • *** ServerHello, TLSv1.2
      • %% Initialized: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
      • Found trusted certificate
      • *** ECDH ServerKeyExchange
      • *** ServerHelloDone
      • *** ECDHClientKeyExchange
      • SESSION KEYGEN:
      • CONNECTION KEYGEN:
      • *** Finished

      When disabling SSL both on WildFly and Logstash side, everything works fine.

      Attachments

        Activity

          People

            jperkins-rhn James Perkins
            pkleindienst_jira Patrick Kleindienst (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: