Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-1017

Reload operation causes crash of EAP server

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • 2.0.1.Final
    • 2.0.0.CR1
    • Management
    • None
    • Hide

      Configuration:
      1 server with default standalone-full-ha.xml configuration
      Client which repeatedly sends reload command to the server.

      for (int i = 0; i < 100; i++) {
    CLI cli = CLI.newInstance();
    cli.connect(cliConf.getHost(), cliConf.getPort(), cliConf.getUser(), cliConf.getPassword());
    cli.cmd("reload");
    cli.disconnect();
    Assert.assertTrue(CheckServerAvailableUtils.waitHornetQToAlive(cliConf.getHost(), cliConf.getPort(), 15000));
}
      Show
      Configuration: 1 server with default standalone-full-ha.xml configuration Client which repeatedly sends reload command to the server. for ( int i = 0; i < 100; i++) { CLI cli = CLI.newInstance(); cli.connect(cliConf.getHost(), cliConf.getPort(), cliConf.getUser(), cliConf.getPassword()); cli.cmd( "reload" ); cli.disconnect(); Assert.assertTrue(CheckServerAvailableUtils.waitHornetQToAlive(cliConf.getHost(), cliConf.getPort(), 15000)); }

    Description

      Repeated execution of reload request causes EAP server crash.

      10:43:15,179 ERROR [org.jboss.msc.service] (MSC service thread 1-4) MSC000002: Invocation of listener "org.jboss.as.controller.operations.common.ProcessReloadHandler$1$1$1@3d119a3c" failed: java.lang.IllegalArgumentException: Container is shutting down
      at org.jboss.msc.service.ServiceControllerImpl.internalSetMode(ServiceControllerImpl.java:762) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
      at org.jboss.msc.service.ServiceControllerImpl.setMode(ServiceControllerImpl.java:753) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
      at org.jboss.as.controller.operations.common.ProcessReloadHandler$1$1$1.transition(ProcessReloadHandler.java:90) [wildfly-controller-2.0.0.Beta4.jar:2.0.0.Beta4]
      at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceControllerImpl.java:1533) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
      at org.jboss.msc.service.ServiceControllerImpl.access$2800(ServiceControllerImpl.java:51) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
      at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceControllerImpl.java:2099) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]
      at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]

      On IBM Java reload resquests cause even JVM crash.

      #0: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x8a455f) [0x7f44cce6055f]
      #1: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x1042f1) [0x7f44cc6c02f1]
      #2: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9prt28.so(+0x255ee) [0x7f44d8b515ee]
      #3: /lib64/libpthread.so.0(+0x10430) [0x7f44dafd9430]
      #4: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x74909e) [0x7f44ccd0509e]
      #5: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x749372) [0x7f44ccd05372]
      #6: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x437922) [0x7f44cc9f3922]
      #7: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x4385bb) [0x7f44cc9f45bb]
      #8: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x448e10) [0x7f44cca04e10]
      #9: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x449855) [0x7f44cca05855]
      #10: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44dd5d) [0x7f44cca09d5d]
      #11: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44eb3e) [0x7f44cca0ab3e]
      #12: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44dc40) [0x7f44cca09c40]
      #13: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44eb3e) [0x7f44cca0ab3e]
      #14: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44dc40) [0x7f44cca09c40]
      #15: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44eb3e) [0x7f44cca0ab3e]
      #16: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44ec75) [0x7f44cca0ac75]
      #17: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x44effb) [0x7f44cca0affb]
      #18: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x2cd94d) [0x7f44cc88994d]
      #19: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x2cdc19) [0x7f44cc889c19]
      #20: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x2cee94) [0x7f44cc88ae94]
      #21: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x7c748b) [0x7f44ccd8348b]
      #22: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x7cee4c) [0x7f44ccd8ae4c]
      #23: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x10c7b5) [0x7f44cc6c87b5]
      #24: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x10dcab) [0x7f44cc6c9cab]
      #25: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9prt28.so(+0x25975) [0x7f44d8b51975]
      #26: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x10fdf0) [0x7f44cc6cbdf0]
      #27: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x1110ed) [0x7f44cc6cd0ed]
      #28: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9prt28.so(+0x25975) [0x7f44d8b51975]
      #29: /opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so(+0x111a04) [0x7f44cc6cda04]
      Unhandled exception
      Type=Segmentation error vmState=0x00051cff
      J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
      Handler1=00007F44D920CD90 Handler2=00007F44D8B513F0 InaccessibleAddress=0000000000000000
      RDI=00007F440613D710 RSI=FFFFFFFFFFDF3B65 RAX=0000000000000000 RBX=00007F4406141390
      RCX=0000000000000026 RDX=0000000000000001 R8=00007F44CCF4EA52 R9=0000000000000000
      R10=00007F4406145740 R11=0000000000000003 R12=00007F44076AEF00 R13=0000000000000000
      R14=0000000000000000 R15=0000000000000000
      RIP=00007F44CCD0509E GS=0000 FS=0000 RSP=00007F44C9A84C20
      EFlags=0000000000210246 CS=0033 RBP=0000000000000000 ERR=0000000000000004
      TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
      xmm0 00007f44003b6e6f (f: 3894895,000000, d: 6,913462e-310)
      xmm1 0000000000000000 (f: 0,000000, d: 0,000000e+00)
      xmm2 616c2f6176616a4c (f: 1986095744,000000, d: 1,981285e+161)
      xmm3 6a00656c62617468 (f: 1650553984,000000, d: 4,016150e+202)
      xmm4 0000000000000000 (f: 0,000000, d: 0,000000e+00)
      xmm5 03d303d203d103d0 (f: 64029648,000000, d: 3,048729e-290)
      xmm6 49676e697274532f (f: 1920226048,000000, d: 4,180283e+45)
      xmm7 4573646e756f4266 (f: 1970225792,000000, d: 3,751018e+26)
      xmm8 03d603d503d403d3 (f: 64226260,000000, d: 3,529736e-290)
      xmm9 03de03dd03dc03db (f: 64750556,000000, d: 4,812424e-290)
      xmm10 03e603e503e403e3 (f: 65274852,000000, d: 7,059551e-290)
      xmm11 03ee03ed03ec03eb (f: 65799148,000000, d: 9,624926e-290)
      xmm12 03f603f503f403f3 (f: 66323444,000000, d: 1,411926e-289)
      xmm13 03fe03fd03fc03fb (f: 66847740,000000, d: 1,925001e-289)
      xmm14 0406040504040403 (f: 67372032,000000, d: 2,823883e-289)
      xmm15 040e040d040c040b (f: 67896328,000000, d: 3,850033e-289)
      Module=/opt/ibm/java-x86_64-80/jre/lib/amd64/compressedrefs/libj9jit28.so
      Module_base_address=00007F44CC5BC000

      Method_being_compiled=org/jboss/as/controller/OperationContextImpl$ContextServiceBuilder.install()Lorg/jboss/msc/service/ServiceController;
      Target=2_80_20150630_255633 (Linux 4.1.6-201.fc22.x86_64)
      CPU=amd64 (8 logical CPUs) (0x3d3def000 RAM)
      ----------- Stack Backtrace -----------
      (0x00007F44CCD0509E [libj9jit28.so+0x74909e])
      (0x00007F44CCD05372 [libj9jit28.so+0x749372])
      (0x00007F44CC9F3922 [libj9jit28.so+0x437922])
      (0x00007F44CC9F45BB [libj9jit28.so+0x4385bb])
      (0x00007F44CCA04E10 [libj9jit28.so+0x448e10])
      (0x00007F44CCA05855 [libj9jit28.so+0x449855])
      (0x00007F44CCA09D5D [libj9jit28.so+0x44dd5d])
      (0x00007F44CCA0AB3E [libj9jit28.so+0x44eb3e])
      (0x00007F44CCA09C40 [libj9jit28.so+0x44dc40])
      (0x00007F44CCA0AB3E [libj9jit28.so+0x44eb3e])
      (0x00007F44CCA09C40 [libj9jit28.so+0x44dc40])
      (0x00007F44CCA0AB3E [libj9jit28.so+0x44eb3e])
      (0x00007F44CCA0AC75 [libj9jit28.so+0x44ec75])
      (0x00007F44CCA0AFFB [libj9jit28.so+0x44effb])
      (0x00007F44CC88994D [libj9jit28.so+0x2cd94d])
      (0x00007F44CC889C19 [libj9jit28.so+0x2cdc19])
      (0x00007F44CC88AE94 [libj9jit28.so+0x2cee94])
      (0x00007F44CCD8348B [libj9jit28.so+0x7c748b])
      (0x00007F44CCD8AE4C [libj9jit28.so+0x7cee4c])
      (0x00007F44CC6C87B5 [libj9jit28.so+0x10c7b5])
      (0x00007F44CC6C9CAB [libj9jit28.so+0x10dcab])
      (0x00007F44D8B51975 [libj9prt28.so+0x25975])
      (0x00007F44CC6CBDF0 [libj9jit28.so+0x10fdf0])
      (0x00007F44CC6CD0ED [libj9jit28.so+0x1110ed])
      (0x00007F44D8B51975 [libj9prt28.so+0x25975])
      (0x00007F44CC6CDA04 [libj9jit28.so+0x111a04])
      (0x00007F44D8F8E9EC [libj9thr28.so+0x99ec])
      (0x00007F44DAFD0555 [libpthread.so.0+0x7555])
      clone+0x6d (0x00007F44DA8F0B9D [libc.so.6+0x102b9d])

      Attachments

        1. javacore.20150917.161241.19157.0002.txt
          2.11 MB
        2. log
          185 kB
        3. server.log
          111 kB
        4. server-openjdk.log
          80 kB
        5. Snap.20150917.161241.19157.0003.trc
          396 kB

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-1100 Reload operation causes crash of EAP server

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              tomazcerar Tomaž Cerar (Inactive)
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: