Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Rejected
    • Affects Version/s: 3.6.11
    • Fix Version/s: 3.6.12
    • Labels:
      None

      Description

      We are running our JVMs with : -XX:OnOutOfMemoryError="kill -9 %p"

      we have been experiencing OOMs fairly often, and the OOMs happen at:

      Object / Stack Frame                                                              |Name                                                                                             | Shallow Heap | Retained Heap |Context Class Loader                         |Is Daemon
      ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      java.lang.Thread @ 0x81bdf838                                                     |Connection.Receiver [144.77.77.53:50363 - 144.77.77.53:50363],sis-cluster.service,prodpmwsv5-6461|          120 |           456 |sun.misc.Launcher$AppClassLoader @ 0x800175a8|false
      |- at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)             |                                                                                                 |              |               |                                             |
      |- at org.jgroups.blocks.cs.TcpConnection$Receiver.run()V (TcpConnection.java:310)|                                                                                                 |              |               |                                             |
      |- at java.lang.Thread.run()V (Thread.java:745)                                   |                                                                                                 |              |               |                                             |
      ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      

      the Code where it happens is in TcpConnection.java:

      while(canRun()) {
                      try {
                          int len=in.readInt();
                          if(buffer == null || buffer.length < len)
                              buffer=new byte[len];
                          in.readFully(buffer, 0, len);
                          updateLastAccessed();
                          server.receive(peer_addr, buffer, 0, len);
                      }
                      catch(OutOfMemoryError mem_ex) {
                          t=mem_ex;
                          break; // continue;
                      }
                      catch(IOException io_ex) {
                          t=io_ex;
                          break;
                      }
                      catch(Throwable e) {
                      }
                  }
      

      when allocating: buffer=new byte[len];

      it looks to me that some invalid large value is received and the process OOMs when allocating a huge byte array

      Running JVMs without kill on OOM would make this issue "dissapear" in the sense that it is swallowed by:

                      catch(OutOfMemoryError mem_ex) {
                          t=mem_ex;
                          break; // continue;
                      }
      

      Handling OutOfMemoryError is a strange implementation choice...
      instead a size limit should be employed to protect from receiving invalid sizes...

      My heap limit is 1GB and my heap dumps are 50Mb so the attempted allocation size is huge...

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                belaban Bela Ban
                Reporter:
                zolyfarkas Zoltan Farkas
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: