Application Server 3  4  5 and 6
  1. Application Server 3 4 5 and 6
  2. JBAS-3986

Active Flushing: OutOfMemory exception due to TimedCachePolicy in JAAS

    Details

    • Type: Bug Bug
    • Status: Closed Closed (View Workflow)
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: JBossAS-4.0.4.GA, JBossAS-4.0.5.GA
    • Fix Version/s: 6.0.0.M1
    • Component/s: Security
    • Security Level: Public (Everyone can see)
    • Labels:
      None
    • Environment:
      JDK1.5, solarisx86, JBoss-4.0.4.GA
    • Similar Issues:
      Show 10 results 

      Description

      We have a big amount of users who perform logon to jboss.
      After 15 000 user logins we have an OutOfMemory exception.
      During profiling we see that JAASSecurityManager$DomainInfo takes almost all memory.
      We are using <attribute name="DefaultCacheTimeout">120</attribute> and <attribute name="DefaultCacheResolution">60</attribute>
      But memory is only growing. So no objects are removed from authentication cache.
      We tried to disable caching but in that case we had from time to time Authentication failure exception then did logon from multiple clients.
      After digging into source code we saw that object never removed from cache !
      Only then user do re-logon it is checked that principa is expired and removed.
      But it means that If user logged on once it will be always (!!) in cache.
      And it leads to OutOfMemory.
      We had to extend a run() method of TimedCachePolicy to do remove expired objects:

      public void run() {
      super.run();
      synchronized (entryMap) {
      Iterator iter = entryMap.entrySet().iterator();
      List<Object> removeentries = new ArrayList<Object>();
      while (iter.hasNext()) {
      Map.Entry entry = (Map.Entry) iter.next();
      TimedEntry value = (TimedEntry) entry.getValue();

      if (value.isCurrent(now) == false) {
      if(log.isDebugEnabled())

      { log.debug("destroying object:"+value); }

      value.destroy();
      removeentries.add(entry.getKey());
      }

      }
      for (Object object : removeentries) {
      if(log.isDebugEnabled())

      { log.debug("removing object from Map:"+object); }

      entryMap.remove(object);
      }
      }

      }

      Is not it will be much better to do it on original TimedCachePolicy class ?

        Issue Links

          Activity

          Hide
          Anil Saldhana
          added a comment -

          Are your users doing a web login? If yes, are they logging out? When they log out, are you invalidating the session? A session invalidation routine does flush the security cache. So if 1 user is logging in, on log out, the entry is removed. If not, the entry exists until the cache times out.

          If not, set your cache time outs to a lower value. In a high volume system, system resources are precious (Http Sessions, cache objects). You have to set the time outs to a manageable setting.

          Show
          Anil Saldhana
          added a comment - Are your users doing a web login? If yes, are they logging out? When they log out, are you invalidating the session? A session invalidation routine does flush the security cache. So if 1 user is logging in, on log out, the entry is removed. If not, the entry exists until the cache times out. If not, set your cache time outs to a lower value. In a high volume system, system resources are precious (Http Sessions, cache objects). You have to set the time outs to a manageable setting.
          Hide
          Ramil Israfilov
          added a comment -

          During tests we did correct login and logout and memory was growing. Problem is that TimedCachePolicy as described in javadoc: "This is a lazy cache policy in that objects are not checked for expiration until they are accessed."
          So it means if user A login and logout it will still remain in cache until next login.
          and another problem is that TimedCachePolicy is used by default: inside JaasSecurityManagerService.java there is a DefaultCacheObjectFactory class which instantiate TimedCachePolicy

          So for a moment in our production environment we are using my proposition by extending TimedCachePolicy's run() method.
          It is up to you to make your product better !

          Show
          Ramil Israfilov
          added a comment - During tests we did correct login and logout and memory was growing. Problem is that TimedCachePolicy as described in javadoc: "This is a lazy cache policy in that objects are not checked for expiration until they are accessed." So it means if user A login and logout it will still remain in cache until next login. and another problem is that TimedCachePolicy is used by default: inside JaasSecurityManagerService.java there is a DefaultCacheObjectFactory class which instantiate TimedCachePolicy So for a moment in our production environment we are using my proposition by extending TimedCachePolicy's run() method. It is up to you to make your product better !
          Hide
          Niklas Hellberg
          added a comment -

          Same issue with JBossAS-4.2.2.GA

          Show
          Niklas Hellberg
          added a comment - Same issue with JBossAS-4.2.2.GA
          Hide
          Marcus Moyses
          added a comment -

          Created AuthenticationCacheFlushThread to actively scan all the caches and flush expired entries.

          Show
          Marcus Moyses
          added a comment - Created AuthenticationCacheFlushThread to actively scan all the caches and flush expired entries.
          Hide
          Marcus Moyses
          added a comment -

          The period of this thread is 1 hour (3600 seconds) and it can be modified by the DefaultCacheFlushPeriod attribute of the JaasSecurityManagerService mbean. A value of 0 means that the thread will not be created.

          Show
          Marcus Moyses
          added a comment - The period of this thread is 1 hour (3600 seconds) and it can be modified by the DefaultCacheFlushPeriod attribute of the JaasSecurityManagerService mbean. A value of 0 means that the thread will not be created.

            People

            • Assignee:
              Marcus Moyses
              Reporter:
              Ramil Israfilov
            • Votes:
              5 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: