Uploaded image for project: 'Red Hat Decision Manager'
  1. Red Hat Decision Manager
  2. RHDM-71

Slow performance using business-central with GlusterFS

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: 6.x.x
    • Fix Version/s: 7.1.0.GA
    • Component/s: Decision Central
    • Labels:
      None
    • Environment:

      Openshift image based on EAP 6.4, BPMS 6.4 and and a gluster volume mounted at /opt/data

    • Target Release:
    • Fix Build:
      CR1

      Description

      Any operation in business-central, eg. trying to access the Authoring -> Project Authoring or Administration option takes minutes to finish. When temporarily changing the location -Dorg.uberfire.nio.git.dir=/opt/data/jbpm-wb-git to local file system location other than gluster volume the response times were normal.

      The gluster log shows in a 70 second period that 175971 LOOKUPs are being done for files which don't exist:

      [root ~]# grep LOOKUP /var/log/glusterfs/bricks/jbpm-data.log | cut -f 2 -d P | cut -f 2 -d ' ' | grep -v scheduled | sort | uniq -c
      		 
        23844 /jbpm-wb-git/.niogit/my-poc.git/master
      		 
        23833 /jbpm-wb-git/.niogit/my-poc.git/refs/heads/master
      		 
        23844 /jbpm-wb-git/.niogit/my-poc.git/refs/master
      		 
        23840 /jbpm-wb-git/.niogit/my-poc.git/refs/tags/master
      		 
         6312 /jbpm-wb-git/.niogit/my-poc.git/shallow
      		 
          360 /jbpm-wb-git/.niogit/system.git/locks
      		 
        16770 /jbpm-wb-git/.niogit/system.git/master
      		 
          348 /jbpm-wb-git/.niogit/system.git/refs/da68722-uf-user
      		 
          360 /jbpm-wb-git/.niogit/system.git/refs/heads/locks
      		 
        16791 /jbpm-wb-git/.niogit/system.git/refs/heads/master
      		 
          360 /jbpm-wb-git/.niogit/system.git/refs/locks
      		 
        16773 /jbpm-wb-git/.niogit/system.git/refs/master
      		 
          360 /jbpm-wb-git/.niogit/system.git/refs/remotes
      		 
          360 /jbpm-wb-git/.niogit/system.git/refs/tags/locks
      		 
        16776 /jbpm-wb-git/.niogit/system.git/refs/tags/master
      		 
         4336 /jbpm-wb-git/.niogit/system.git/shallow
      		 
      [root ~]# grep LOOKUP /var/log/glusterfs/bricks/jbpm-data.log | cut -f 2 -d P | cut -f 2 -d ' ' | grep -v scheduled | sort | wc -l
      		 
      175971

      From the Gluster side, the recommended approach to address this is to enable "negative lookup caching" to prevent recurring access to files that do not exist to go through the whole filesystem.

      The aim of this ticket is to investigate if anything can be done from the uberfire / JGit side to prevent such a huge number of lookups to happen.

        Gliffy Diagrams

          Attachments

            Issue Links

            cloned to

            Bug - A problem which impairs or prevents the functions of the product. AF-918 Slow performance of VFS with GlusterFS

            • Major - A request that should be considered seriously but is not a show stopper.
            • Open
            links to

            Web Link OpenShift Issue Report

              Activity

                People

                • Assignee:
                  ederign Eder Ignatowicz
                  Reporter:
                  mputz Martin Weiler
                  Tester:
                  Karel Suta
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  9 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: