Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-3907

Deployment takes a lot of time in latest patched Fabric environment with ~70 profile.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • jboss-fuse-6.2
    • jboss-fuse-6.1
    • Fabric8 v1
    • None
    • % %

    Description

      • Customer has ~70 profiles, updating a profile and thus applying it to fabric takes around 10 to 15 minutes for deployment. CPU usage also spikes to 90 to 100 % during deployment. 
        "pool-367-thread-1" prio=6 tid=0x000000000c2d9800 nid=0x3198 runnable [0x0000000014f2d000]
   java.lang.Thread.State: RUNNABLE
	at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:164)
	- locked <0x00000000c3747670> (a org.eclipse.jgit.internal.storage.file.PackFile)
	at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:255)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject1(ObjectDirectory.java:371)
	at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObjectImpl1(FileObjectDatabase.java:173)
	at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObject(FileObjectDatabase.java:158)
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
	at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:853)
	at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:143)
	at org.eclipse.jgit.revwalk.RewriteTreeFilter.include(RewriteTreeFilter.java:109)
	at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:141)
	at org.eclipse.jgit.revwalk.BlockRevQueue.<init>(BlockRevQueue.java:65)
	at org.eclipse.jgit.revwalk.FIFORevQueue.<init>(FIFORevQueue.java:64)
	at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:157)
	at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:414)
	at org.gitective.core.CommitUtils.getLastCommit(CommitUtils.java:418)
	at io.fabric8.git.internal.CachingGitDataStore.addProfileData(CachingGitDataStore.java:162)
	at io.fabric8.git.internal.CachingGitDataStore.addProfileData(CachingGitDataStore.java:152)
	at io.fabric8.git.internal.CachingGitDataStore.addProfileData(CachingGitDataStore.java:152)
	at io.fabric8.git.internal.CachingGitDataStore.pouplateVersionData(CachingGitDataStore.java:132)
	at io.fabric8.git.internal.CachingGitDataStore$1$1.call(CachingGitDataStore.java:91)
	at io.fabric8.git.internal.CachingGitDataStore$1$1.call(CachingGitDataStore.java:88)
	at io.fabric8.git.internal.GitDataStore.gitOperation(GitDataStore.java:934)
	- locked <0x00000000c3469200> (a java.lang.Object)
	at io.fabric8.git.internal.GitDataStore.gitOperation(GitDataStore.java:901)
	at io.fabric8.git.internal.GitDataStore.gitOperation(GitDataStore.java:896)
	at io.fabric8.git.internal.CachingGitDataStore$1.load(CachingGitDataStore.java:88)
	at io.fabric8.git.internal.CachingGitDataStore$1.load(CachingGitDataStore.java:85)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
	- locked <0x00000000c346f9e0> (a com.google.common.cache.LocalCache$StrongEntry)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
	at io.fabric8.git.internal.CachingGitDataStore.getVersionData(CachingGitDataStore.java:116)
	at io.fabric8.git.internal.CachingGitDataStore.hasProfile(CachingGitDataStore.java:187)
	at io.fabric8.internal.VersionImpl.getProfile(VersionImpl.java:100)
	at io.fabric8.internal.ContainerImpl$ContainerProfile.getParents(ContainerImpl.java:271)
	at io.fabric8.internal.ProfileOverlayImpl.fillParentProfiles(ProfileOverlayImpl.java:248)
	at io.fabric8.internal.ProfileOverlayImpl.getInheritedProfiles(ProfileOverlayImpl.java:243)
	at io.fabric8.internal.ProfileOverlayImpl.getConfigurations(ProfileOverlayImpl.java:346)
	at io.fabric8.internal.ProfileOverlayImpl.getContainerConfiguration(ProfileOverlayImpl.java:121)
	at io.fabric8.internal.ProfileImpl.getContainerConfigList(ProfileImpl.java:149)
	at io.fabric8.internal.ProfileOverlayImpl.getRepositories(ProfileOverlayImpl.java:106)
	at io.fabric8.features.FabricFeaturesServiceImpl.listInstalledRepositories(FabricFeaturesServiceImpl.java:327)
	at io.fabric8.features.FabricFeaturesServiceImpl.run(FabricFeaturesServiceImpl.java:93)
	- locked <0x00000000c3202c18> (a io.fabric8.features.FabricFeaturesServiceImpl)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- <0x00000000c3202e80> (a java.util.concurrent.ThreadPoolExecutor$Worker)


"pool-365-thread-1" prio=6 tid=0x000000000c2ed000 nid=0x2960 waiting for monitor entry [0x00000000123cf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at io.fabric8.git.internal.GitDataStore.gitOperation(GitDataStore.java:906)
	- waiting to lock <0x00000000c3469200> (a java.lang.Object)
	at io.fabric8.git.internal.GitDataStore.gitOperation(GitDataStore.java:901)
	at io.fabric8.git.internal.GitDataStore.gitOperation(GitDataStore.java:891)
	at io.fabric8.git.internal.GitDataStore.pull(GitDataStore.java:1251)
	at io.fabric8.git.internal.GitDataStore$1.run(GitDataStore.java:186)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
	- <0x00000000c3468a90> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      • As described in https://issues.jboss.org/browse/ENTESB-2581, this issue should be fixed with R2P1 and a property "gitGcOnLoad=true" within root container as described in ENTESB-2581.
      • But User is with latest Patch R2P6 and still see the delay in deployment with above thread dump.
      • User perform deployment in child-container.
      • System Details: 
        Processor: Intel Xeon X5675 3.75 Ghz(2 Processors)
RAM: 16 GB
System Type: 64 bit OS
OS: Windows Server 2008 R2 Enterprise SP1

      Attachments

        Issue Links

          duplicates

          Bug - A problem that impairs or prevents the functions of the product. ENTESB-2581 Git repo very slow after adding 100s of profiles

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              janstey@redhat.com Jonathan Anstey
              rhn-support-cpandey Chandra Shekhar Pandey (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: