Details
-
Bug
-
Resolution: Done
-
Major
-
jboss-fuse-6.2
-
None
-
%
-
-
Description
In an installation where:
- there are many containers, > 400,
- alternating live and stage profile versions [see below for more detail]
- more than one fabric-git version in play
- 1.2.0.redhat-161
- 1.2.0.redhat-165
It is often seen in the logs:
2016-04-21 10:59:18,877 | INFO | pool-36-thread-1 | io.fabric8.git.internal.DefaultPullPushPolicy | 69 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | - - | Performing a pull on remote URL: http://d154.mydomain.org:1234/git/fabric/ 2016-04-21 10:59:26,199 | WARN | pool-36-thread-1 | io.fabric8.git.internal.DefaultPullPushPolicy | 69 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | - - | Pull failed because of: org.eclipse.jgit.api.errors.TransportException: Read timed out after 5,000 ms 2016-04-21 10:59:26,858 | INFO | pool-36-thread-1 | io.fabric8.git.internal.DefaultPullPushPolicy | 69 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | - - | Performing a pull on remote URL: http://d154.mydomain.org:1234/git/fabric/ 2016-04-21 10:59:33,782 | WARN | pool-36-thread-1 | io.fabric8.git.internal.DefaultPullPushPolicy | 69 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | - - | Pull failed because of: org.eclipse.jgit.api.errors.TransportException: Read timed out after 5,000 ms 2016-04-21 10:59:46,963 | INFO | pool-36-thread-1 | io.fabric8.git.internal.DefaultPullPushPolicy | 69 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | - - | Performing a pull on remote URL: http:/XXXX:1234/git/fabric/
and infrequently seen:
run-em_p10.log:java.lang.IllegalStateException: Push rejected: [RemoteRefUpdate[remoteName=refs/heads/60.2, REJECTED_NONFASTFORWARD, (null)...bec7f3e3383f43c61d6f7f4ff1029ae2e5fa08a5, srcRef=refs/heads/60.2, message=null], RemoteRefUpdate[
remoteName=refs/heads/50.2, REJECTED_NONFASTFORWARD, (null)...592df1b0ea7a98e34aa22311e9ed10c67ef7fd4c, srcRef=refs/heads/50.2, message=null]]
run-em_p12.log:2016-05-10 07:56:38,168 | WARN | qtp1672381131-66692 | io.fabric8.git.internal.DefaultPullPushPolicy | 79 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | Rejected push: {}RemoteRefUpdate[remoteName=refs/heads/60.1, REJ
ECTED_NONFASTFORWARD, (null)...f80d25ef779b93e3ab7f616da372a715e6403364, srcRef=refs/heads/60.1, message=null]
or
2016-05-13 12:16:15,720 | INFO | Thread-61794 | io.fabric8.git.internal.DefaultPullPushPolicy | 79 - io.fabric8.fabric-git - 1.2.0.redhat-161 | - - | Push result: [accepted=25,rejected=1,error=null]
Where the timeout had previously been increased to 15,000ms with no change in behaviour and the version strategy is to have 2 versions for each of the regions so that regions can be moved forward at their own pace and are not tied to another region. Then it alternates between the versions having 1 be a live version and the other a stage version for the next go-live update:
i.e.
10.1 - 10.2
11.1 - 11.2
20.1 - 20.2
30.1 - 30.2
40.1 - 40.2
50.1 - 50.2
60.1 - 60.2
70.1 - 70.2
As well rarely it has been seen:
2016-05-17 14:10:23,912 | WARN | pool-22-thread-1 | io.fabric8.service.ZkDataStoreImpl | 67 - io.fabric8.fabric-core - 1.2.0.redhat-153 | - - | - - | Caught: java.lang.IllegalStateException: Cannot obtain profile read lock in time java.lang.IllegalStateException: Cannot obtain profile read lock in time at io.fabric8.api.gravia.IllegalStateAssertion.assertTrue(IllegalStateAssertion.java:55)[66:io.fabric8.fabric-api:1.2.0.redhat-133] at io.fabric8.git.internal.GitDataStoreImpl.aquireReadLock(GitDataStoreImpl.java:434)[69:io.fabric8.fabric-git:1.2.0.redhat-161] at io.fabric8.git.internal.GitDataStoreImpl.getVersionFromCacheRO(GitDataStoreImpl.java:485)[69:io.fabric8.fabric-git:1.2.0.redhat-161] at io.fabric8.git.internal.GitDataStoreImpl.getVersionFromCache(GitDataStoreImpl.java:477)[69:io.fabric8.fabric-git:1.2.0.redhat-161] at io.fabric8.git.internal.GitDataStoreImpl.getVersion(GitDataStoreImpl.java:642)[69:io.fabric8.fabric-git:1.2.0.redhat-161] at io.fabric8.internal.ProfileServiceImpl.getVersion(ProfileServiceImpl.java:126)[67:io.fabric8.fabric-core:1.2.0.redhat-153] at io.fabric8.internal.ContainerImpl.getVersion(ContainerImpl.java:203)[67:io.fabric8.fabric-core:1.2.0.redhat-153] at io.fabric8.internal.ContainerImpl.getContainerProfile(ContainerImpl.java:762)[67:io.fabric8.fabric-core:1.2.0.redhat-153] at io.fabric8.internal.ContainerImpl.getOverlayProfile(ContainerImpl.java:306)[67:io.fabric8.fabric-core:1.2.0.redhat-153]
The end resut of which is that the Fabric Git repository is "reverting" to previous values after updating profiles.
Attachments
Issue Links
- duplicates
-
-
- Closed
-
- is related to
-
ENTESB-5654 Emergency command fabric:git-synchronize to trigger git pull across ensemble containers
-
- Closed
-
