Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-8643

[OCP 3.6] f-m-p stuck in waitUntilBuildFinished

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: FIS 2.0
    • Fix Version/s: FIS 2.0-patches
    • Component/s: FIS-Fabric8
    • Labels:
      None
    • Environment:

      openshift v3.6.92
      kubernetes v1.6.1+5115d708d7
      <fabric8.maven.plugin.version>3.1.80.redhat-000013</fabric8.maven.plugin.version>
      <fabric8.version>2.2.170.redhat-000013</fabric8.version>

    • Sprint:
      FIS 2.0 R4

      Description

      Running builds via f-m-p on current OCP 3.6 builds make f-m-p stuck waiting for the build to finish, even when the build has already finished

      "main" #1 prio=5 os_prio=0 tid=0x00007f016c00a000 nid=0x3fb3 waiting on condition [0x00007f0173372000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000794a7e8a0> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
      	at io.fabric8.maven.core.openshift.BuildService.waitUntilBuildFinished(BuildService.java:99)
      	at io.fabric8.maven.core.openshift.BuildService.waitForOpenShiftBuildToComplete(BuildService.java:84)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.executeOpenShiftBuild(BuildMojo.java:314)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.buildAndTag(BuildMojo.java:237)
      	at io.fabric8.maven.docker.BuildMojo.executeInternal(BuildMojo.java:47)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.executeInternal(BuildMojo.java:218)
      	at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:193)
      	at io.fabric8.maven.plugin.mojo.build.BuildMojo.execute(BuildMojo.java:201)
      	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:352)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:197)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
      	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
      	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
      	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
      	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
      	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
      	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
      	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
      	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
      	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
      	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
      
      

      Comparing behavior with OCP 3.5 and OCP 3.6 suggests that there may be a race in which the websocket watcher is started after the build has ended, thus missing the build completition event... (or perhaps the new ocp version does not allow that many connections at once, thus blocking the watcher websocket connection until the build log watcher ends?)

      OCP 3.6:

      [INFO] F8: Starting Build spring-boot-camel-amq-s2i
      [INFO] F8: Waiting for build spring-boot-camel-amq-s2i-1 to complete...
      [INFO] F8: Receiving source from STDIN as archive ...
      [INFO] F8: ==================================================================
      [INFO] F8: Starting S2I Java Build .....
      [INFO] F8: S2I binary build from fabric8-maven-plugin detected
      [INFO] F8: Copying binaries from /tmp/src/maven to /deployments ...
      [INFO] F8: ... done
      [INFO] F8: 
      [INFO] F8: Pushing image 172.30.158.237:5000/maschmid/spring-boot-camel-amq:latest ...
      [INFO] F8: Pushed 5/6 layers, 84% complete
      [INFO] F8: Pushed 6/6 layers, 100% complete
      [INFO] F8: Push successful
      [DEBUG] WebSocket successfully opened
      

      ... and hangs

      OCP 3.5:

      [INFO] F8: Starting Build spring-boot-camel-amq-s2i
      [INFO] F8: Waiting for build spring-boot-camel-amq-s2i-1 to complete...
      [INFO] F8: Receiving source from STDIN as archive ...
      [DEBUG] WebSocket successfully opened
      [INFO] F8: ==================================================================
      [INFO] F8: Starting S2I Java Build .....
      [INFO] F8: S2I binary build from fabric8-maven-plugin detected
      [INFO] F8: Copying binaries from /tmp/src/maven to /deployments ...
      [INFO] F8: ... done
      [INFO] F8: 
      [INFO] F8: Pushing image 172.30.45.210:5000/maschmid/spring-boot-camel-amq:latest ...
      [INFO] F8: Pushed 4/6 layers, 67% complete
      [INFO] F8: Pushed 5/6 layers, 97% complete
      [INFO] F8: Pushed 6/6 layers, 100% complete
      [INFO] F8: Push successful
      [INFO] F8: Build spring-boot-camel-amq-s2i-1 Complete
      [INFO] F8: Found tag on ImageStream spring-boot-camel-amq tag: sha256:8a665c4d0c0c77318b3051d8f861de26c8ee16dbba1fa3c0372ff63023d845d5
      [INFO] F8: Imagestream spring-boot-camel-amq written to /home/maschmid/devel/xpaas-qe/master/test-fuse/tmp/spring-boot-camel-amq1415973405485968931/spring-boot-camel-amq/target/spring-boot-camel-amq-is.yml
      [INFO] 
      [INFO] --- maven-install-plugin:2.4:install (default-install) @ spring-boot-camel-amq ---
      ...
      

      ..and continues...

        Gliffy Diagrams

          Attachments

          1. jstack.txt
            24 kB
          2. log-3.5.txt
            446 kB
          3. log-3.6.txt
            434 kB

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                maschmid Marek Schmidt
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: