Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-2301

Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852]

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 3.0.0.Beta3, 2.2.1.CR2
    • Component/s: Management
    • Labels:
      None
    • Environment:
    • Steps to Reproduce:
      Hide
      • start or configure WildFly in a Docker environment with overlayfs
      • invoke backup of config, e.g., by using the cli during build or start script, e.g., /opt/jboss/bin/jboss-cli.sh --file=/opt/jboss/jboss-config.cli
      Show
      start or configure WildFly in a Docker environment with overlayfs invoke backup of config, e.g., by using the cli during build or start script, e.g., /opt/jboss/bin/jboss-cli.sh --file=/opt/jboss/jboss-config.cli

      Description

      Mount point not found exception raised by createTempFileWithAttributes on overlayfs [JDK-8165852], i.e.,

      /opt/jboss/bin/jboss-cli.sh --file=/opt/jboss/jboss-config.cli

      inside a Docker container running on overlayfs as storage driver

      causes (full stack trace below):

      java.io.IOException: Mount point not foundImage
          at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
      

      triggered by

          at org.jboss.as.controller.persistence.FilePersistenceUtils.createTempFileWithAttributes(FilePersistenceUtils.java:117)
          at org.jboss.as.controller.persistence.FilePersistenceUtils.writeToTempFile(FilePersistenceUtils.java:104)
      

      due to OpenJDK bug/overlayfs bug.

      We acknowledge that this is in fact an OpenJDK AND/OR overlayfs bug. However everything seems to run fine in WildFly except once the backup of the config is triggered, for example by using the cli. Hence WildFly is of limited functionality when the more and more popular overlayfs storage driver is used, and the WildFly team might be interested in providing a workaround on their side since there is no indication the OpenJDK bug will be promptly fixed. We are happy to help in any way, we are still trying to find a workaround on the Java or WildFly side; but we might need insights on why findMountEntry is invoked.

      Full Stack Trace:

       java.io.IOException: Mount point not foundImage
               at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
               at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65)
               at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44)
        at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51)
               at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39)
        at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368)
               at java.nio.file.Files.getFileStore(Files.java:1461)
               at org.jboss.as.controller.persistence.FilePersistenceUtils.getPosixAttributes(FilePersistenceUtils.java:129)
               at org.jboss.as.controller.persistence.FilePersistenceUtils.createTempFileWithAttributes(FilePersistenceUtils.java:117)
               at org.jboss.as.controller.persistence.FilePersistenceUtils.writeToTempFile(FilePersistenceUtils.java:104)
               at org.jboss.as.controller.persistence.ConfigurationFilePersistenceResource.doCommit(ConfigurationFilePersistenceResource.java:55)
               at org.jboss.as.controller.persistence.AbstractFilePersistenceResource.commit(AbstractFilePersistenceResource.java:58)
               at org.jboss.as.controller.ModelControllerImpl$4.commit(ModelControllerImpl.java:781)
               at org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:743)
               at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:680)
               at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204)
        at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:659)
        at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:649)
        at org.jboss.as.controller.client.helpers.DelegatingModelControllerClient.execute(DelegatingModelControllerClient.java:63)
        at org.jboss.as.cli.embedded.ThreadContextsModelControllerClient.execute(ThreadContextsModelControllerClient.java:59)
        at org.jboss.as.cli.handlers.batch.BatchRunHandler.doHandle(BatchRunHandler.java:91)
        at org.jboss.as.cli.handlers.CommandHandlerWithHelp.handle(CommandHandlerWithHelp.java:88)
        at org.jboss.as.cli.impl.CommandContextImpl.handle(CommandContextImpl.java:776)
        at org.jboss.as.cli.impl.CommandContextImpl.handleSafe(CommandContextImpl.java:799)
        at org.jboss.as.cli.impl.CliLauncher.processFile(CliLauncher.java:334)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:262)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jboss.modules.Module.run(Module.java:329)
        at org.jboss.modules.Main.main(Main.java:507)
      
      

      Java Bug Overview:

      https://bugs.openjdk.java.net/browse/JDK-8165852

      http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/fs/LinuxFileStore.java#91

      Wildfly Stack Overflow issue, not solved:

      https://stackoverflow.com/questions/41022393/mount-point-not-found

      Background Info:

      http://mail.openjdk.java.net/pipermail/nio-dev/2016-October/003915.html

      A) chroot environment [1]
      B) Docker container with overlay and overlay2 storage drivers [2]
      C) btrfs file system with an unmounted sub-volume [2]

      [1] https://bugs.openjdk.java.net/browse/JDK-8165323 - cannot get FileStore in chroot environment
      [2] https://bugs.openjdk.java.net/browse/JDK-8165852 - cannot get FileStore for a file in overlayfs in Docker

      Docker file system/storage driver:

      https://docs.docker.com/engine/userguide/storagedriver/selectadriver/)

      Yum yum-plugin-ovl, similar problem:

      https://github.com/CentOS/sig-cloud-instance-images/issues/15

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  brian.stansberry Brian Stansberry
                  Reporter:
                  bjoernstuetztss Bjoern Stuetz
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: