Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-17767

Error logs present when hitting too many open files

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Do
    • Major
    • None
    • 7.1.4.GA
    • Deployment Scanner
    • None
    • Hide
      • Set up the ulimit: ulimit -n 500
      • Apply provided patch to the EAP 7.1.4 (available here)
      • Start the EAP with the -Dxnio.nio.alt-queued-server=true
      • Deploy the long-running-servlet (available here)
      • (From another terminal, which should have a marginally higher ulimit than the running EAP) Exeute the load test: ab -n 1000 -c 1000 -k http://127.0.0.1:8080/long-running-servlet/HeavyProcessing?duration=2500

      After a while you'll see mentioned exception:

      07:42:19,598 WARN  [org.xnio.nio] (default Accept) XNIO008001: Socket accept failed, backing off for 250 milliseconds: java.io.IOException: Too many open files
07:42:19,848 WARN  [org.xnio.nio] (default Accept) XNIO008001: Socket accept failed, backing off for 1000 milliseconds: java.io.IOException: Too many open files
07:42:20,533 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /home/hudson/users-tmp/spriadka/JBEAP-17713/jboss-eap-7.1/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0032: Failed to list files in directory /home/hudson/users-tmp/spriadka/JBEAP-17713/jboss-eap-7.1/standalone/deployments. Check that the contents of the directory are readable.
	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1361)
	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scanDirectory(FileSystemDeploymentService.java:846)
	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:598)
	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.nio.file.FileSystemException: /home/hudson/users-tmp/spriadka/JBEAP-17713/jboss-eap-7.1/standalone/deployments: Too many open files
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
	at java.nio.file.Files.newDirectoryStream(Files.java:589)
	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1356)
	... 12 more
      Show
      Set up the ulimit: ulimit -n 500 Apply provided patch to the EAP 7.1.4 (available here ) Start the EAP with the -Dxnio.nio.alt-queued-server=true Deploy the long-running-servlet (available here ) (From another terminal, which should have a marginally higher ulimit than the running EAP) Exeute the load test: ab -n 1000 -c 1000 -k http://127.0.0.1:8080/long-running-servlet/HeavyProcessing?duration=2500 After a while you'll see mentioned exception: 07:42:19,598 WARN [org.xnio.nio] ( default Accept) XNIO008001: Socket accept failed, backing off for 250 milliseconds: java.io.IOException: Too many open files 07:42:19,848 WARN [org.xnio.nio] ( default Accept) XNIO008001: Socket accept failed, backing off for 1000 milliseconds: java.io.IOException: Too many open files 07:42:20,533 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /home/hudson/users-tmp/spriadka/JBEAP-17713/jboss-eap-7.1/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0032: Failed to list files in directory /home/hudson/users-tmp/spriadka/JBEAP-17713/jboss-eap-7.1/standalone/deployments. Check that the contents of the directory are readable. at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1361) at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scanDirectory(FileSystemDeploymentService.java:846) at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:598) at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493) at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: java.nio.file.FileSystemException: /home/hudson/users-tmp/spriadka/JBEAP-17713/jboss-eap-7.1/standalone/deployments: Too many open files at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) at java.nio.file.Files.newDirectoryStream(Files.java:589) at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1356) ... 12 more

    Description

      When investigating JBEAP-17713 we hit a java.nio.file.FileSystemException exception with too many open files, the ERROR logs are being present in the server.log. Having ERROR logs present in the log is not a desired state.

      Attachments

        Activity

          People

            rhn-cservice-bbaranow Bartosz Baranowski
            rhn-engineering-spriadka Simon Priadka (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: