Uploaded image for project: 'JBoss Data Grid'
  1. JBoss Data Grid
  2. JDG-1002

"org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: JDG 7.0.0 GA, JDG 7.1.0 GA
    • Fix Version/s: JDG 7.0.1
    • Component/s: Querying
    • Labels:
      None
    • Target Release:
    • Fix Build:
      DR2
    • Steps to Reproduce:
      Hide

      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven
      2. Deploy it to a standalone EAP 6.4 server.
      3. Run the following requests concurrently.

      http://localhost:8080/jdg-test/jdg4
      		 
      http://localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895
      		 
      http://localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88

      jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more.

      Siege is available on Fedora (dnf install siege) and can be used by the following command. "urls.txt" has the above contents.

      siege -c 40 -b -f urls.txt

      Show
      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven 2. Deploy it to a standalone EAP 6.4 server. 3. Run the following requests concurrently. http://localhost:8080/jdg-test/jdg4 http://localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895 http://localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88 jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more. Siege is available on Fedora ( dnf install siege ) and can be used by the following command. "urls.txt" has the above contents. siege -c 40 -b -f urls.txt
    • Git Pull Request:

      Description

      A customer is using a spatial query in a library mode JDG web application. The query sometimes fails with the following exceptions under high load.

      18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-3) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index: 181
      		 
              at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:158) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
      		 
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
      		 
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
      		 
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
      		 
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
      		 
       
      		 
      18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-5) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: java.lang.ArrayIndexOutOfBoundsException: 100
      		 
              at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:172) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
      		 
              at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
      		 
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
      		 
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
      		 
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
      		 
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
      		 
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]

      This is reproducible even on a single server. It seems the error happens if there are concurrent multiple queries with different query parameters. But all objects around LuceneHSQuery.queryEntityInfos() are local to the thread and doesn't look like a thread safety issue at a glance.

      Cache configuration is REPL_ASYNC mode, "ram" directory provider with "near-reql-time" index manager and "async" worker.

        Gliffy Diagrams

          Attachments

            Issue Links

            cloned to

            Bug - A problem which impairs or prevents the functions of the product. JDG-1127 "org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

            • Major - A request that should be considered seriously but is not a show stopper.
            • Verified
            relates to

            Bug - A problem which impairs or prevents the functions of the product. ISPN-7885 "org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

            • Major - A request that should be considered seriously but is not a show stopper.
            • Open
            is caused by

            HSEARCH-2691 Loading...

              Activity

                People

                • Assignee:
                  gustavonalle Gustavo Fernandes
                  Reporter:
                  osamu.nagano Osamu Nagano
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: