Uploaded image for project: 'mod_cluster'
  1. mod_cluster
  2. MODCLUSTER-570

Multiple invalid nodes created on graceful httpd restart

    XMLWordPrintable

Details

    • Hide

      Start httpd with mod_cluster modules enabled.

      Send httpd a SIGUSR1:

      $ kill -SIGUSR1 $(pgrep httpd | head -1)
      
      Show
      Start httpd with mod_cluster modules enabled. Send httpd a SIGUSR1: $ kill -SIGUSR1 $(pgrep httpd | head -1)

    Description

      mod_cluster creates a bunch of invalid nodes after a graceful restart of Apache httpd (SIGUSR1, SIGWINCH). The issue happens with or without backend servers in the pool.

      httpd logs:

      [mpm_event:notice] [pid 1:tid 140313388616520] AH00493: SIGUSR1 received.  Doing graceful restart
      [:notice] [pid 1:tid 140313388616520] Advertise initialized for process 1
      [mpm_event:notice] [pid 1:tid 140313388616520] AH00489: Apache/2.4.25 (Unix) mod_cluster/1.3.5.Final configured -- resuming normal operations
      [core:notice] [pid 1:tid 140313388616520] AH00094: Command line: 'httpd -D FOREGROUND'
      [:notice] [pid 116:tid 140313388616520] Created: worker for ://: failed: Unable to parse URL
      [:notice] [pid 116:tid 140313388616520] Created: worker for ://: failed: Unable to parse URL
      [:notice] [pid 116:tid 140313388616520] Created: worker for ://: failed: Unable to parse URL
      # ... trimmed 48 lines ...
      [:notice] [pid 145:tid 140313388616520] Created: worker for ://: failed: Unable to parse URL
      [:notice] [pid 145:tid 140313388616520] Created: worker for ://: failed: Unable to parse URL
      [:notice] [pid 145:tid 140313388616520] Created: worker for ://: failed: Unable to parse URL
      [:notice] [pid 145:tid 140313388616520] Created: worker for ://:\x9d\x7f failed: Unable to parse URL
      [:notice] [pid 145:tid 140313388616520] add_balancer_node: balancer safe-name (balancer://4:6666\r\nX-Manager-Url: /c8dae916-7642-4deb-b841-2f3435613749\r\nX-Manager-Protocol: http\r\nX-Manager-Host: 172.17.0.4\r\n\r\n) too long
      

      Application server logs (tomcat8):

      Feb 01, 2017 11:34:25 AM org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler sendRequest
      ERROR: MODCLUSTER000042: Error MEM sending STATUS command to 172.17.0.4/172.17.0.4:6666, configuration will be reset: MEM: Can't read node with "java1-depl1-1234" JVMRoute
      Feb 01, 2017 11:34:35 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
      SEVERE: Exception invoking periodic operation: 
      java.lang.IllegalArgumentException: Node: [1],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait: 0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
      Node: [2],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait: 0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
      Node: [3],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait: 0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
      # ... trimmed lines ...
      Node: [18],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait: 0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
      Node: [19],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait: 0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
      Vhost: [0:0:1], Alias: 
      Vhost: [0:0:2], Alias: 
      Vhost: [0:0:3], Alias: 
      # ... trimmed lines ...
      Vhost: [0:0:19], Alias: 
      Vhost: [0:0:20], Alias: 
      Context: [0:0:1], Context: , Status: REMOVED
      Context: [0:0:2], Context: , Status: REMOVED
      Context: [0:0:3], Context: , Status: REMOVED
      # ... trimmed lines ...
      Context: [0:0:97], Context: , Status: REMOVED
      Context: [0:0:98], Context: , Status: REMOVED
      Context: [0:0:99], Context: , Status: REMOVED
      Context: [0:0:1], Context: , Status: REMOVED
      
      	at org.jboss.modcluster.mcmp.impl.DefaultMCMPResponseParser.parseInfoResponse(DefaultMCMPResponseParser.java:96)
      	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:396)
      	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
      	at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:457)
      	at org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:252)
      	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
      	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
      	at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1374)
      	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1546)
      	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1524)
      	at java.lang.Thread.run(Thread.java:745)
      
      Feb 01, 2017 11:35:05 AM org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler sendRequest
      

      Attachments

        Activity

          People

            rhn-engineering-jclere Jean-Frederic Clere
            antoineco Antoine Cotten (Inactive)
            Athanasios Ploumis Athanasios Ploumis (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: