Loading...

Details

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: None
Affects Version/s: 1.1.0.CR1
Component/s: None
Labels:

Steps to Reproduce:
Hide

(non multicast environment - manual proxy registration required)

1. Start all servers.
2. Register servers with proxy, by invoking addProxy httpd-server-address, 6666 on each backend server.
3. Open up a session via http (e.g. jmx-console or node-info) from a client
4. Determine which backend server is being used (e.g. via inspecting cookie on client with firebug)
5. kill -9 jboss process on active node.
6. Refresh on client. 503 error.
7. From a different client / or clear session & cookies, visit URL

Configuration
------------------
server.xml:
<Listener className="org.jboss.mocluster.catalina.ModClusterListener"
advertise="false"
stopContextTimeout="5"
socketTimeout="5000"
maxAttempts="10"
stickySessionsForce="false"
balancer="loadbalancer"
workerTimeout="30"
nodeTimeout="10"
/>

mod_cluster-jboss-beans.xml:
<bean name="ModClusterListener" class="org.jboss.mocluster.catalina.CatalinaEventHandlerAdapter" mode="On Demand">
<constructor>
<parameter><inject bean="HAModClusterService"/></parameter>
<parameter><inject bean="JMXKernel" property="mbeanServer"/></parameter>
</constructor>
</bean>

all other config is default.

httpd config (excluding LoadModule directives. mod_advertise is not loaded).
CreateBalancers 0
ManagerBalancerName loadbalancer

Have tried with this enabled: ProxyTimeout 10
<Location /mod_cluster_manger>
SetHandler mod_cluster-manager
</Location>

Listen 6666
<VirtualHost *:6666>
KeepAliveTimeout 60
MaxKeepAliveRequests 0
ManagerBalancerName loadbalancer
</VirtualHost>

Dump on mod_cluster-manager before node is killed (Timeout: 5 minutes?? - this is true even when nodeTimeout is set on the Listener to send to http and when configured on httpd via ProxyTimeout):
balancer: [1] Name: loadbalancer Sticky: 1 [JSESSIONID]/[jsessionid] remove: 1 force: 0 Timeout: 30000000 Maxtry: 10
node: [1:1],Balancer: loadbalancer,JVMRoute: dev-test-01,Domain: [],Host: 192.168.73.21,Port: 8009,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 1,ttl: 60,timeout: 10
node: [2:2],Balancer: loadbalancer,JVMRoute: dev-test-02,Domain: [],Host: 192.168.75.22,Port: 8009,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 1,ttl: 60,timeout: 10
host: 1 [localhost] vhost: 1 node: 1
host: 2 [localhost] vhost: 1 node: 2
context: 1 [/jmx-console] vhost: 1 node: 1 status: 1
context: 2 [/node-info] vhost: 1 node: 1 status: 1
context: 3 [/jmx-console] vhost: 1 node: 2 status: 1
context: 4 [/node-info] vhost: 1 node: 2 status: 1

backend1 # kill -9 <pid of JBoss>

The mod_cluster-manager handler shows:
Balancer: loadbalancer,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Status: NOTOK,Elected: 3,Read: 846,Transferred: 0,Connected: 0,Load: 100

INFO output shows:
Node: [1],Name: dev-test-01,Balancer: loadbalancer,Domain: ,Host: 192.168.73.21,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 3,Read: 846,Transfered: 0,Connected: 0,Load: 100
Node: [2],Name: dev-test-02,Balancer: loadbalancer,Domain: ,Host: 192.168.75.22,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: -1
Vhost: [1:1:1], Alias: localhost
Vhost: [2:1:2], Alias: localhost
Context: [1:1:1], Context: /jmx-console, Status: ENABLED
Context: [1:1:2], Context: /node-info, Status: ENABLED
Context: [2:1:3], Context: /jmx-console, Status: ENABLED
Context: [2:1:4], Context: /node-info, Status: ENABLED
Show
(non multicast environment - manual proxy registration required) 1. Start all servers. 2. Register servers with proxy, by invoking addProxy httpd-server-address, 6666 on each backend server. 3. Open up a session via http (e.g. jmx-console or node-info) from a client 4. Determine which backend server is being used (e.g. via inspecting cookie on client with firebug) 5. kill -9 jboss process on active node. 6. Refresh on client. 503 error. 7. From a different client / or clear session & cookies, visit URL Configuration ------------------ server.xml: <Listener className="org.jboss.mocluster.catalina.ModClusterListener" advertise="false" stopContextTimeout="5" socketTimeout="5000" maxAttempts="10" stickySessionsForce="false" balancer="loadbalancer" workerTimeout="30" nodeTimeout="10" /> mod_cluster-jboss-beans.xml: <bean name="ModClusterListener" class="org.jboss.mocluster.catalina.CatalinaEventHandlerAdapter" mode="On Demand"> <constructor> <parameter><inject bean="HAModClusterService"/></parameter> <parameter><inject bean="JMXKernel" property="mbeanServer"/></parameter> </constructor> </bean> all other config is default. httpd config (excluding LoadModule directives. mod_advertise is not loaded). CreateBalancers 0 ManagerBalancerName loadbalancer Have tried with this enabled: ProxyTimeout 10 <Location /mod_cluster_manger> SetHandler mod_cluster-manager </Location> Listen 6666 <VirtualHost *:6666> KeepAliveTimeout 60 MaxKeepAliveRequests 0 ManagerBalancerName loadbalancer </VirtualHost> Dump on mod_cluster-manager before node is killed (Timeout: 5 minutes?? - this is true even when nodeTimeout is set on the Listener to send to http and when configured on httpd via ProxyTimeout): balancer: [1] Name: loadbalancer Sticky: 1 [JSESSIONID] / [jsessionid] remove: 1 force: 0 Timeout: 30000000 Maxtry: 10 node: [1:1] ,Balancer: loadbalancer,JVMRoute: dev-test-01,Domain: [],Host: 192.168.73.21,Port: 8009,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 1,ttl: 60,timeout: 10 node: [2:2] ,Balancer: loadbalancer,JVMRoute: dev-test-02,Domain: [],Host: 192.168.75.22,Port: 8009,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 1,ttl: 60,timeout: 10 host: 1 [localhost] vhost: 1 node: 1 host: 2 [localhost] vhost: 1 node: 2 context: 1 [/jmx-console] vhost: 1 node: 1 status: 1 context: 2 [/node-info] vhost: 1 node: 1 status: 1 context: 3 [/jmx-console] vhost: 1 node: 2 status: 1 context: 4 [/node-info] vhost: 1 node: 2 status: 1 backend1 # kill -9 <pid of JBoss> The mod_cluster-manager handler shows: Balancer: loadbalancer,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Status: NOTOK,Elected: 3,Read: 846,Transferred: 0,Connected: 0,Load: 100 INFO output shows: Node: [1] ,Name: dev-test-01,Balancer: loadbalancer,Domain: ,Host: 192.168.73.21,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 3,Read: 846,Transfered: 0,Connected: 0,Load: 100 Node: [2] ,Name: dev-test-02,Balancer: loadbalancer,Domain: ,Host: 192.168.75.22,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: -1 Vhost: [1:1:1] , Alias: localhost Vhost: [2:1:2] , Alias: localhost Context: [1:1:1] , Context: /jmx-console, Status: ENABLED Context: [1:1:2] , Context: /node-info, Status: ENABLED Context: [2:1:3] , Context: /jmx-console, Status: ENABLED Context: [2:1:4] , Context: /node-info, Status: ENABLED
Forum Reference:
http://community.jboss.org/thread/151093?tstart=0
Affects:

Compatibility/Configuration

Description

If 1 backend AS process is killed, the frontend proxy still connects clients to the dead node. The node is removed from mod_proxy after 5 minutes. All requests are directed to the dead node.

mod-cluster manager status shows the node as status=NOTOK.
client connections to this node result in 503 errors.
After backend is removed from configuration after 5 minutes, requests to httpd still result in 503 errors as request is routed to the unavailable node. Even new sessions from different hosts try to get routed to the dead node, resulting in 503 errors.

The note on www.jboss.org/mod_cluster/java/properties.html:
When nodeTimeout is not defined, the ProxyTimeout directive Proxy is used. If ProxyTimeout is not defined the server timeout (Timeout) is used (default 300 seconds). nodeTimeout, ProxyTimeout or Timeout is set at the socket level.

As per the configuration below, nodeTimeout is set in the Listener configuration. In addition to that, ProxyTimeout has been enabled and disabled on the httpd server. Timeout is configured with default of 120.

mod_proxy timeout setting is not set correctly from nodeTimeout (JBoss) or ProxyTimeout (httpd) configuration in mod_cluster

Details

Description

Attachments

Activity

People

Dates