### Eclipse Workspace Patch 1.0 #P docs-cluster_guide Index: en-US/Clustering_Guide_Building_Blocks.xml =================================================================== --- en-US/Clustering_Guide_Building_Blocks.xml (revision 90941) +++ en-US/Clustering_Guide_Building_Blocks.xml (working copy) @@ -52,16 +52,205 @@
Group Communication with JGroups - - -
- The JGroups Shared Transport - -
+ JGroups provides the underlying group communication support for + JBoss AS clusters. Services deployed on JBoss AS which need group + communication with their peers will obtain a JGroups Channel + and use it to communicate. The Channel handles such + tasks as managing which nodes are members of the group, detecting node failures, + ensuring lossless, first-in-first-out delivery of messages to all group members, + and providing flow control to ensure fast message senders cannot overwhelm + slow message receivers. + + + The characteristics of a JGroups Channel are determined + by the set of protocols that compose it. Each protocol + handles a single aspect of the overall group communication task; for example + the UDP protocol handles the details of sending and + receiving UDP datagrams. A Channel that uses the + UDP protocol is capable of communicating with UDP unicast and + multicast; alternatively one that uses the TCP protocol + uses TCP unicast for all messages. JGroups supports a wide variety of different + protocols (see for details), but + the AS ships with a default set of channel configurations that should meet + most needs. + + + By default, UDP multicast is used by all JGroups channels used by + the AS (except for one TCP-based channel used by JBoss Messaging).
The Channel Factory Service - + A significant difference in JBoss AS 5 versus previous releases + is that JGroups Channels needed by clustering services (e.g. a channel + used by a distributed HttpSession cache) are no longer configured in + detail as part of the consuming service's configuration, and are no longer + directly instantiated by the consuming service. Instead, a new + ChannelFactory service is used as a registry for named + channel configurations and as a factory for Channel instances. + A service that needs a channel requests the channel from the ChannelFactory, + passing in the name of the desired configuration. + + The ChannelFactory service is deployed in the + server/all/deploy/cluster/jgroups-channelfactory.sar. + On startup the ChannelFactory service parses the + server/all/deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml + file, which includes various standard JGroups configurations identified + by name (e.g "udp" or "tcp"). Services needing a channel access the channel + factory and ask for a channel with a particular named configuration. + + +
+ Standard Protocol Stack Configurations + The standard protocol stack configurations that ship with AS 5 + are described below. Note that not all of these are actually used; + many are included as a convenience to users who may wish to alter the + default server configuration. The configurations actually used in a + stock AS 5 all config are udp, + jbm-control and jbm-data, with + all clustering services other than JBoss Messaging using udp. + + + + + udp + UDP multicast based stack meant to be shared between different channels. + Message bundling is disabled, as it can add latency to synchronous + group RPCs. Services that only make asynchronous RPCs (e.g. JBoss + Cache configured for REPL_ASYNC) and do so in high volume may + be able to improve performance by configuring their cache to use + the udp-async stack below. Services that only + make synchronous RPCs (e.g. JBoss Cache configured for REPL_SYNC + or INVALIDATION_SYNC) may be able to improve performance by using + the udp-sync stack below, which does not + include flow control. + + + udp-async + Same as the default udp stack above, + except message bundling is enabled in the transport protocol + (enable_bundling=true). Useful for services + that make high-volume asynchronous RPCs (e.g. high volume JBoss Cache + instances configured for REPL_ASYNC) where message bundling may + improve performance. + + + udp-sync + UDP multicast based stack, without flow control and without + message bundling. This can be used instead of udp if + (1) synchronous calls are used and (2) the message volume (rate and size) + is not that large. Don't use this configuration if you send + messages at a high sustained rate, or you might run out of memory. + + + tcp + TCP based stack, with flow control and message bundling. + TCP stacks are usually used when IP multicasting cannot be used + in a network (e.g. routers discard multicast). + + + tcp-sync + TCP based stack, without flow control and without message + bundling. TCP stacks are usually used when IP multicasting + cannot be used in a network (e.g.routers discard multicast). + This configuration should be used instead of tcp + above when (1) synchronous calls are used and (2) the message + volume (rate and size) is not that large. Don't use this + configuration if you send messages at a high sustained rate, + or you might run out of memory. + + + jbm-control + Stack optimized for the JBoss Messaging Control Channel. + By default uses the same UDP transport protocol config as is + used for the default 'udp' stack defined above. This allows the + JBoss Messaging Control Channel to use the same sockets, network + buffers and thread pools as are used by the other standard + JBoss AS clustered services (see + . + + + jbm-data + Stack optimized for the JBoss Messaging Data Channel. TCP-based + + + + You can add a new stack configuration by adding a new stack + element to the server/all/deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml + file. You can alter the behavior of an existing configuration by + editing this file. Before doing this though, have a look at the + other standard configurations the AS ships; perhaps one of those + meets your needs. Also, please note that before editing a + configuration you should understand what services are using that + configuration; make sure the change you are making is appropriate + for all affected services. If the change isn't appropriate for a + particular service, perhaps its better to create a new configuration + and change some services to use that new configuration. +
+ + It's important to note that if several services request a channel + with the same configuration name from the ChannelFactory, they are not + handed a reference to the same underlying Channel. Each receives its own + Channel, but the channels will have an identical configuration. A logical + question is how those channels avoid forming a group with each other + if each, for example, is using the same multicast address and port. + The answer is that when a consuming service connects its Channel, it + passes a unique-to-that-service cluster_name argument + to the Channel.connect(String cluster_name) method. + The Channel uses that cluster_name as one of the factors + that determine whether a particular message received over the + network is intended for it. +
+ +
+ The JGroups Shared Transport + As the number of JGroups-based clustering services running in + the AS has risen over the years, the need to share the resources + (particularly sockets and threads) used by these channels became + a glaring problem. A stock AS 5 all + config will connect 4 JGroups channels during startup, and a total of + 7 or 8 will be connected if distributable web apps, clustered EJB3 + SFSBs and a clustered JPA/Hibernate second level cache are all used. + So many channels can consume a lot of resources, and can be a real + configuration nightmare if the network environment requires + configuration to ensure cluster isolation. + + Beginning with AS 5, JGroups supports sharing of transport + protocol instances between channels. A JGroups channel is composed of + a stack of individual protocols, each of which is responsible for + one aspect of the channel's behavior. A transport protocol is a + protocol that is responsible for actually sending messages on the + network and receiving them from the network. The resources that are + most desirable for sharing (sockets and thread pools) are managed by + the transport protocol, so sharing a transport protocol between channels + efficiently accomplishes JGroups resource sharing. + + + To configure a transport protocol for sharing, simply add a + singleton_name="someName" attribute to the protocol's + configuration. All channels whose transport protocol config uses the + same singleton_name value will share their transport. + All other protocols in the stack will not be shared. The following + illustrates 4 services running in a VM, each with its own channel. + Three of the services are sharing a transport; the fourth is using + its own transport. + + + +
+ Services using a Shared Transport + + + + + +
+ + The protocol stack configurations used by the AS 5 ChannelFactory + all have a singleton_name configured. In fact, if you add a stack to the + ChannelFactory that doesn't include a singleton_name, + before creating any channels for that stack, the ChannelFactory will + synthetically create a singleton_name by concatenating + the stack name to the string "unnamed_", e.g. unnamed_customStack.
@@ -70,9 +259,12 @@ HAPartition is a general purpose service used for a variety of tasks in AS clustering. At its core, it is an abstraction built on top of - a JGroups Channel that provides support for making/receiving RPC - invocations on/from one or more cluster members. HAPartition also - supports a distributed registry of which clustering services are + a JGroups Channel that provides support for making/receiving RPC + invocations on/from one or more cluster members. HAPartition allows + services that use it to share a single Channel and + multiplex RPC invocations over it, eliminating the configuration complexity + and runtime overhead of having each service create its own Channel. + HAPartition also supports a distributed registry of which clustering services are running on which cluster members. It provides notifications to interested listeners when the cluster membership changes or the clustered service registry changes. HAPartition forms the core of many @@ -84,60 +276,95 @@ - The following example shows the HAPartition MBean definition packaged with the standard JBoss AS distribution. - So, if you simply start JBoss servers with their default clustering settings on a local network, you - would get a default cluster named DefaultPartition that includes all server - instances as its nodes. - -<mbean code="org.jboss.ha.framework.server.ClusterPartition" - name="jboss:service=DefaultPartition"> - - <! -- Name of the partition being built --> - <attribute name="PartitionName"> - ${jboss.partition.name:DefaultPartition} - </attribute> - - <! -- The address used to determine the node name --> - <attribute name="NodeAddress">${jboss.bind.address}</attribute> - - <! -- Determine if deadlock detection is enabled --> - <attribute name="DeadlockDetection">False</attribute> - - <! -- Max time (in ms) to wait for state transfer to complete. - Increase for large states --> - <attribute name="StateTransferTimeout">30000</attribute> + The following snippet shows the HAPartition service definition packaged with the standard JBoss AS distribution. + This configuration can be found in the server/all/deploy/cluster/hapartition-jboss-beans.xml file. + + + + ha-partition + + + - <! -- The JGroups protocol configuration --> - <attribute name="PartitionConfig"> - ... ... - </attribute> -</mbean> - - Here, we omitted the detailed JGroups protocol configuration for this channel. JGroups handles the - underlying peer-to-peer communication between nodes, and its configuration is discussed in . The following list shows the available configuration attributes - in the HAPartition MBean. + jboss:service=Naming + + @org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss:service=HAPartition,partition=${jboss.partition.name:DefaultPartition}", exposedInterface=org.jboss.ha.framework.server.ClusterPartitionMBean.class, registerDirectly=true) + + + + + + ${jboss.partition.name:DefaultPartition} + + + ${jboss.bind.address} + + + 30000 + + + 60000 + + + + + + + @org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss:service=DistributedState,partitionName=${jboss.partition.name:DefaultPartition}", exposedInterface=org.jboss.ha.framework.server.DistributedStateImplMBean.class, registerDirectly=true) + + + + + ]]> + Much of the above is boilerplate; below we'll touch on the key points + relevant to end users. There are two beans defined above, the + HAPartitionCacheHandler and the HAPartition itself. + + The HAPartition bean itself exposes the following + configuration properties: - PartitionName is an optional attribute to specify the + partitionName specifies the name of the cluster. Its default value is DefaultPartition. Use the -g (a.k.a. --partition) command line switch to set this value at JBoss startup. - NodeAddress is an optional attribute used to help generate a unique name for this node. + nodeAddress is unused and can be ignored. - DeadlockDetection is an optional boolean attribute that - tells JGroups to run message deadlock detection algorithms with every request. Its default - value is false. + stateTransferTimeout specifies the timeout (in milliseconds) for initial application state transfer. State transfer refers to the process of obtaining a serialized copy of initial application state from other already-running cluster members at service startup. Its default value is 30000. - StateTransferTimeout is an optional attribute to specify the timeout for state replication across the cluster (in milliseconds). State replication refers to the process of obtaining initial application state from other already-running cluster members at service startup. Its default value is 30000. + methodCallTimeout specifies the timeout (in milliseconds) for obtaining responses to group RPCs from the other cluster members. Its default value is 60000. + + + The HAPartitionCacheHandler is a small utility service that + helps the HAPartition integrate with JBoss Cache + (see ). HAPartition exposes + a child service called DistributedState (see ) + that uses JBoss Cache; the HAPartitionCacheHandler helps ensure + consistent configuration between the JGroups Channel used by + Distributed State's cache and the one used directly by HAPartition. + + - PartitionConfig is an element to specify JGroup - configuration options for this cluster (see ). + cacheConfigName the name of the + JBoss Cache configuration to use for the HAPartition-related cache. + Indirectly, this also specifies the name of the JGroups protocol + stack configuration HAPartition should use. See + for more on + how the JGroups protocol stack is configured. - In order for nodes to form a cluster, they must have the exact same PartitionName and the ParitionConfig elements. Changes in either element on some but not all nodes would cause the cluster to split. + + In order for nodes to form a cluster, they must have the exact same partitionName + and the HAPartitionCacheHandler's cacheConfigName + must specify an identical JBoss Cache configuration. Changes in either + element on some but not all nodes would prevent proper clustering behavior. You can view the current cluster information by pointing your browser to the JMX console of any Index: en-US/images/clustering-SharedTransport.png =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: en-US/images/clustering-SharedTransport.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream