CAPS 6 & Java MQ: Part 2 - Conventional Clusters
By Louis Polycarpou on Aug 15, 2008
How to achieve high service availability
In Java MQ terminology, if you require only high service availability, you should use either conventional clusters or parallel brokers. Running parallel brokers essentially means disconnected brokers with no special broker-side configuration and therefore no destination sharing between them. Conventional clustering in Java MQ requires that all brokers in the cluster are fully connected.
Conventional clusters offer the following features:
- Scalability and load balancing of clients
- Support for both file and JDBC message stores
- High service availability only (i.e. no high data availability - although you still get some data availability if you're persisting messages).
- Messages are stored in only one broker (no data replication). Thus if you lose a broker and it's data store is not recoverable, you will lose messages.
- Delay-time configurable client failover to other brokers if a broker fails and the clients are still running.
- Inter-broker communication and state sharing and synchronisation for destinations and clients (not messages)
- Use of a single master broker instance which can optionally be used to keep track of the configuration change record.
Key considerations when setting up conventional clusters
- Should I use a master broker? A master broker provides the use of a configuration change record which maintains state for destinations and durable subscriptions. If a master broker is assigned, but fails, any new operations that affect the cluster state - such as creating new durable subscriptions or destinations - are not allowed. However, message delivery for existing destinations and durable subscribers may continue. A master broker helps to ensure cluster synchronisation integrity after broker failure and recovery but is a single point of failure. If having a single point of failure of the master broker is an unacceptable risk, consider HA clustering instead.
- Should I use a file store or JDBC store? You can use either but managing and monitoring using a database store is typically more expensive in terms of cost and performance than using the local, optimised file store. This really comes down to customer preference and what the relative TCO will be. If multiple Java MQ clusters are to be deployed and some require data availability with JDBC HA, it makes sense to use the JDBC store throughout for common broker administration.
- How many brokers should I run per cluster?
The temptation is normally to run a broker per machine or per app server domain but doing this restricts the scalability of the solution. If the number of deployments is likely to grow vastly, it is worth considering running your brokers in a separate tier which can be scaled independently of the app server. However, in small deployments of 2 dedicated app server boxes, if you run your brokers on the same machine as the app servers it is simpler to manage and your applications benefit from the performance gains of mostly local communication.
There are also resource constraints to consider that may warrant running more brokers. For instance, for each persistent and non-persistent uncommitted message in the system, the broker responsible for that message requires a small additional amount of memory (approx. 2K) to provide caching of message headers (which is later recovered upon consumption). Java MQ provides configurable throttling capability to prevent a message flooding scenario in which hundreds of thousands of uncommitted messages can leave the broker starved of memory, however, this has a performance impact on the JMS producers and it is worth also considering adding more brokers and configuring the maximum number of messages and clients per broker to compensate.
It is also important to consider how the application will run in the event of failure or planned shutdown of one of the brokers; will the remaining brokers be able to support the additional load demands and is it acceptable for the remaining applications to rely on having just one broker running if a broker needs to shut down for maintenance / upgrade purposes? My recommendation is to start with either 2 or 3 brokers.
Configuring Java MQ conventional clusters for CAPS 6
Some simple steps to follow to configure and start a cluster of brokers:
- Set up multiple GlassFish instances (optional) If you require service availability of your messaging layer, it is likely you'll also require service availability (and possibly load balancing) of your application clients using multiple app server instances. Java MQ provides load balancing to your CAPS 6 JMS deployments without the need to cluster your GlassFish domains.
- Configure each domain's lifecycle management
If you've read Part 1, you should already know that MQ clustering with CAPS 6 requires that you set the MQ lifecycle management to use REMOTE mode and where to set this. Alternatively, if you also require EMBEDDED support for some deployments, you can leave the default broker running in EMBEDDED mode and just start your clustered brokers outside of the cluster. As long as the clustered broker lifecycle is managed manually, you are supported.
Begin by installing two CAPS GlassFish instances - domain1 and domain2, either on separate machines or on the same box (at least for testing purposes). Read how to create additional CAPS 6-ready instances here.
- Start the brokers
Brokers can be started using the
imqbrokerdcommand in the .../CAPS6/appserver/imq/bin folder. You need to supply configuration parameters on the command line or provide the name and varhome arguments to look up the required property file for the broker and cluster settings from the props directory of your broker instance directory e.g. ...\\CAPS6\\appserver\\domains\\domain1\\imq\\instances\\props\\config.properties. It is possible to configure common cluster properties to all brokers in the cluster by placing them in a separate file referenced with imq.cluster.url and setting that from each broker's config.properties.
The second option is to supply all the parameters at the command line or via your own script. I prefer this option since I can use the ps and pargs command to review the process parameters easily. Here's how you do it assuming we have two brokers - one per domain - running on ports 7675 (master) and 7677. Since the brokers only need to synchronise state on startup (as opposed to entire message stores), they are available for client connections very quickly.
Although not strictly required, when possible, I recommend that you start the master broker first...
imqbrokerd -tty -name broker1 -port 7675 -Dimq.hostname=hostname-pc -Dimq.cluster.clusterid=mycluster -cluster hostname-pc:7675,hostname-pc:7677 -Dimq.cluster.masterbroker=hostname-pc:7675 -varhome \\dev\\caps6\\appserver\\domains\\domain1\\imq
...followed by the other broker(s)...
imqbrokerd -tty -name broker2 -port 7677 -Dimq.hostname=hostname-pc -Dimq.cluster.clusterid=mycluster -cluster hostname-pc:7675,hostname-pc:7677 -Dimq.cluster.masterbroker=hostname-pc:7675 -varhome \\dev\\caps6\\appserver\\domains\\domain2\\imq
Note 1: MQ brokers are not allowed to use localhost or the the loopback IP address 127.0.0.1 so it is good practice to explicitly define the host name.
Note 2: I try to avoid using the default port (7676) for clustered brokers to avoid clashing with other app server domains running default EMBEDDED brokers.
Explanation of the above startup arguments:
Argument name Explanation
sends errors and warnings to the command window (for standard output):
set to the name of your host if using a multihomed machine (optional)
a unique id for the cluster; for conventional clusters this provides additional validation that the broker is joining the correct cluster. Note that the cluster id is required for HA clustering (see Part 3) and may also be used for other purposes in the future, and although this is currently optional for conventional clusters, I strongly recommend using it.
the instance name to assign to the broker
the port (or ports) that the local broker is running on
the comma-separated list of brokers in the cluster. NB: The hostname part is optional if running locally.
host:port of a single designated master broker in this cluster
directory where the broker instance resides (or will be created upon first use) to maintain the broker's file message store, configuration and log files
Once both brokers are started for the first time, they will establish cluster connections to their peers but can receive and deliver messages even before this point unless a master broker is specified and not yet running.
Starting the brokers in this way creates a vanilla configuration for each broker by default in the ...\\appserver\\domains\\domain1\\imq\\instances\\<brokername> directory (on Windows) with configuration files in props, message store in fs370 and log files in the log directory. The ...props/config.properties can be modified to include the above startup arguments so that subsequent restarts only require the varhome and broker name parameters to be specified; the rest will be read from the config.properties file.
Configuring CAPS 6 applications to use Java MQ clustersWith the cluster up and running, we can now consider the client-side configuration made in CAPS 6 applications. JMS connections formed by CAPS 6 deployments using JMSJCA must pass client-side options required for failover to the Java MQ client runtime. The aim is to provide a list of brokers that are available to a JMS client and a strategy for initial connections and reconnects, in the event of broker failure. The properties to consider setting are:
imqAddressList=mq://hostname1:portno1,mq://hostname2:portno2 imqAddressListBehavior=RANDOM imqAddressListIterations=1 imqReconnectEnabled=true imqReconnectInterval=5000
Explanation of the client side parameters:
|imqAddressList||a comma-separated list of broker URLs avilable to the client|
|imqAddressListBehavior||Set to PRIORITY to navigate the list in order until a connection is formed or RANDOM to select a broker randomly.|
|imqAddressListIterations||how many times to iterate over the entire list|
|imqReconnectEnabled||set to true if client failover is required|
|imqReconnectInterval||how long to pause between reconnect attempts|
NB: JMSJCA has its own redelivery handling which is preferred with CAPS 6 deployments and that is why imqAddressListIterations is set to 1.
CAPS 6 provides multiple options of where to specify those properties, the suitability of each may differ depending on whether you are developing applications using repository-based projects, JMSJCA MDBs or using the JMS BC.
Repository-based applicationsCustomers using CAPS 5.1.x would typically set the JMS client properties in their environment and for Java MQ would create a new Sun Java System Message Queue under a logicalhost in the environment. CAPS 6 adds the ability to also create a URA which would allow direct use of any JMS provider or ability to retrieve non-managed connection factories using JNDI. These options are also available to CAPS 5.1.3 customers from UR1.
- Option 1: Create a new Sun Java System Message Queue under a logicalhost in the environment with the following connection URL:
In the above URL, the imqAddressList property is set directly from the comma-separated list provided with the rest of the client options appended as a query string. It is possible that you may also want to append other properties to the Java MQ client runtime for reliability and flow control, as required.
- Option 2: Create a new Unified JMS Resource Adapter under a logicalhost in the environment with the same connection URL, username and password as above:
- Option 3: Create a new Unified JMS Resource Adapter under a logicalhost in the environment which retrieves standard non-managed Java MQ XA Connection Factories that are bound in your JNDI store.
For this option, you should firstly create some JMS XA connection factories and bind them to your JNDI store. You can do so in the Java MQ Administration Console by launching imqadmin from the command line. For example, to use the reference file store implementation, firstly create a new directory on your file system (e.g. c:\\dev\\CAPS6\\jndistore) and then use imqadmin to create a new object store with the following properties:
Next connect to the object store and create three XA connection factories - for each of XAQueueConnectionFactory, XATopicConnectionFactory and the unified JMS 1.1 XAConnectionFactory). You will need to set the client-side connection handling properties for each connection factory as follows:
Once these have been added, use the URA as with Option 2 but set the the message server URL as jndi:// and use the following advanced options to locate the factories created with imqadmin:
For maximum reuse, I recommend that you configure each of JMSJCA.UnifiedCF, JMSJCA.QueueCF and JMSJCA.TopicCF even if you think your application will only initially use one of these factories.
NB: If you currently use the jndi:// method, it actually bypasses Java MQ specific code (that would be invoked when using mq://). This essentially gives you a message listener running in serial mode which may cause problems with Java MQ. A convenience switch will be provided to turn these on in future updates.
I recommend using either option 2 or 3 above since both of these options allow you to change your provider without changing your deployment profiles. You may need to create multiple URA definitions for multiple clusters and with different client-side options for your applications. Given the caveat above, you should go with option 2 if you are deploying applications against CAPS 6 GA.
JMSJCA MDB applicationsIf you need to create JMS MDBs with CAPS 6, you must create JMSJCA MDBs to ensure that they use the sun-jms-adapter. Michael Czapski provides a step-by-step guide on how to do this here. Do not use standard MDBs - this is a gotcha that you must avoid as explained in my Grok entry.
There are three options for retrieving / creating the connection factory.
- Option 1: Use lookup:// to retrieve a JMSJCA-managed connection factory.
The lookup://<resource_name> syntax configures the activation to retrieve a JMSJCA-managed connection factory from a given resource name and is only currently available for JMSJCA MDBs. There are some managed pools and resources pre-installed as part of your CAPS 6 installation, which all point to the default_JMS_host (which is by default an embedded broker on mq://localhost:7676).
You can either create your own managed pool or modify one of the existing connector pool settings to use your new client-side cluster configuration. The CAPS 6 installation adds the following managed resources and pools to the default GlassFish installation:
Connector Resource Connector Connection Pool Description jms/tx/jmq1 jmq1-tx Default transaction pool for Java MQ (uses default_JMS_host settings) jms/notx/jmq1 jmq1-notx Default non-transaction pool for Java MQ (uses default_JMS_host settings) jms/tx/default jmq1-tx (if STCMS is not installed or stcms1-tx if it is) Default non-transaction pool jms/notx/default jmq1-notx (if STCMS is not installed or stcms1-notx if it is) Default non-transaction pool
I would always recommend creating a new pool and leaving the defaults alone but this is not enforced. If you choose to modify an existing pool, you will need to update its Connection URL. For example, to modify the default transactional pool, navigate to Resources> Connectors> Connector Connection Pools> jmq1-tx and change the Connection URL from (default_JMS_host) to mq://hostname-pc:7675,mq://hostname-pc:7677?imqAddressListIterations=-1&imqReconnectEnabled=true&imqReconnectInterval=5000 as shown below:
NB: The username and password can be left as (default_JMS_host) to use the admin username and password defined in Configuration> Java Message Service> JMS Hosts> default_JMS_host.
When you create the JMSJCA MDB, set the context name for lookup:// to the name of the connector resource (not the pool itself) e.g. lookup://jms/tx/jmq1:
NB: You can also retrieve app server-defined queues and topics by specifying lookup://jndi_name. When you deploy your MDB, a random, reconnect-enabled connection will be formed to one of the clustered brokers.
- Option 2: Use jndi:// to retrieve a non-managed connection factory.
As with repository-based projects, you will need to create the non-managed connection factories using imqadmin. Set the same JNDI provider options and JNDI names of the three connection factories (as before) in the Advanced tab of the Edit JCA Activation dialog.
You can also retrieve administered queues and topics bound to JNDI using imqadmin by specifying jndi://jndi_name
- Option 3: Use mq://<url> to specify in-line connection factory settings.
The final options is to set the entire connection URL with query string can be set directly in the activation spec so that it does not need to be looked up, as shown below:
NB: If you specify non-JNDI destinations, like foo above, and foo does not already exist in the broker cluster, it will be dynamically created. If foo does not receive any messages, it will be destroyed by the broker shortly after its creation.
JMS Binding ComponentThe JMS BC ships and is available to use in CAPS 6 and Open ESB but is not currently supported. If you are using the JMS BC within a composite application, it is currently possible to use the Java MQ connection URL directly or use jndi:// when configuring the WSDL extensions for this component. I will cover these initial options here and revisit when it is released:
- Option 1: Use jndi:// to retrieve a non-managed connection factory.
As with repository-based and JMSJCA MDB projects, you will need to create the non-managed connection factories using imqadmin. You must set the JNDI provider options and JNDI name in the service jms:address of either the xaQueueConnectionFactory or xaTopicConnectionFactory depending on the destinationType defined in the jms:operation of the WSDL binding. You can set this either by modifying the WSDL directly or via the CASA editor:
- Option 2: Use mq://<url> to specify in-line connection factory settings.
You can set the entire connection URL with query string directly in the jms:address element of your WSDL so that it does not need to be looked up from JNDI, as shown below:
Testing failover and recovery of your CAPS 6 applications You now know all the options and gladly we won't have to go through them all again for Part 3! Now you can test what you've built, using any of the options discussed. Deploy your application and test client failover by shutting down one of the brokers. The client should failover and resume processing of that destination, if possible. Shutting down the entire cluster will cause the clients to go into a retry loop until one of the brokers is restarted.
Reading the log files to understand what is going on with failover and recovery can be difficult due to repetition of errors due to the retry logic going on. Here are some of the messages to look out for in the log as you test failover and recovery.
If your client cannot connect to a broker from the list of addresses, you may see one of the following errors in the server.log:
[C4003]: Error occurred on connection creation [hostname-pc:7675]. - cause: java.net.ConnectException: Connection refused: connect
###### Connect exception for : mq://hostname-pc:7675/jms
A successful connection or reconnection attempt may show:
JMSJCA-E015: [sync-QueueReceiver(Queue1) @ [mq://hostname-pc:7675,mq://hostname-pc:7677?imqAddressListBehavior=RANDOM&imqAddressListIterations=1&imqReconnectEnabled=true&imqReconnectInterval=5000]]: message delivery initiation was successful.
When a client disconnects, it will go through the following states of recovery:
[I107]: Connection recover state: RECOVER_INACTIVE, broker: hostname-pc:7675(65521) [I107]: Connection recover state: RECOVER_TRANSPORT_CONNECTED, broker: hostname-pc:7677(49189) [I107]: Connection recover state: RECOVER_STARTED, broker: hostname-pc:7677(49189) [I107]: Connection recover state: RECOVER_IN_PROCESS, broker: hostname-pc:7677(49189) [I107]: Connection recover state: RECOVER_SUCCEEDED, broker: hostname-pc:7677(49189) [I107]: Connection recover state: RECOVER_INACTIVE, broker: hostname-pc:7677(49189)
The final part of this blog entry, Part 3 (coming soon), discusses the use of both service and data availability with Java MQ and CAPS 6 using JDBC HA provided by MySQL Cluster.