Sunday Oct 28, 2007
Monday Oct 22, 2007
Friday Oct 19, 2007
Wednesday Oct 17, 2007
By lindaschneider on Oct 17, 2007
Still, its 4:00 am and I have two more hours to kill until I go to work, so this is a good time to actually make another entry. Last night, I had dreams of blogging and content fill my head, invading my dreams and leaving me here typing. These thoughts were triggered after a meeting with an open source evangelist-ish person, who reminded me that if I work on a good product (which in my mind I do) that I need to write about it and inform others why its important.
So where do I start? I could write about how I spent yesterday - but that would bore me - a day spent in meetings doing what next planning (although once we know what next is - I might). I could write about the inner workings of the product (but that requires brain power which is noticeably lacking at this moment) Since I'm operating on too little sleep, I'll start with the simple stuff and talk about what is MQ.
Open Message Queue is the open source version of the Sun Java System Message Queue. Now, I realize that that sentence I just wrote was not terribly useful, so a little better is that it is a type of Message Orientated Middleware. Making your head hurt yet, a more accurate version is that it is a product that implements the JMS specification. (still no clue ?)
Its asynchronous messaging, which means that the sending process doesn't have to wait until the receiving process handles the data before it continues processing (unlike something like RMI). Additionally, the content of the data doesn't have to be defined up front (although obviously the receiver needs to know what to do with it when it arrives).
When I try to describe what I work on to my friends, I tell them I work in middleware which immediately triggers yawns. Its actually more interesting and useful than it appears to be based on that description. MQ is really plumbing, an infrastructure which allows you to connect two disparate applications together through a messaging layer. You send a message (which can be a text message, a java object or several other constructs) through the api and it gets placed on a destination awaiting deliver to another project. At some point, that other project retrieves the data and begins to process it. It allows you to disconnect the timing of when the messages are processed from each other. For example, you may take a reservation order and want to actually process it at a later point. One application sends the message when the customer places the order, another later retrieves it and continues the processing.
JMS is also the messaging layer used with J2EE servers (and specifically open Message Queue is used in glassfish). If you are using MessageDrivenBeans to connect your applications in glassfish, its probably running on MQ.
Why not write your own
Well, obviously someone could. On the surface it doesn't sound very hard - you write a tcp socket that connects the applications. But the JMS api provides reliability guarantees that are difficult to achieve with a simple server. We promise that a message that you send will remain in the system until it is picked up by the other application (storing it if necessary) and promise that the messages will be delivered in order. We support transactions. We also take out the complication of determining who gets the message and when it is removed. In general, unless you have a lot of time for development on your hands, you probably don't want to do all the work yourself. Well, I know that if I didn't work on it for a living, I wouldn't want to write it - too much work. Many (many, many) man years went into making sure that it functions seamlessly.
So why use MQ specifically
That's always a hard question, my first answer is of course, that we are better, which is just a flippant response (I doubt that there is an engineer on any messaging product who doesn't feel that their implementation is on par or superior to the competition. Its why we became software nerds in the first place - to build software we are proud of). A less biased answer is that the product has a history, making it more stable than most (I'd like to think all, but thats no doubt over reaching) of the open source competition. It may not have all of the really cool features but it should run out of the box in anyone's environment.
How do I use JMS
Well, the real place to start is by reading the JMS specification. That where I started many years go when my second level manager at the time came into my office and said "Congratulations, you are now a Message Orientated Messaging person" (no, I didn't know from birth that its what I wanted to do). Still I'm sure a HelloWorld example would be much better. However, its almost morning so thats a topic for another day (maybe tomorrow). For now, I'll just give a quick overview of the two domains (messaging types) supported by the JMS api. They are Topics and Queues.
Topics are destinations (repositories for messages) that are publish-subscribe (one to many messaging). This means that an application (producer or specificaly Publisher) sends out a single message and multiple applications (called Consumers or in this case Subscribers) get the same message. The message hangs on the system until it is delivered to everyone. A good example of a place to use this would be a stock quote which should go to multiple people.
Queues are destinations that are point-to-point (one-to-one). This means that one message goes to only one interested party. For example, in the reservation example listed about, there may be multiple applications processing orders, if an order comes in it should be delivered to only one of the order processors (or you would get multiple reservations).
My alarm just went off, notifying me that a new day has begun. Its time to stop this entry and get moving. I'll try to keep my blogging promise and do a hello world example next.
Wednesday Aug 08, 2007
By lindaschneider on Aug 08, 2007
This week is a pretty cool on for MQ because we just did what is hopefully out last build of 4.1 - our latest release and the release were we finally provide some sort of solution for High Availability (specifically data availability - we provided service availability in the past)
Its been a long few weeks cleaning up the last of the known high priority bugs and going through the lower priority ones to make sure nothing was missed.
Now everything is quiet.
I'll try to talk about HA a little more this week (how it works, why we did it that way) but for now my mind is on the future - what happens next.
In a few weeks we will be starting feature analysis for the MQ.next release. Its going to be the merging of the features of two JMS Products available from Sun (MQ and JMSGrid). The intention is to develop the product in a more open manner using OpenMessageQueue (
What are the features of the next release? Well, we aren't sure yet but at a minimum we will have to include:
- a file based HA story - which will provide message availability without the need for a highly available JDBC database
- Network Connections - the ability to connect to a remote cluster of brokers over a WAN connection
Note: this doesn't mean that support is going away for our current (JDBC) based HA implementation. Customer's using it now will not be left in the cold in the next release.
Tomorrow - I hope - I'll talk about HA and why we did it the way we did.
Wednesday Feb 07, 2007
By lindaschneider on Feb 07, 2007
For example, my morning was spent thinking about training for the next release of MQ.
This reminded me that we have free training for the 4.0 release of the product that I'm sure no one knows about (I wouldn't if I hadn't stayed up for several nights in a row writing it). If you are new to MQ or new to JMS or just have a few hours to kill its available for free from http://www.sun.com/training/catalog/enterprise/application.xml:
Ok - on to what I originally planned to talk about ....
Warning: This is going to be one of those blog entries that could serve as a replacement for sleeping pills ... still if you want to know a little bit about MQ clustering here it is.
Now - if you have a burning desire to hear about something different, let me know. Until then I'll just have to write about whatever comes to mind.
(I have the concept of adding in nice pretty pictures however I don't think I'm going to have time for this entry. We'll see. Right now I'm sticking to ugly ones.)
What are clustersClusters are a group of some sort of server which are grouped together to address scalability or availability. In MQ, our servers currently address scalibility and service availability but they dont address message availability (yet). I'll talk about how we did High Availability (an MQ 4.1 feature) which provides message availability in a seperate blog - it deserves its own attention. Clusters are generally broken into several types of configurations
- disconnected servers -servers provide scaliablity but dont communicate to each other.
- Fully connected clusters - ever server speaks with every other server
- Asymmetric clusters servers are connected together in some way but a server doesn't connect to everyone else.
- Primary/Backup clusters where one server mirrors the other and can take over if a failure occurs.
- Store and forward clusters where only some information passes between the servers or clusters or servers (e.g. a customer configures what is passed on). Its useful if you have servers running in different geographical areas over a WAN (where a fully connected setup just wouldn't make sense)
What does message queue do with clustersMessage Queue currently only supports Fully Connected Clusters. The original intention of clustering in message queue was to provide scaling (the ability to add more consumers in the system) and service availability (the ability for a producer or consumer to continue to send and receiver messages in the case of failure). We chose this clustering scheme because it seemed the simplest choice to give us the behavior we wanted. If we had gone for Asymetric cluster then we'd also have to deal with all the issues required to calculate the path that takes the least amount of hops (and pass a lot of information about how the cluster is connected between brokers).
Actually we also support the idea of disconnected servers (in fact we have some customers using it for failover) because it was absolutely no work (once we put in the reconnect behavior for clusters on the client side we were done).
Where are we looking at going in the future
- support data availability during failover (high availability) - targeted for 4.1
- support some sort of bridge (to allow store and forward clusters) - targeted for a future release.
How do brokers communicate ?Message Queue brokers send messages around the cluster. The messages are either broadcast (are sent to all brokers in the cluster) or unicast (are sent to only a specific broker). The format of these messages is a GPacket. Generally this packet is composed of three sections:
- a header - which includes a magic number, version, size (for parsing the rest of the packet) and enough information (timestamp, sequence) to create a unique id. A set of bit flags are also attached to the header to make it easy to mark packets in a specific way (e.g. the A bit - standing for acknowledge if a reply should be sent)
- a variable length header - these are name,value pairs
- the body - a byte buffer designed to hold any type of data.
The specifics of the protocol (what we put in the packet) is here. But the general gist is that control messages (messages that are sent when a consumer is added, a new destination is created or a lock is needed) go to all brokers in the cluster ... messages and jms acknowledgements go to only the brokers responsible for this message.
At some point, I'll talk about the ins and outs (what we send, where) but not today. Maybe next I'll talk about High Availability (possible solutions and why we chose the one we did) then again, maybe not.
Tuesday Jan 16, 2007
By lindaschneider on Jan 16, 2007
THe original intention of this blog when it was set up is that it would be a shared responsibility of the entire MQ team so different engineers would write something up on different weeks. However 3 months later, we are all still very busy and we still don't have a blog to discuss the product. For the moment, I'll be writing it. I hope to rope more folks in over time.
At this point, I'm not sure how to change the user from being generic to me (maybe I'll handle that tomorrow) but at least I'll say a few words about who I am ...
Hi, my name is Linda and I'm the technical lead of the Sun Java System Message Queue (including the open message queue product). Before my life as a technical lead, I worked on the design of the core infrastructure of the broker (the Message Queue server component). Now, I'm still working on determining what would be useful to journal about in a tech journal so we'll see how helpful this all turns out to be.
Friday Jan 12, 2007
By lindaschneider on Jan 12, 2007
I find the concept of a technology related blog to be a little strange so I'm not sure how this is going to go. But my plan is to cover the following types of content: \*
- overview of how the product is architected (overview)
- where the product is going (futures)
- what a specific feature does (feature)
- basics about jms or examples on how to use a feature (basics)
- general updates about the open source product (source)
- product updates and information (updates)
For today, I'm just going to give a quick 10 second overview about what the product is.
Open Message Queue is an open source project based off of Sun Java System Message Queue. The latest version currently available is 4.1 beta. It is available at:
https://mq.dev.java.net/It is a Java Message Service (JMS) provider which means that it provides reliable messaging between applications. For more information about the Java Message Service see:
In addition to the basic JMS features it also provides:
- clustering of servers
- integrated SOAP / HTTP messaging
- a C Client API
- a JCA 1.5 compliant Resource Adapter
- JMX support for both administration and monitoring
- High Availability (HA) support (starting in 4.1 beta)
Next Blog: an overview of the MQ architecture
A blog for Open Message Queue, the JMS provider in GlassFish Server, Open Source Edition