Spring UG Meeting recap - Oleg Zhurakousky on Spring Integration, Rabbit MQ, and GemFire

Oleg Zhurakousky talking about S/I, Rabbit MQ, GemFireLast night we had the privilege of hosting SpringSource's Oleg Zhurakousky at Chariot's training center for the Philly Spring User Group.  Oleg talked about Spring Integration, and showed us how he recently put together a proof-of-concept of an integration platform using Spring Integration, the Rabbit MQ AMQP Server engine, and Gemfire, a distributed caching engine.

All of these tools are owned by VMWare/SpringSource.  RabbitMQ and Gemfire are recent acquisitions.

Oleg started by giving us a basic overview of the challenges of JMS as a standard.  Since it just defines an interface, interoperability is a big challenge.  Also, since the producers and consumers both are wired to a destination via code / configuration, moving to another destination generally involves recompiling code on both sides.  Also JMS developers generally have to deal with the fact that non-Java systems can't directly interface with JMS, various middleware, and connection management, to name a few issues.

Enter AMQP

The AMQP (advanced message queuing protocol) was created by a group of developers known as the AMQP working group.  It provides a wire-level message protocol that is built to support interoperability across all implementations.  The key distinction between JMS and a wire-level protocol like AMQP is that AMQP does not specify an API, only the format of the data on the wire.  The working group is comprised of companies from Microsoft to RedHat to financial companies like JP Morgan Chase, Credit Suisse, and Bank of America, where high-volume, real-time feeds of data need to be processed from a variety of systems.

The RabbitMQ Acquisition

SpringSource/VMWare acquired RabbitMQ this year with the goal of owning a highly scalable AMQP messaging service.  Licensed using the Mozilla Public License (with of course, support contracts available) the platform runs on just about any platform, and the code is primarily written in ERLang.  Oleg explained how easy it is to configure a cluster - once you start the cluster, elements are immediately replicated.  You can set up things like in-memory or persistent replication stores, passive/active models, and a number of other configurations.

Oleg said the biggest thing that struck him about RabbitMQ is how fast it is.  And he should know, working for a number of years in various message oriented middleware, and now becoming a member of the Spring Integration team with Mark Fisher.  He showed us how quickly RabbitMQ starts, consumes and delivers messages, and how easy it is to administer using a simple command-line interface.

Exchanges and Queues

Unlike JMS, where developers attach connections and sessions, then produce messages and communicate with a wired Destination, RabbitMQ (and I assume AMQP) deals in terms of Exchanges and Queues.  Exchanges receive messages from a sender, and Queues deliver messages to a receiver.  Furthermore, these elements are just referred to by a name, not a physical class reference, so they can be reconfigured on the fly, at runtime.

In the demonstration, Oleg sent messages from a Spring client using Spring Integration, delivering them to an Exchange in Rabbit MQ.  From there, he persisted the data in a persistent cache called Gemfire (more on that in a minute) and then another component delivered the message to the consumer via a queue.

Oleg says SpringSource is working on Spring components for RabbitMQ like a RabbitMQ Template and Proxy Factory Bean, to bring it into the Spring fold.  From there, we can orchestrate the activity via Spring Integration, which can define workflow to move things along.  

Oleg spent some time discussing Spring Integration's pipe-and-filters approach to integration, and also urged us all to pick up a copy of Enterprise Integration Patterns, which can be found online at www.eaipatterns.com (you can order the book via the website as well).  He explained that Spring Integration is based very heavily on the EI patterns, so that the book will be a worthwhile investment if you are looking into using Spring Integration.

About Gemfire

Gemfire is a distributed cache engine that SpringSource also recently purchased.  In the demonstration, as I said above, Oleg is persisting the data from the first (incoming) message flow into this cache so that it can be sent as quickly as possible.  He says the team has been working with Gemfire's developers to take advantage of features like sequencing messages, which we assume will be forthcoming in a future release.  

One of the interesting things about tools like Gemfire is the fact that large-scale sites like Amazon and eBay are actually already using huge, widely distributed caches to store data in memory across a large number of redundant hosts.  This, in a way, makes the data safer than if it was just being backed up with a disaster recovery plan - if there are thousands of hosts replicating the information, and delivering it to consumers who need it in the regions they are requesting it from, it becomes difficult to stop hosting that data.  A fascinating topic I'm really not doing justice to.

Wrap-up

Oleg's talk was really amazing.  He showed how VMWare / SpringSource is acquiring companies that can deal with massive data delivery and storage problems.  In my opinion VMWare is going after Oracle's business by side-stepping the physical box issue.  Why use a series of physical servers with CPU licenses when you can just use time on a large scale cloud?  Amazon does it.  Google does it.  With VMWare, Spring and some of these advanced technologies, it will be interesting to see some big customer rollouts in cloud-based systems, and to see how they stack up over offerings put together with conventional iron.

Note:  a recent blog post by Peter Ledbrook describes Rabbit MQ a bit more in detail.

Ken Rimple works as the Education Director and delivers Spring training courses for Chariot Solutions, a SpringSource partner, and is host of the Chariot TechCast, which can be found at techcast.chariotsolutions.com.