I've written previously about ActiveMQ Network Connectors as a way to help describe some of the responsibilities of the classes that are involved with configuring a network of brokers for ActiveMQ. I recently had to dive back into that section of the code and found that blog post invaluable to help recall things about network connectors that I wouldn't otherwise retain in memory. Since it was so helpful to me, I'd like to continue on with how the network bridges and demand-forwarding subscriptions work.
Hopefully this information will be helpful to others that are interested in knowing more about what goes on under the covers, or to those contributing to the ActiveMQ project and are going to dig around in that area of the code.
This entry keeps it closer to the code details. If you are interested in viewing this from a little higher level, my colleague Jakub Korab has an excellent post with great diagramsthat explain broker networks.
So recall briefly from the last blog that a Network Connector can be defined like this the configuration:
[xml]
<networkConnector uri="static://(tcp://localhost:61626,tcp://localhost:61627)" />
[/xml]
The DiscoveryAgent associated with the NetworkConnector will initiate setting up a bridge for each transport URI specified in the uri attribute of the networkConnector This means if you're expecting multiple URIs to act as master/slave URIs, then the connector will try to establish a bridge to both the master and slave always. This is most likely not what you want (ie, you want to connect to the master only). An alternative would be to wrap the URIs in a masterslave: transport to identify the URIs appropriately and connect to the master at first and only to the slave if the master fails.
Note, that for each URI, if a bridge ends up being destroyed because of an unanticipated exception, the bridge will always attempt to be re-created.
What does a network bridge do?
So what exactly does the bridge do? Well, a network bridge basically coordinates with the local broker (using a VMTransport) and the remote broker (using a Transport specified by the URI: TCP/SSL/HTTP, etc) and implements the "Demand Forwarding" behavior expected from the network of brokers. Let's look step by step how that happens.
How a bridge works
In the DiscoveryNetworkConnector#onServiceAdd method, you can clearly see where the transports are being created. One for the local broker, and one to the remote broker. Next, you'll see the bridge created, passing in the transports, and finally the bridge started.
When the bridge starts, it first sets up TransportListeners for both local and remote transports before starting the transports. This allows the bridge to service commands from the local and remote brokers and act accordingly. This is sort of what the TransportConnection class does for normal broker clients. The meat of the code for the network bridges is in org.apache.activemq.network.DemandForwardingBridgeSupport.java So when our transports consume from advisories, send messages, or ack messages, it's doing so using the ActiveMQ command objects directly.
Once the listeners are set up, the transports (that were created in DiscoveryNetworkConnector#onServiceAdd) are then started, which will kick off the normal exchange of command objects: for the remote broker, a wire-format negotiation takes place to determine what version of OpenWire to use (for the TCP/SSL connections) and then the remote broker sends a BrokerInfo command to the bridge's remote transport to identify itself. For the local transport, there is no wire-format negotiation. The local broker sends a BrokerInfo command to the local transport to identify itself.
Now we have the transports configured to listen for incoming commands and we have them started and talking with the respective brokers (local and remote). Now the key to the Demand Forwarding takes place. We start the bridge to the remote broker. This part is responsible for determining what the "demand" is from consumers on other brokers. Gary Tully has an excellent blog post about the exact terminology of the Bridge, Demand, and Forwarding that explains in plain language the role of each of the components. When a consumer on a remote broker subscribes to a destination that exists on our local broker, we consider that consumer to have "demand" for the messages on the destination on our local broker. For example, consider a local broker bridged to a remote broker: if I have a producer to a destination named "foo" on the local broker, and a consumer attaches to the remote broker and wants to establish a subscription to a destination named "foo", the remote broker will do so without worry. But there are no producers to "foo" on the remote broker, only the one on the local broker. When the bridge is set up, it will see that there is a remote demand (consumer) for messages on "foo" so it will forward messages for the "foo" destination to the remote broker. There we have "demand" (the consumer on the remote broker) and forwarding the message thus: Demand Forwarding.
So how does the local broker determine demand?
In the DemandForwardingBridgeSupport#startRemoteBridge we set up a connection, session, and consumer to the remote broker. The consumer is set to consume from a Advisory Topic named ActiveMQ.Advisory.Consumer on the remote broker. If the networkConnector is set to bridge temp destinations, then the consumer is also going to listen to advisory topics named ActiveMQ.Advisory.TempQueue and ActiveMQ.Advisory.TempQueue. These topics notify listeners of new consumers that establish a subscription to a destination on the remote broker.
Setting up the demand
So now we see that bridge on our local broker is listening to the advisory topics on the remote broker. But how is the "demand" set up?
When we receive an advisory message from the remote broker, we check to see what kind of command was the underlying cause of the advisory. If the underlying cause was a ConsumerInfo (consumer subscription), then we know there is a consumer on the remote broker interested in having messages forwarded to it, so we set up a "Demand Subscription."
Forwarding messages
What happens is the same ConsumerInfo command that was used to create a subscription on the remote broker is altered a little bit (given a new consumerId, sessionId, etc) and passed on to the local broker. This creates a corresponding subscription on the local broker for the specified destination. Now when messages are sent to the destination on the local broker, the local destination will have a subscription. The local broker will dispatch the message to the subscription (which is effectively the bridge that created the subscription) and the bridge will now forward it on to the remote broker. You can see the bridge forward the message from the local broker to the remote broker in DemandForwardingBridgeSupport#serviceLocalCommand in the conditional that checks that its a MessageDispatch.
The DemandSubscription held by the bridge objects will always correlate with a subscription on the local broker, and one on the remote broker. An important key to the DemandSubscription objects is the consumerIds. The localInfo refers to the consumer with the consumerId associated with the subscription on the local broker. The remoteInfo will refer to the consumer with the consumerId associated with the subscription on the remote broker.
So those are the basics. I could have written a bunch more about the different things that pop up in different configurations, duplex bridges, more complex networks, etc, etc, but I think this is enough to jog my memory later or to help others understand how the demand forwarding works. Plus, I've already spent enough time writing, now it's time to hack on Apollo JIRAs.
If you have specific questions, please let me know!