Message Correlation
  • 04 Mar 2019
  • 7 Minutes to read
  • Contributors
  • Comment
  • Dark
    Light
  • PDF

Message Correlation

  • Comment
  • Dark
    Light
  • PDF

Article Summary

One of the things that is interesting when you look at Logic Apps and BizTalk is a comparison of how they route messages between instances of workflow. I would guess the difference really boils down to BizTalk trying to implement a complex and configurable approach to solve ESB and Long Running Transaction patterns where as Logic Apps is aiming towards the API Orchestration and iPaaS approaches which sacrifice some of the more complex features for speed of delivery and simplicity.

If we look at the typical Logic App when used in an API Orchestration approach like in the below diagram:

image.png

You can see here that most of the API Connectors for the Logic App are in process RPC style calls.

The question however is if you need to implement a more complex process and need to use some more asynchronous approaches how would you implement the message correlation required to support this.

Reflecting on Correlation in BizTalk

Before we get into a logic app implementation pattern, from an archtiects perspective I would like to take a step back and discuss how BizTalk implements this approach. One of the answers for how to implement this is that you could use BizTalk alongside your Logic App. You could drive your long running transaction in BizTalk and use a Send Port out to a Logic App to take advantage of the modern connectors.

When it comes to message correlation, one of the often misunderstood things in Microsoft Integration was the really powerful features that BizTalk offered in its implementation of this pattern.

Because BizTalk messaging and orchestration were fully decoupled via the message box the default design required this. In many cases people would take the simpler option in an orchestration to do a solicit response pattern and use the type of port binding that kept things simple. You could however define much more complicated rules by creating your own correlation types and correlation sets to define which messages would route back to your orchestration based on the properties of a message.

If we take a look at how that worked we could use the below diagram.

image.png

  1. The orchestration is ready to send a message. Rather than directly sending a message to an endpoint, instead it creates a message and then creates a correlation set which is associated with the message and these are pesisted to the message box. If the orchestration does not get a response quickly it may also dehydrate its state to the database to save on resources
  2. From the message box an instance of a send port is started. This can run in a completely seperate windows service (BizTalk host instance) and can even run on a seperate server. The send port will begin processing the message
  3. The send port will send the message to the destination endpoint in an RPC fashion and get a response which will be processed
  4. The send port will put the response message back into the message box
  5. The message box uses the response message promoted properties and existing correlations waiting to be activated within the message box to match the response message back to the original orchestration instance which sent the request. At this BizTalk would rehydrate the orchestration if it had been dehydrated and then make it continue at the next step along with the response message it now has.

Message Correlation Identifier Pattern

Before jumping into the implementation, lets take a second to discuss the pattern. This pattern is called the Message Correlation Identifier Pattern.

image.png

Note
The excellent Enterprise Integration Patterns book discusses this pattern in more detail alongside many others - https://www.enterpriseintegrationpatterns.com/patterns/messaging/CorrelationIdentifier.html

The key to the pattern is how you can send a message asynchronously from a component and get a response later and to know which instance of the component should handle each response.

There are a couple of ways we could achieve this with a Logic App, each of them requires us to combine the Logic App with some kind of message queue or database. I think the most effective pattern however is to combine Logic Apps and Service Bus messaging. Service Bus messaging already supports this kind of message exchange pattern if you implement the RPC pattern with queues using the Reply to Session ID and a session based response queue. The thing we need to do is to take advantage of this with a Logic App.

Why do I need this Pattern?

Even though the pattern is widely recognised, lets include a few examples where you may use this. As I said earlier this was the default way that BizTalk used to work, but for Logic Apps the default way is an RPC HTTP call from the workflow to an API Connector so you need to implement this pattern yourself. In this case its a good idea to know when you need to use it and when not.

A couple of examples may include:

  • Load Levelling
    Imagine my process has multiple parts and some of them have resource bottlenecks. I could break up my implementation using this approach so that I could use the queue to slow the processing of interactions with some systems that needed throttling.
  • Long Running Process Scenario
    I could send out requests for data which may take long periods of time to return (hours and days). This approach would let my Logic App just wait around for the data to come back
  • Scatter Gather Scenarios
    I could use this approach to implement scatter gether patterns where more than 1 response return to the Logic App instance

There are others but hopefully this gives you a few ideas of where you can use the pattern.

Conceptual Implementation

Lets now consider the conceptual implementation of this pattern. For demo purposes I am going to activate my Logic App with Postman but the key bit we are looking at is the interaction between the Logic App and Service Bus, see the below picture:

image.png

The way this works is as follows:

  • Postman sends a message to activate my Logic App and will wait for a response
  • The Logic App will create a request message to send to the queue and send it, it will also create a value for the Reply To Session Id property
  • A process will receive messages from the queue, do something with them and then send a response message to the response queue (which is session enabled) and it will set the session id to be the value the Logic App sent to it as the Reply To Session Id
  • The Logic App will be listening on the response queue for messages on a particular session id and when the response arrives the Logic App will receive the message and continue
  • The Logic App will send the response message from the back end process back to Postman as the response message.

Walk Through

The below video is a walk through of this scenario:

Things to Consider

While this pattern is quite good I think there are a few things you need to consider when implementing it with Logic Apps:

  • Your Logic App is going to be actively polling for a response compared to the BizTalk approach which used dehydration. Do you know how long it will take to get a response? If not it may be worth implementing a back off pattern to reduce the frequency of polling and possibly an abort pattern if the response hasnt appeared in a certain amount of time
  • The implementation in Logic Apps is not quite as slick as id like it to be. Its a pain having to use the loop when I know im only getting one response and also having to keep looping until I get the message. It would be cool if Microsoft could implement an API Connector for Service Bus which had better support for this pattern. Maybe a Send RPC action and a Receive RPC action where it could do the looping etc under the hood to keep my Logic App simpler would be really nice
  • One slight concern I have with this pattern is that without the dehydration pattern BizTalk uses, the Logic App instances will remain polling taking up resource. You may want to implement a delay shape within your loop for the message so the Logic App goes to sleep for a while rather than constantly polling. I think this is an area I may do some investigation on to understand how the implementation of the pattern may affect overall performance and the number of Logic Apps you may have running. I would certainly be interested to hear peoples opinion and experiences with this.

Other Resources

There are a couple of other resources out there which discuss this topic with similar and alternative approaches to tackling it, check out these articles to expand your understanding:


Was this article helpful?