Microservices Communication – Part4 – An Eventful World

Architecture, Distributed Systems, Microservices, Software, Technology / Wednesday, March 7th, 2018

Event Driven, Event Sourcing, CQRS, Apache Kafka, RabbitMQ are all words you will inevitably come across if you are in any way or form associated with microservices.

What is the common thread that runs through all of them ? It’s events, events and more events, a whole world of events.

As I read through papers, blogs and product documentation, I was pleasantly surprised by one fact –  it seems that Event driven architecture is not really mainstream and there is a feel of it being treated as a new revelation sweeping across the software world. It seems that it is being adopted widely only of late by the broader community. Surprisingly I wasn’t aware of this at all.  Reason being that I have worked on Event driven systems since my first project at work, all the way up until the last. Event driven systems are not new, they have been around for decades – at least in the Telecom world. In any case, I am glad to see the surge in popularity, it’s about time and I certainly feel right at home with events – a homecoming really.

Although I’m on familiar grounds here, it has been refreshing to see the different perspectives and newer approaches of event driven architectures. When one works with anything for too long, often, our views about it tends to become routine and habituated. It becomes harder to see it from different perspectives or bring novelty to it. This has helped me step out of that mode, look at it fresh and reconcile my experience with newer ideas.

What does Event Driven mean ?

An event is something that represents an interesting occurrence in your system. The initial events are typically, but not necessarily, due to external stimulus and most often occur asynchronously. These external events then result in a cascade of internal events – again occurring either synchronously or asynchronously, resulting in triggering various processing and state changes in the system. The significant point here is that events form the core of the design, they are treated as first class artifacts and they drive the system behavior. Such systems are called Event driven systems and said to have an event driven architecture.

Contrast this with the familiar request/response or request/reply pattern. It is easier to explain this with an example. So I’ll use the classic example of a basic VOIP call where user Alice calls user Bob. This is considered the most fundamental use case in telephony. Irrespective of the underlying signalling protocol used to establish the call, the application layer sees it normalized as a basic call between two users.

In Unified Communications systems the add on features on top of a basic call are numerous – in some cases can run into 100s and they are all built on top of a basic call. Let’s consider a simple case for discussion.

Say on this particular day, Bob is unable to answer calls, so wants them to rollover to his Voicemail. However he wants to be notified on his personal mobile phone of all the calls he has received during that time. If any caller leaves him a voicemail, he also wants to be sent an email with a voice attachment of it.

In the request/response pattern, the basic call service would recognize that the call to Bob went unanswered. It would then have to apply the post conditions set up for unanswered calls. A notification of the call has to be sent to Bob’s mobile. This is implemented by a separate service, say a ‘Call Notification’ service. The basic call has to invoke this service and request it to send a notification. The call also needs to be routed to Bob’s voicemail server. This is implemented by a separate service, the Voicemail service.  The basic call needs to invoke the voicemail processing by sending a request to it. If Alice chooses to leave a voicemail, then the basic call will have to process it further and send a request to the Email service to send an email with the voice attachment.  It hurts just thinking about the control flow and interactions here. Can you imagine what it would be like with 10, 25 or 50 such features interacting ? Even if asynchronous messaging is used between the services, it would still result in a Big ball of mud.

So is there a better way? Yes, an architecture based on events. Thankfully our system used an event driven architecture. It was made all the more powerful by combining the flexibility of events with Finite State Machines (FSM). The basic call emits events related to just the States of a basic call. Any feature interested in these events listens to them and acts as per it’s interests. So in the case of our example, the basic call would emit an ‘call unanswered’ event. The Call Notification service would kick off on seeing this event and send notifications as needed. The Voicemail service would also act on that event and kick off so as to allow Alice to leave a voicemail message. On completion the Voicemail service would emit an event, say – ‘voicemail deposited’. The Email service would then react to this event and send an email notifying Bob. Sounds far simpler and intuitive to follow, right ?

The salient difference to note here is how the control flow occurs in event driven vs the request/response patterns. In the latter, the requesting service is dependent on the other feature services and invokes them as needed and maintains the flow control. In the events case, the control or the dependency is reversed. The producer of events is independent of the consumers of the event. It has no control over who processes the events downstream. It is the consumers of the events who control the behavior on seeing events flowing through the system.

Another way of looking at it is the imperative vs the declarative paradigm in play. In the request/response style, the requesting service acts as the imperator – commanding the other services to do it’s bidding. In the event driven system a declarative style is used, the basic call service declares any interesting occurrences or its state changes to the world. The interested services react to them as needed. The basic call service doesn’t need to know as to who is listening to these events or how exactly they will react to it.

Another important difference is how new features can be introduced into the system. Say on receiving a call instead of answering, Bob wants to forward it to his co-worker’s desk.  In the request/response case, the basic call would need to be modified to handle this case by invoking a new Forwarding service conditionally. The changes will require regressing the basic call and all it’s dependencies. In the event driven system, the basic call service is unaffected. A Forwarding service will just plug into the system, listen to the relevant events and act as needed.

These differences put together, allows the services to be decoupled and offers more flexibility in addition to stream lining interactions. It also aligns better with a distributed architecture, allowing for scalability, evolvability and independence, which are all important for microservices.

One caveat I want to highlight is that although event driven systems offer great flexibility, it makes it harder to identify the complete process workflow through the system as there is no central controller through which all calls flow. For complex workflows involving multiple services, another aspect to watch out for are unintended interactions and how a change in event consumption by one service might affect another. In our system, if multiple services were trying to act on the same events leading to contention, it was resolved by prioritizing services. A service with a higher priority got to process the event before one of a lower priority. This did introduce some coupling, but a tradeoff that was acceptable given other choices.

One thing to note is that microservices can still internally use request/response style within the bounded context, either synchronously or asynchronously but when interacting with other services, they use events. So this is a hybridized approach and what is most commonly found in practice.

In conclusion, Event driven systems, by virtue of all the advantages are a great choice for implementing microservices. I’ve just covered the bare bones on event driven systems here. A big movement with events defines an approach which make events immutable and stores them as the single source of truth in the system. This is the EventSourcing paradigm, catching on in the microservices world, which I hope to cover in another post. There is much more that can be written about events, but this is it for this post. Here are some further reads to that end:

Further Reading:

Event Driven Systems – Martin Fowler This talks about classifying event driven systems into four patterns.

A talk on the same topic as above by Fowler

Microservices with events – This is from the Kafka team, a good read with a simple example

A different way of looking at event patterns – This is from Pivotal and an interesting read

Events as first class citizens – From StichFix and how they use events

Note that this article is part of the Microservices series. You can read the previous ones here : Prelude, Introduction, Evolution, Guiding Principles, Ubiquitous Language, Bounded Contexts, Communication Part 1, Communication Part2, Communication Part 3.

One Reply to “Microservices Communication – Part4 – An Eventful World”

Leave a Reply

Your email address will not be published. Required fields are marked *