Easy Modular Monolith — Part 2— The OutBox Pattern

Published in

ITNEXT

8 min readJun 1, 2021

In part one we prepared an MVP version of a simple app in a modular-monolith way. Having a skeleton allow us to implement further improvements.

In this article, we will try to implement an OutBox patter that will handle a message delivery for us, and what is more important — ensure that consumers of the message will get it.

One thing in the IT systems world is sure — if something can fail — it will fail in some time. Let’s take a look at our AddProductCommandHandler.

What we can see is that after we save data to our database, we produce an event that should be consumed by other “modules”. In a perfect world, sending this event will never fail and consumer will always be able to handle it — but not in ours :)

Not searching very far — MediatR can fail to publish an event for some reason — misconfiguration/infrastructure fails etc. Even after successful publish the event handler still can throw an exception because of some internal rules/validation/problems. The MVP version doesn’t handle that kind of situation well — in case of fail we will end with inconsistent data — what more as we don’t store information about failing anywhere we are lack context and not able to react.

Architecture

To prevent that kind of situation we will implement something that is called the “Outbox Pattern”. The whole idea behind this pattern is to instead of instantly publishing a message — store it somewhere and move responsibility for publishing to someone else.

The important caveat is to remember that saving message and changes in CommandHandler should happen in one transaction!

Take a look at the above image — it looks very simple — we have OutBox Module that handles storing messages into DB and WorkerProcess that uses OutBox Module to get messages and publish them by MediatR Notifications.

Let’s try to connect this to our ModularMonolith application — it will look a little different and more complex but the whole idea is the same.

The Web Api project is our entry point to the application. It registers all other dependencies and therefore has access to any module. The biggest change here is that it will register two new things.

The Outbox module — Migration and fetching stored messages from DB. It contains an OutboxDbContext that will be used to deliver outbox DbSets to other modules.
The Outbox WorkerProcess — Background process that will dispatch all not handled messages and publish them to MediatR notification.

The idea is to store all outbox messages (from any of our modules) into the same database and table in a separate schema — out.OutBoxMessages.

Implementation

Contracts

First, we will need to create an abstraction over the message publishing interface. This will allow us to implement multiple ways of handling message publishing in future. Right now it will have two methods like below:

As you can see there is a new interface too. IIntegrationEven interface will be a marker interface for all events that should be published between modules.

OutBox Module

Outbox module will be much simpler than History or Product. Let’s make it flatter.

As you can see most code here belongs to the Infrastructure layer. From a pragmatic point of view, there was no reason to split this same way as in the Product and History Module. No plans for any Domain/Application related stuff here and even if then we are able to extend this approach by splitting layers by folders instead of small projects.

What's more — this module will be used differently. It contains a base logic for our OutBox patter therefore it will be referenced by any module that would like to enable Outbox support.

The most important part here is our OutBoxDbContext that is used to deliver a code responsible for saving/fetching OutBox messages.

The OutBoxMessage entity looks like this:

In the Type property, we will store information about the type of our message (e.g ProductCreatedEvent).
In Message property— the whole serialized object of “Type”.
Both properties will be used later in WorkerProcess to deserialize the message.

ModularMonolith.Infrastructure

In this project, we will implement cross-cutting functionality* — in this case, it will be an InMemoryEventBus that will be handling publishing messages. In this version, we will deal only with IIntegrationEvent.

As you can see instead of immediate publishing messages by MediatR, we store them in our database. Thanks to that we will be able to grab all these messages later and handle them.

The most tricky part in the implementation of this EventBus was how to do that without breaking the DRY principle. As I mentioned earlier the caveat is that changes in command handler and saving the message has to happen in one transaction.

Let’s take a look at our AddProductCommandHandler. As you can see there are new lines here:
One is calling Publish on our EventBus (line 27) and the second one calling Commit() on the Repository (line 29).

Ef Core works in this way that all changes added to the context in the life cycle of it will not be committed to the database until SaveChange is called.

In Part1 of this series, our repository called SaveChanges in the “Add” method committing a transaction right away. In this version, we will call commit later — after “Publish” on the EventBus called.

So what was so tricky here?

We can have multiple modules and all of them will use the same entity - OutBoxMessage. We don’t want to create multiple DB context and share a DB connection between them to allow committing in one transaction — it would make our life much harder in future when we would like to move our Modules to separate microservices*.

Let’s look one more time at our InMemoryEventBus:

As you can see as an input parameter it accepts an OutBoxDbContext.
The whole idea is to — in modules that we would like to apply Outbox — inherit “Module” DbContext by OutBoxDbContext like this:

Then we should create a separate, concrete module EventBus that will inherit from our base InMemoryEventBus — it will allow us to resolve proper EventBus in runtime:

Ok — so what happens here?

This allows us to share an instance of DbContext (as it is registered as scoped) between InMemoryEventBus and CommandHandler — hence Commit() will save all changes from Command Handler and “Publish” in one transaction.

Worker Process

The worker process is our special guy who will be responsible for grabbing all our messages from the database and publish them using MediatR.
Let’s take a look:

First, we need to grab two services from our DI container. (line 34)
OutBoxDbContext for DB access and MediatR for publishing.

The most important part is happening in the loop (line 38).
First, we need to get a type of our message. As our Modular Monolith is one application we are able to Load a DLL that contains all Contracts (ModularMonolith.Contracts project) and resolve type from there. (line 42)
Having a contract type we are able to deserialize a message. (line 44).

Next, we use MediatR to publish a message. If all is good, we update ExecutionData, log information about handling message and move to the next one.
In case of an error, log error information and try to handle the next message. What important in this version we won’t implement any more sophisticated ways of handling errors (Retries/Poison Messages/Notifications). This is something that will happen and is placed in the roadmap :)

The drawback of this approach is that the worker process handles messages in the loop. In our example, it executes messages every 1 second. If we are forced to handle a lot of messages then this place WILL become our bottleneck. We will take a closer look at this problem in future.

That it is!

We have a working solution that guarantees our messages to be stored and handled. In the nearest feature, we will improve the way that we are handling exceptions in our Outbox to make the solution much more resistant to failures.

Summary:

OutBoxPattern is a pattern that guaranty the delivery of a message to the consumer by persisting it (in one transaction) and delegate dispatching to an external “application”.
OutBoxModule delivers an OutBoxDbContext that can be used in other modules to deliver OutBox support. To add support for Outbox for the next module we just need to inherit DBContext by OutBoxDbContext and create a module EventBus by inheriting InMemoryEventBus.

The whole code is available here:

https://github.com/Ridikk12/ModularMonolith/tree/OutBoxPattern

Easy Modular Monolith — Part 1 — MVP

ridikk12.medium.com

In Next Part:

Easy Modular Monolith — Part 3 — Logging

In a perfect world, in IT systems there is no place for bugs and maintenance. As developers, we prefer to deliver new…