Active-Passive Background Service In .Net 6
Background story:
Imagine that you have multiple servers that try to access and manipulate some data in some time interval (like 1 second or faster). The data that servers try to manipulate can be accessed only by one process simultaneously. In other cases, it will cause some unpredicted and hard to track issues.
We should be able to run multiple instances on the same background service on multiple servers. At the same time, only one instance should be run in "Active" mode, which means do work. Others should run in "Passive" mode, waiting until the "Active" one goes down, and then automatically switch from "Passive" to "Active".
As you can see in the image above, we will implement a push-oriented service discovery functionality to fulfil requirements. Each instance will register itself at the start in the external data source (can be whatever Azure Database/SQL Server/No-SQL). All instances will contain a logic that will tell them which mode they should work. In case of an unexpected server failure, we will implement the logic of auto un-registration of instance.
Requirements:
Functional requirements:
- Each Background Service has to be able to manipulate data.
- Each Background Service has to be able to work in "Passive" mode.
- Each Background Service has to be able to work in "Active" mode.
- Each Background Service will register itself at the start.
- Each Background Service will re-register itself in X seconds.
- Each Background Service will un-register itself at the exit.
- If Background Service is unable to unregister itself — for example, server crashes — then it should be un-registered by another service.
Non Functional Requirements:
- NET 6 Worker Process.
- Servers have limited access to the internet.
- Servers can crash un-expected.
Background Services
NET 6 Host allow registering multiple Hosted Services as part of the same host. In this case, the application will contain two long-running background services that work in parallel.
Implementation
Let's start with background services implementation.
RegisterInstanceWorkerProcess
In Execute Async method (line 24) lives our whole logic.
AS this is Worker Process, we have a while loop which iterates each 1second and do below:
- Register worker process instance with unique instance id.
- Delete orphaned instances
- Wait 1 second and repeat the whole operation.
In case of any error, we log the error, stop the instance and try to unregister it (StopAsync method). StopAsync method will trigger only if we gracefully stop the worker process. In case of unexpected critical failure (for example, the server goes down as electricity is off), the method will not execute. In this scenario, another instance of the worker process will unregister this one (DeleteOprhantInstanes method in line 36).
Worker
The worker process is much simpler — in a while loop, we check if the instance should be active or passive. Depending on the outcome, we execute corresponding logic (DoInActive and DoInPassive methods).
RegisterInstanceService
The core of this article is our RegisterInstanceService class.
This class contains four methods.
Let's take a look:
In Line 20, we have a Register method.
The method fetches all registered instances, and If there isn't any currently active in registered instances, we register a new instance and set it as active. If our instance already exists (as we register each 10s), we just update the registration date.
Unregister method (line 36) simply deletes an instance for our datastore.
IsActive (line 41) method is quite simple too. From all registered instances, we try to find the one with the current id and return the active flag.
DeleteOrphantInstances is more interesting. This method allows us to remove all instances that are still in the data store but hasn't been updated for at least 1 minute. This method guarantee that even though some instance can crash without unregistering, we still automatically remove them from the datastore, allowing other instance to take over an "Active" mode.
Summary
Sometimes we cannot use dedicated tools/frameworks to achieve the functionality requested by our stakeholders. This time it was an active-pasive functionality.
The whole code and working solution are available on my GitHub.
