By: Levi Masonde | Updated: 2024-08-06 | Comments | Related: > Cloud Strategy
Problem
As discussed in my previous article on microservices, one main reason some developers do not approve of the microservice architecture is how the communication between the services is carried out. More specifically, how does the data persist across different services? This is a valid concern since microservices need to hide their internal state whilst they need to have the freedom to "slice and dice" shared data.
Solution
Modern microservices employ the "data per service" approach, which encourages each microservice to have separate business logic, data, data store, and custom operations. This sounds good until you consider what happens when different microservices need to share data amongst themselves. The traditional synchronous communication between services can cause dependencies and tight coupling between microservices, which defeats the purpose of using this architecture style.
Using Restful APIs can eventually lead to a centralized data service being formed to contain all the data. API communication can quickly get complex as a Service's data sources accumulate over time. Furthermore, as services scale, the dependencies they have on other services can slow down the development cycle, which might lead teams to move datasets to their Service to give them more autonomy.
Multiple architectural patterns, like outbox and event sourcing, have been developed to address such Microservice communication issues. These are event-driven architectural solutions designed to help with microservices communication by generating event streams for an application.
This article looks at how you can use event sourcing to maintain persistent and asynchronous communication between microservices.
Event streaming can be used as a middle ground between service interfaces, i.e., the Services do not communicate directly with each other, but they interact through event streams.
What is Event Streaming?
Event streaming is the continuous sequence of events in a publisher/subscriber architecture design pattern. A publisher/subscriber pattern is a system that has a set of components:
- Publisher/Producer: Services, databases, or users that generate events data and publish it to topics/channels.
- Subscribers: Services, databases, or users that consume events and subscribe to receive messages from a certain topic.
- Topic: Subsets of events in a single category that move data from producers to consumers.
Event streams have temporal durability to ensure that the system does not remove events immediately after being processed. This way, systems can store data in memory for later retrieval, allowing the systems to react to both historical and real-time data/events.
Event Streams and Event Sourcing
Event sourcing ensures that every event or change in state in an application is captured as event objects, which are stored in the sequence they were applied. So, instead of saving the latest state of an application's data like traditional databases, event sourcing looks to save all the state changes of an application's data.
For this architecture pattern, a microservice generates an event or change state, and this would be stored as a local transaction and then forwarded to the microservice's aggregate; the aggregate's database is considered the event store in this instance. New entries on the event store are then published to an event broker like Kafka, which processes the events into predefined topics where they will broadcast the new events to any subscribed microservice:
Event sourcing employs the command query responsibility segregation CQRS pattern to query event stores. You can use event store write databases with Azure Cosmos DB to publish events to the event broker and use a no-SQL database for querying the event broker.
Benefits of Event Streams
Benefits of using event streams include:
- Real-time Operation: Events streams process single points of data rather than batches of data rather than batches of data. This allows the systems to react to application state changes in real-time or near real-time.
- Flexibility and Resilience: Adding a new microservice to an application is a lot of work. This means you need to make a lot of changes to your original application's microservices. But, with event streams, you can easily add new services without affecting the whole system due to the asynchronous and loosely coupled nature of the structure.
- Event-driven API: You can establish communication between backend systems and web clients.
Integrate Event Streams with Microservice Communication
Event streams work great with microservices, although you might need to optimize the architecture by employing other design patterns, like publisher and subscriber.
Event streams can act as a middle ground between the event store, databases, and services. Event streams achieve this solution by using event stream platforms, like Apache Kafka, which are designed to establish asynchronous communication between microservices using the publisher and subscriber pattern.
Kafka clients allow you to write distributed microservices that read, write, and process streams of events in parallel. Kafka also has APIs for the following:
- Admin API: Manage and inspect Kafka objects like topics and brokers.
- Producer API: Publish a stream of events to one or more topics.
- Consumer API: Subscribe to one or more topics.
- Kafka Streams API: Provide high-level functions to process event streams, i.e., take input from one or more topics and give an output to one or more topics.
- Kafka Connect API: Develop data import/export connectors that enable external systems to produce data into Kafka's topics or consume data from Kafka's topics.
Furthermore, Kafka stores topics in partitioned buckets located on different Kafka brokers.
Conclusion
Microservice architecture is a great solution for modern applications that generate, consume, and process tons of data per minute. This architecture can easily mirror real-life business logic processes of companies into software applications and models, making the design and development easier with domain experts being involved.
The microservice architecture can become complex as you build more into your application. To avoid the microservices from becoming too dependent on each other and having data that is not synced across all microservices (no source of truth), you must implement techniques and patterns such as event sourcing or outboxing. A future article will use event sourcing to create a microservice application.
Next Steps
- Learn how to manage Azure DataBricks using the Azure CLI
- Learn how to use Eventstream in Microsoft Fabric.
- Learn how to build real-time data pipelines with Azure Event Hub, Stream Analytics and Cosmos DB.
- Learn how to Monitor SQL Server Analysis Services with Extended Events.
About the author
Levi Masonde is a developer passionate about analyzing large datasets and creating useful information from these data. He is proficient in Python, ReactJS, and Power Platform applications. He is responsible for creating applications and managing databases as well as a lifetime student of programming and enjoys learning new technologies and how to utilize and share what he learns.This author pledges the content of this article is based on professional experience and not AI generated.
View all my tips
Article Last Updated: 2024-08-06
