%20alt.svg){kind=small}
Introduction
Developing applications within a complex business environment can be a nightmare due to the web of interdependencies among the database, codebase, and the business. Anyone who has experienced this first-hand knows how challenging it is to understand and make changes to these monolithic and unwieldy systems.
Event sourcing helps you design systems that are easy to understand and simple to decouple even within complex environments. This can help you transform and scale your systems and business in the hyper-competitive digital world.
This series of three articles will explain why event sourcing is effective at tackling this complex challenge.
In this first article, we will explore the historical reasons for the increasing cost of managing complex data. The second article will discuss the unique features of event sourcing. The last article will explain the core benefits of event sourcing and how it can transform your business.
The Growing Costs of Complex and Core business data
Simple and Complex Data
There are many types of data in the world.
Some are simple. Like the day of the week, a street name, or the number of steps you walked today.
And some are complex because they are frequently updated, widely distributed, and have to comply with business rules.
Core Business Data and its Growing Costs
Core business data is often complex but extremely valuable because it can be used to improve your competitive advantage in the market.
Examples of core business data include:
- Customer profiles and preferences
- Medical patient electronic health records
- Banking account and transactions
- Sales transactions and agreements
However, systems that process core data are also complex and often hard to understand and change.
Why are complex systems hard to understand and change?
In a world that is becoming more complex, the cost of managing core data is driven up. This in turn hurts business’ capability to acquire deeper insight and adapt to the ever-changing market.
Since databases are the primary storage of core business data, they play a critical role in this problem.
Limitations of Transactional Consistency and the Rise of Eventual Consistency
Domination of Traditional Databases in 1990s
Traditional databases such as relational has been the de-facto database since it went mainstream in 1990s.
It was designed to provide transactional consistency across core business data, which works well for applications that need to adhere to strict business rules. An example of this is a wire transfer between two bank accounts.
TRANSACTIONAL CONSISTENCY ensures that a series of actions in a system are carried out completely and correctly or not at all. This prevents the system from ending up in an unpredictable or erroneous state. As a result, users can trust the system to maintain its integrity even after numerous actions.
Reaching the Limits of Transactional Consistency in 2000s
However, shortly after the internet went supernova in 2000s, developers discovered that it was hard to distribute data to an increasing number of external systems.
This is because applications that use traditional databases were designed for transactional consistency, which does not scale well - especially at an internet scale:
However, developers continued to use traditional databases because there was no better choice at the time.
And this, along with the complexity of core business data, resulted in databases that were monolithic and tightly coupled to many parts of business.
Rise of Eventual Consistency in the 2010s
With the popularity of cloud and big data in the 2010s, demand for data distribution continued to see explosive growth.
To work around the coupling that comes with transactionally consistent operations, many developers began to look for integration patterns that utilize eventual consistency instead.
EVENTUAL CONSISTENCY ensures that a system will become consistent over time, given that no new updates are made. While users may see outdated or conflicting data temporarily, the system guarantees that all systems will eventually converge to the same value. This allows systems to provide high scalability and availability, even at the cost of immediate consistency
Comparison of transactional vs eventual consistency:
Eventual consistency is not a new concept. Scheduled jobs and ETL workflows have been popular since the 1990s.
However, there was a huge uptick of new event-driven solutions that support eventually consistent operations during 2000s and 2010s. These include tools such as message queues/brokers and practices like event-driven architecture.
These solutions made it simpler for developers to implement event-driven applications.
Read our Beginner's Guide to Event Driven Architecture to learn how using events as the core of your system can help you build more resilient, scalable systems.
Problems with Eventual Consistency as of Now
While these solutions have been immensely helpful in the advancement of eventually consistent operations, it still takes a lot of work and skills to develop fully decoupled systems.
The common problem with these event-driven solutions is that they are still built on top of data that comes from traditional databases.
Problems such as message duplication, out-of-order, and stale data is not trivial to handle even with these solutions. And this is because the backing traditional databases don't have the features to support eventual consistency.
Event sourcing and state transition databases, on the other hand, are designed for eventual consistency in mind and are a better alternative to traditional databases for decoupling complex systems. Something we explore more in parts 2 & 3.
Summary
In this article, we have:
- Described how core business data is complex and typically volatile, distributed and highly compliant
- Examined why traditional databases tend to lead to tightly coupled systems due to their lack of eventual consistency support
- Listed out recent developments of event-driven solutions over the past few decades, and
- Explained why applications are still unable to fully decouple from the data because these event-driven solutions are built on top of data from traditional databases