Streaming events? You should be storing them.

It was 1494. Luca Pacioli - a Franciscan Monk, mathematician, and friend of Leonardo De Vinci - published a book he’d spent decades working on. Titled Summa de arithmetica, it would be the book that would see him gain his title: “the Father of accounting”.

Pacioli’s book contained the first published detailed description of the double-entry accounting system: a method of bookkeeping that relies on a two-sided accounting entry, known as a debit and credit.

This meant that should any changes occur in the log, the bookkeeper would add new entries into the book rather than overwrite previous entries. As such, the bookkeeper had visibility of any changes, when they occurred and by whom.

What on earth does this have to do with Event Sourcing?

Event Sourcing is a pattern for storing data as events - a fact that took place in your business - in an append-only log. This means that, similarly to Pacioli’s double-entry method, original entries are retained rather than overwritten, keeping the context of any events.

The database you use for Event Sourcing matters. Consider that it was a good 500 years ago that Pacioli realised the importance of permanently storing changes made to logs. So you might be surprised to learn that today in our data-rich world, businesses are streaming their data into traditional databases that subsequently lose data and its context.

How your traditional database is losing your data

Traditional databases can be fantastic, there’s no denying that. Relational databases, for example, have been around since the 1970’s. There’s been a huge amount of effort and research put into them and we now have incredible distributed relational databases. But these have problems when it comes to Event Sourcing.

Common issues include:

Loss of business context: your traditional database might keep your current state, but you’ll lose the who, why and when of each change
Loss of original data: if any changes occur to your data, the original record is overwritten
You’ll need to query your data: traditional databases require you to pull data rather than have it pushed to you, so it’s not streaming
Leads to ETL - the anti pattern: you’ll face complexities when you need to get data out

As your data becomes richer over time, these issues will be amplified.

Operational databases built for Event Sourcing

To address the downfalls of traditional databases when it comes to Event Sourcing, a new category of operational database has emerged: the state-transition database.

State-transition databases are natively focused on storing events as an immutable sequence, keeping the order and business context of any state changes. Whilst events in the log cannot be changed, their effects can be altered by later events.

To demonstrate how this works in practice, imagine an “InvoiceIssued” event is appended to the log one day, only the address the invoice was issued to was incorrect. In your traditional database, the old address would likely be overwritten. But in your state-transition database:

An “InvoiceVoided” event will be added, to show the original invoice was incorrect
Then a new “InvoiceIssued” event will occur with the correct address

For each event, the dates and times will also be stored. You can even store who made any changes to your data (e.g. customer or agent), allowing you to go back in time to understand what occurred. You’ll therefore retain the business context - information that’s usually lost in traditional databases.

The advantages of event based state transition databases

The advantages of this model to your business are huge and span far beyond obvious benefits like having an audit-level amount of information for every event in your system.

Gain business agility and flexibility for downstream technologies: when you’re trying to export your data out to another system, you'll have options for how you want to do that, what data you care about and what that looks like
Have meaningful data at your fingertips: with the right configuration, you can have useful reports and analysis of data whenever you need it
Separate concerns: appending your data to your data store is separate from how you consume data. This enables you to leverage query and analytics, plus challenges the limitations of traditional operational databases.
Replay events from any point in time: you may want to rebuild state for each user and keep it in memory, or listen to certain events within a set time period. This means you not only gain context of future events, but past ones too

EventStoreDB: your source of truth

EventStoreDB is an open-source event based state transition database that will allow you to keep the story behind your data. With use cases in almost every industry and big names like Walmart and Xero already using the database, find out how you can also harness the true power of event-driven architecture here.

This blog is also available as a webinar, where you'll learn more about applicable use cases. Watch on demand.