00:00:07 Event sourcing and its benefits over traditional relational databases.
00:00:31 Event sourcing and its core concept.
00:01:52 The rise of relational databases and CRUD software in the 70s.
00:04:29 Constraints and assumptions introduced by CRUD-based systems.
00:06:10 Benefits of event sourcing in bug fixing and system maintenance.
00:08:00 Frustration of dealing with mutable systems and quantum bugs in software engineering.
00:09:02 Event sourcing as a solution for supply chain problems and its relation to cheap data storage.
00:12:38 Adoption of event sourcing in large companies and its benefits for transparency and efficiency.
00:14:36 Event sourcing as an alternative perspective to the crude methodology in supply chain software design.
00:15:37 Future prevalence of event sourcing and potential barriers to its widespread adoption.
00:17:18 Comparing event sourcing principles to accounting principles and their importance.
00:19:22 The necessity of being data-driven in supply chain management.
00:20:25 Speculating on Amazon’s future growth and potential limitations.
00:21:07 Conclusion and closing remarks.
Summary
In an interview with Joannes Vermorel, founder of Lokad, host Kieran Chandler explores the concept of event sourcing and its benefits in supply chain management. Event sourcing involves representing a computer system’s state as a sequence of events, which provides a more transparent and immutable view of past events, making it easier to reproduce and fix problems. Vermorel notes that the popularity of the CRUD model, which allows users to create, read, update, and delete data, can sometimes make it difficult to recognize the potential benefits of event sourcing. However, he believes that as data storage becomes cheaper, event sourcing will become more prevalent, driven by the growth of tech-driven companies like Amazon.
Extended Summary
In this interview, host Kieran Chandler and Joannes Vermorel, the founder of Lokad, discuss the concept of event sourcing and its increasing popularity in comparison to traditional relational databases. Event sourcing is an approach where the state of a computer system is only a reflection of a series of events, with the idea that there is no direct representation of data other than the events themselves. For example, in inventory management software, there is no specific stock level, but instead a stream of stock movements that, when consolidated, provide a synthetic number representing the current stock level.
During the 1970s, choices were made about how to represent a company’s data electronically. The relational model became popular, and most supply chain software now falls under the category of CRUD (create, read, update, delete) applications. CRUD software is based on tables with fields, allowing users to add, read, modify, or delete lines within the table. This simple model has been successful and widely adopted across industries.
However, Vermorel notes that there are alternative approaches to data representation, such as event sourcing. He suggests that the popularity of CRUD can sometimes make it difficult to recognize the potential benefits of other methods. One significant difference between CRUD and event sourcing is the idea of mutability; in CRUD, the past is mutable, which can be puzzling from a philosophical perspective. In contrast, event sourcing offers a more transparent view of past events and the changes that have occurred in a system.
The discussion highlights the increasing interest in event sourcing as an alternative to traditional relational databases for supply chain management and other applications. Although CRUD has been widely adopted, it may not always be the best approach, and exploring alternative data representation methods like event sourcing can potentially provide better clarity and understanding of past events.
Vermorel explains that the vast majority of software systems used to manage supply chains, such as ERPs and MRPs, are built with mutable pasts, which can lead to complex and counterintuitive situations. However, he suggests that a better approach would be to view the past as immutable, as this can help avoid such problems.
One significant advantage of using event sourcing, according to Vermorel, is its usefulness in bug fixing. He notes that large supply chain software systems often have numerous bugs and undesirable features, which can persist for years and be frustrating to deal with. With event sourcing, all information in the system is represented as a series of events. When debugging, engineers can simply replay the events one by one to reproduce the exact situation that created the problem and fix it.
Vermorel contrasts this with the alternative situation where the state of the system is mutable, making it difficult for IT teams to reproduce and fix problems. This can result in a frustrating cycle of bug reports and attempted fixes, with bugs often reappearing after a short period of time. Event sourcing, on the other hand, provides a deterministic way to regenerate and fix problems.
The discussion then shifts to the idea of event sourcing in more detail, with Vermorel explaining that it involves looking at every operation that occurs within a supply chain. He suggests that one might wonder why event sourcing isn’t the default approach, as it aligns with the way accountants have used ledgers for centuries. Ledgers are typically immutable, with new transactions added to correct previous errors rather than rewriting past transactions.
Vermorel explains that the reason event sourcing isn’t the default approach is due to the high cost of computer storage in the late 1970s and early 1980s. To save on storage costs, software systems were designed to overwrite past data with updated totals. However, he points out that data storage has since become much cheaper, making it more feasible to implement event sourcing as a standard practice.
When asked whether smaller companies should be interested in event sourcing, Vermorel responds that it is a philosophical perspective on the past, with the idea that the past should be immutable in computer systems. While the majority of ERPs are not using event sourcing, Vermorel believes that adopting this approach can help prevent issues associated with mutable pasts in supply chain management software.
In summary, Joannes Vermorel advocates for the use of event sourcing in supply chain optimization, emphasizing its benefits in bug fixing and aligning with the immutable nature of the past. Although not yet widely adopted, event sourcing provides a more deterministic way to handle problems in supply chain software systems and could be of interest to companies of all sizes.
They explore the benefits of this approach, its adoption by tech leaders, and its potential future prevalence in the market.
Event sourcing is a method of designing software that stores the state of a system as a sequence of events, rather than through the more traditional Create, Read, Update, Delete (CRUD) methodology. Vermorel notes that tech leaders, such as Amazon, Zalando, and Alibaba, have embraced event sourcing for their supply chain systems, with Amazon being a major driver of its adoption in the market.
Vermorel argues that event sourcing offers several benefits to supply chain management. First, it provides greater transparency and efficiency, as well as reducing waste and improving responsiveness. Second, it highlights the limitations of CRUD, which was heavily influenced by economic constraints that were only relevant in the 70s and 80s.
Looking ahead, Vermorel believes that event sourcing will become more prevalent as data storage becomes cheaper and tech-driven companies continue to grow. He emphasizes that market adoption is driven by a “great filter” process, in which companies using less effective methods go out of business, leaving those using more successful approaches to thrive.
Vermorel also highlights the importance of event sourcing for companies with in-house software systems. Even if they don’t adopt event sourcing outright, understanding its principles can help them mitigate the problems of mutability in their existing CRUD systems. He asserts that event sourcing should be seen as a cornerstone of supply chain practice, much like the concept of a non-rewritable ledger in accounting.
As supply chain practitioners are data-driven by necessity, Vermorel contends that it is essential for them to have guiding principles for data collection, organization, and processing. He suggests that exploring event sourcing can help practitioners identify and address many underlying issues within their organizations.
Vermorel sees a future in which event sourcing continues to gain market share, driven by the growth of tech-driven companies like Amazon. However, he acknowledges that economies of scale may eventually limit the dominance of these companies, allowing for more diverse approaches to supply chain management.
Full Transcript
Kieran Chandler: So today, we’re going to understand a little bit more about how it works and discuss how it can provide better clarity of what happened in the past. So, Joannes, what exactly is event sourcing?
Joannes Vermorel: Event sourcing, technically, is the idea that the state in your computer system is only the reflection of a series of events. So, all that you can see in a computer system, or all that it does, is based on events. You only have events, and if you have anything like a total on display or any information in the system, it is only the direct result of a series of events. To make that very concrete, the idea would be, for example, if you have a stock level on display in your inventory management software, there is no such thing as a stock level in the software. All you have is just a stream of stock movements, where quantities were put or picked from, and when you consolidate all those events, you end up with a synthetic number that is the current stock level. But the idea with event sourcing is that the system does not have anything but events as far as the past is concerned.
Kieran Chandler: So where are these event sourcing techniques used? Is it something that we’re seeing often in the marketplace?
Joannes Vermorel: In the 70s, some choices were made about what would be the best way to represent a company’s data in general, and its past in particular. It might sound very philosophical, but it’s actually a very broad question. When you want to digitalize your company, you need an electronic representation of your company and its past. Even if it’s a very recent past, it is still the past. So, basically, you want to have a reflection of the past, and it turned out that there are many ways to do that. A few of those ways became incredibly popular in the late 70s and maintained their popularity for the last couple of decades that followed. But it’s not the only way to look at it. It’s very intriguing that when you have a certain way of doing things that becomes exceedingly popular, it might become so big that at some point, you don’t even realize that it’s actually possible to do it in very different ways.
So what happened was this relational model won, and with that, there was the type of application that was built on top of it. Basically, the vast majority of supply chain software nowadays falls into the category of what is known as CRUD software. CRUD is actually an acronym that stands for Create, Read, Update, Delete. So the idea is that you have tables, each table has fields, and then what your software does is that you can add a line (that’s Create), you can read an existing line, you can modify (that’s Update) an existing line, and you can delete an existing line. This model is very simple when you think about it, and those four variables, CRUD, managed to take over the world of supply chain, so that you can reflect pretty much everything with that. It has been very successful, very popular, but it’s not the same approach. And the interesting thing is that there are other alternative approaches, and event sourcing is one of them.
Kieran Chandler: Let’s look at the popularity of that CRUD approach then. Did that introduce any constraints or assumptions that we’re now used to working with?
Joannes Vermorel: Yes, it did introduce something that is
Kieran Chandler: Something that is very puzzling when you think about it, from almost a philosophical perspective, is that the past is mutable. I mean, when you think about your day-to-day experience, something happened in the past, you can’t change it, right? That’s just how it is. And yet, in computer systems, modern enterprise systems like ERPs, MRPs, etc., all those classes of complex software systems that run companies and their supply chains, the vast majority of those software systems are built on the crude design principle that the past is completely mutable. If you stop to think about that, you end up with all sorts of bizarre situations that are only the result of the fact that the past is mutable, and that can lead to a sort of warped way of thinking about the systems. It creates a lot of complexity and overall, it just collides with what you would intuitively think about the systems.
Joannes Vermorel: Yes, there is one benefit to event sourcing that is actually very significant and incredibly mundane: bug fixing. Large supply chain software systems have bugs, undesirable features all over the place, and typically, it goes on for literally decades. There are thousands of tickets that are open and pushed to IT, and most of them never get fixed. It’s so frustrating. Event sourcing actually relates quite a lot to bug fixing. At Lokad, we have been using event sourcing internally for a decade, and it’s very interesting. When you face a bug with an event-sourced system, all the information in the system is represented as a series of events. So when you want to debug something, you only need to replay the events one by one. Due to the fact that the only data that exists in your system is this stream of events, you can reproduce the exact same situation that created the problem and fix it.
Let’s compare this to the alternative situation where all your system state is mutable. The problem is that by the time you face a problem, like a glitch in stock levels, the problem is that stock levels keep changing. By the time the IT team looks at the problem, it will be gone. You can’t reproduce it, and you can’t fix it. It’s exceedingly frustrating as a software engineer when dealing with a system that is highly mutable. Most of the bugs turn out to be like quantum bugs, where there is a bug, but when you look at it, it vanishes. Then you stop looking at it, and it’s back again. It’s so frustrating.
Event sourcing is very interesting in this respect because it gives you a completely deterministic way to regenerate those problems and fix them for good. If you look at supply chain problems at large, even if software is not going to solve all the problems, it turns out that actually…
Kieran Chandler: Software is frequently creating a first share of the problems that modern supply chains are facing. Ok, so let’s move back and look a bit more in detail at event sourcing, this idea about looking at every single mundane operation that’s happened in a supply chain. It sounds a little bit dull. How easy then would it be to go back and check what’s happened and get to the root cause of a problem?
Joannes Vermorel: Why isn’t it the default to just record the events as they come and say this is the past, you know? When you look at what I would say people have done for centuries, like accountants, they have ledgers. Ledgers have this property of normally being immutable. You only append stuff to a ledger. If you get a transaction wrong, you don’t delete a transaction in the past. You just add another transaction that says this previous transaction was incorrect, and here is a corrective, and that’s how you work. So you don’t rewrite the past, you just append more stuff at the end.
But at the end of the 70s, computer storage was incredibly expensive, and thus the idea of having this immutability of the past was kind of a nice feature if you’re really tight. If your ledger, as an accountant, only has 10 pages and if you run out of space the only thing you have is an eraser. So, what you do is you try to summarize the totals for your books, add a few transactions, and whenever you run out of space, you just erase the past transaction, update the totals, and that’s it. That’s literally what happened in the late 70s and early 80s. The entire world of software went to a certain path that was making sense due to the fact that data storage was so incredibly expensive at the time.
Fast forward, 40 years later, data storage is just insanely cheap. You can buy a one terabyte hard drive for fifty dollars. That’s insanely cheap. So, if you want to have a super reliable data storage, you need multiple copies and a system to manage those copies, but we are below $1 per gigabyte, and data is just very, very cheap.
Event sourcing is very much the raw architecture of an entire ERP system. These are subjects that are really of interest to large multinationals and big companies like SAP. Should smaller companies really be interested in something like this?
Kieran Chandler: That’s an interesting question. Should smaller companies be interested in event sourcing?
Joannes Vermorel: I think event sourcing is a philosophical perspective on the past, which is basically the past is immutable, and the computer systems that you have need to reflect that, or you’re going to end up with suboptimal solutions. The vast majority of ERPs are not using event sourcing, so that’s the first catch. The vast majority of ERPs are mostly not using it. When you look at who is using event sourcing, it’s tech leaders who are typically not producing supply chain software, maybe except for Amazon, Zalando, and Alibaba, the companies who operate their own supply chain with their own bespoke software.
In terms of benefits, a lot of companies try to have more transparency on the supply chain. They want to be more efficient, less wasteful, more lean, more reactive, and they are struggling a lot with their IT systems.
Kieran Chandler: Although it’s the sort of problems that are very diffuse, there is no single answer to the problem. But what I see is that event sourcing is very interesting because it points out a specific flaw of, I would say, the overly dominant way to design software that is used for supply chain problems. Usually, most practitioners are not even aware of this perspective that there are other perspectives. So I think event sourcing is very interesting not necessarily because it’s the ultimate answer to those problems, but just because the very fact that it exists proves that actually this CRUD methodology, where CRUD stands for Create, Read, Update, Delete, it just shows that this CRUD methodology, which is a way to build software systems to approach supply chains, it’s just not the end game. It’s just one of the ways, and this design pattern was actually strongly influenced by economic constraints that were only relevant during the ’70s and ’80s, and already by the end of the ’90s, that was kind of fading away. So now, if we look ahead to the future, data storage is incredibly cheap, and we’ve kind of overcome those barriers. Can you see event sourcing becoming more commonplace across the market in the next couple of decades, and are there any barriers that we still need to overcome for that to happen?
Joannes Vermorel: First, event sourcing is becoming more prevalent just because Amazon is growing bigger. Also, just every single time that Amazon takes one percent market share, event sourcing has kind of become one percent more prevalent in the market. And when I say Amazon, I mean all the other tech-driven companies that are just eating the market. So yes, it’s definitely becoming more prevalent. The market is a great filter, it’s not an educator, so things don’t become more popular because they get adopted; they become more popular just because the companies that are not using them just go bust and disappear, and thus the companies that are left just happen to be using the stuff that works more. So, can it become more popular? I would say certainly. And when I look across the customer base of Lokad, we have about a quarter or a fifth of our clients who have bespoke, in-house software systems to run the company.So if you have in-house software systems, then I would strongly advise that you have a look at event sourcing. Even if you don’t adopt event sourcing straight away because it’s a complete paradigm change for the design of your software, there is still a lot to be learned. And if you are very familiar with event sourcing, that lets you actually do CRUD, you know, the usual way of doing supply chain.
Kieran Chandler: Software nowadays can be used in ways where you mitigate those problems of mutability of the past. If you’re very careful about the way you design it, you can actually mutate the past in your code and get something that is much closer to events or things. You don’t necessarily have to rewrite everything to get closer to that. I would say it’s not just a problem for IT. Some people might dismiss that as a super geeky thing that is only of interest for database architects, but I think it’s more than that.
Joannes Vermorel: I agree, those sort of very high-level concepts should be the cornerstone of supply chain practice. Just like the fact that a ledger cannot be rewritten is a cornerstone of accounting principles. In accounting, you know that you’re not supposed to go back in the books of the previous years and rewrite the transactions. People would not say this thing about ledgers is just a software detail. Those principles were established before the advent of computers.
If we take into account that being a supply chain practitioner involves dealing with a lot of data, it’s hard to perceive the reality of supply chains because they are vast, complex, and multi-site. By design, as a supply chain practitioner, you can’t say you will just see everything with your own eyes. You can see a store, a warehouse, but you can’t be everywhere. So, you are data-driven by necessity. The only things you see about your supply chain are basically the numbers pulled through many systems.
I believe that just like an accountant, you need to have certain guiding principles on the way this data is collected, organized, and processed. Going back to event sourcing, I think it’s something that supply chain practitioners should really have a good look at, because it will help them identify tons of underlying problems in their organization.
Kieran Chandler: Brilliant. Well, if we keep on waiting for Amazon, they will sooner take another percent and be closer to 100%. At some point, large companies like Amazon can collapse under their own weight due to the difficulty of becoming absolutely gigantic. That will prevent them from taking 100% of the market share, but the problem is that it will just be other competitors that are like Amazon that will take the rest of the market.
Joannes Vermorel: Yes, watch this space.
Kieran Chandler: That’s something for this week. Thanks very much for tuning in, and we’ll see you again next time. Bye for now.
