00:00:08 Introduction of the topic of improving accuracy of forecasts in the supply chain industry.
00:01:22 Explanation of what a more accurate forecast means and the use of forecasting metrics.
00:03:21 Discussion on the limitations of using mathematical metrics to measure the performance of supply chains.
00:05:30 Emphasis on measuring forecasting accuracy in dollars instead of percentages.
00:08:42 Explanation of how maximizing accuracy in percentages can be misleading for supply chain performance.
00:09:04 Discussion about the limitations of using a straightforward metric for forecasting sales.
00:10:20 Explanation of how using a model that forecasts zero sales leads to a disastrous outcome for the company.
00:11:23 Explanation of the problem with symmetrical metrics in the context of supply chain management.
00:13:02 Explanation of how forecasts are just educated opinions about the future and their impact on the supply chain.
00:16:32 Discussion of the dangers of creating divisions dedicated to improving the accuracy of forecasts.
00:18:09 Discussion of how forecasting will get better over time.
00:19:01 Explanation that better forecasting metrics do not always result in improved supply chain performance.
00:21:41 Realization that the growth and profitability of the company did not necessarily mean that the clients’ supply chains were improving.
00:22:04 Explanation of the difference between a monthly subscription model and multi-year plans.
00:25:53 Explanation of how the product evolved from just a forecast to a tool that helps discover accuracy metrics.
00:26:56 Discussions on metrics used in supply chain management.
00:27:20 The benefits of using probabilistic forecasting and specific metrics like cross-entropy and continuous rank probability score.
00:27:54 The shift in perspective from improving forecasting accuracy to maximizing supply chain performance.
00:29:51 The importance of having one person responsible end-to-end for the entire supply chain.
00:32:23 The importance of having a monolithic optimization process in supply chain management.
Summary
The founder of Lokad, Joannes Vermorel, spoke to host Nicole Zint about the limitations of using accuracy metrics to improve supply chain performance. Vermorel argued that companies often focus too much on improving forecast accuracy without considering the impact on the bottom line. He proposed that measuring forecasting accuracy in dollars of error, rather than percentages, can better assess whether a company is moving in the right direction. Vermorel also emphasized the importance of finding key supply chain decisions that maximize profit or performance, rather than solely focusing on improving forecasting accuracy. He warned against the dangers of misleading metrics that can lead to nonsensical decisions that harm the company.
Extended Summary
The topic of the interview is the accuracy of forecasts in the supply chain industry. The host, Nicole Zint, notes that despite decades of effort to improve forecast accuracy, better accuracy has not led to better performing supply chains. She wonders if the industry is looking at the problem the wrong way or focusing on the wrong problem altogether. Joannes Vermorel, the founder of Lokad, explains that a more accurate forecast means that one forecasting model is more accurate than another according to a forecasting metric. There are a variety of forecasting metrics, but they are all mathematical objects that may not be relevant to the supply chain industry. Vermorel notes that the expectation that one can just pick a mathematical metric from a textbook to fit a problem is wrong. He adds that maximizing accuracy in percentages can be misleading for supply chain performance. He also notes that shrinking forecast error through the optimization of mathematical metrics does not generate extra supply chain performance. However, increasing forecasting error does not necessarily improve supply chain performance either. Vermorel believes that the supply chain is not a one-dimensional problem and that there is a false duality between accuracy and performance.
Vermorel explains that in order to improve supply chain performance, reducing forecasting errors is crucial. However, it’s not always as simple as just reducing errors, as supply chains are multi-dimensional. The key to improving performance lies in connecting the quality of the forecast with the performance of the supply chain by injecting an economic driver. Vermorel proposes measuring forecasting accuracy in dollars of error, rather than percentages, to assess whether the company is moving in the right direction. He notes that a forecasting error expressed in percentages does not always coincide with forecasting accuracy in dollars, which is the crux of the problem.
Zint asks Vermorel how to measure supply chain performance, to which Vermorel responds that the metrics used can be hard to define. The challenge is in finding a good forecast, which is situation-dependent. Vermorel suggests looking at a specific example, such as a supermarket, to understand how to optimize forecasting. He explains that at the store level, the vast majority of products have an average demand much lower than one unit per week, meaning the most likely outcome for the vast majority of products on any given day is zero sales. If companies optimize against a metric that maximizes forecasting accuracy in percentages, they will end up with a model that just forecasts zero every day, which would be catastrophic for the company. Even worse, if a model forecasts zero, the store will resupply zero, which will lead to lost revenue and customers.
Overall, Vermorel argues that the key to improving supply chain performance lies in connecting the quality of the forecast with the performance of the supply chain by injecting an economic driver. He suggests measuring forecasting accuracy in dollars of error and taking into account the costs associated with not having enough inventory, rather than just optimizing against a metric that maximizes forecasting accuracy in percentages. By doing this, companies can avoid optimizing their forecast for the wrong outcome, such as zero sales every day, and instead achieve a better balance between supply and demand.
Vermorel discusses the issue of forecasting accuracy in supply chain management, highlighting the problem with using a symmetric forecasting accuracy metric that puts the same weight on over-forecasting and being overstocked. Vermorel argues that this is a problem as being overstocked is a significant issue with asymmetrical consequences. He argues that while forecasting accuracy is important, it should be linked to the endgame result, which is taking the right decision at the right time for every single product every single day.
Vermorel argues that introducing numerical artifacts, such as safety stocks, ABC classes, and service levels, can create a temptation to create a subgroup of specialists within a company who are experts in dealing with these numerical artifacts. However, Vermorel argues that these artifacts are not real and that creating a team of specialists who only work on improving the quality of the forecast is a root cause of the problem. He believes that such a team operates in its own bubble, producing forecasts according to their own goal and metric, and not taking into account the endgame results.
In Vermorel’s view, large companies struggle with how to spread out the workload, and introducing a numerical artifact does not necessarily mean that a team should be created to optimize it. Instead, Vermorel argues that supply chain managers should focus on the endgame results and take the right decisions at the right time for every single product every single day. While forecasting accuracy is important, it should be linked to the endgame result, and supply chain managers should be careful not to focus solely on improving forecasting accuracy without taking into account the consequences of their decisions.
They discussed about the limitations of using accuracy metrics to improve supply chain performance. Vermorel believes that companies are too focused on improving forecast accuracy without considering the impact on the bottom line. Drawing a parallel with the cargo cults of the Pacific Islands during World War II, Vermorel notes that forecasting teams optimize metrics without considering the deeper impact on the supply chain. By focusing on the deeper impact of their forecasting methods, Vermorel believes that Lokad can help clients achieve a better performing supply chain through a monthly subscription model that prioritizes positive results.
Vermorel discusses how the feedback loop in supply chain optimization is tighter, meaning that the company must be more responsive to changes in the market. He also notes that it is not enough to optimize according to a given metric, as this can create numerous problems. Instead, companies must discover the accuracy metrics that are specific to their business, which can be challenging given the many edge cases and factors unique to each industry.
Vermorel emphasizes the importance of finding key supply chain decisions that maximize profit or performance, rather than solely focusing on improving forecasting accuracy. He argues that companies must have a person responsible for end-to-end decision making and that fragmentation can lead to nonsensical decisions that harm the company. Vermorel warns against the dangers of misleading metrics that sound rational but are ultimately deeply irrational, such as focusing solely on the survival of a single chess piece rather than winning the game. He concludes by advising companies to have a monolithic optimization process rather than slicing and dicing the process, which is broken by design and counterproductive.
Full Transcript
Nicole Zint: The whole of supply chain industry has, for decades, been trying to improve the accuracy of their forecast. Every large big company has even their own division dedicated to that problem alone. But however, the outcome of this effort, perhaps counterintuitive, has shown that better accuracy and forecast has not resulted in better performing supply chains. Are we looking at the problem the wrong way or even perhaps the wrong problem to begin with? And what changes if we measure accuracy in dollars rather than percentages? This is the topic of today’s episode, so let’s kick that off with Jonas. What does it mean that a forecast is more accurate?
Joannes Vermorel: A more accurate forecast means that, according to a given forecasting metric, you have one model that is more accurate than the other. So, more specifically, when we say we have a more accurate forecast, it is a bit an abuse of language. In fact, what we are truly saying is that we have a forecasting model that is more accurate than another forecasting model, and according to what? According to a certain forecasting metric, which is just a metric, just a measurement that quantifies the forecasting error that you have on the two forecasting models. So in fact, forecasting accuracy depends directly on the metric that you use, absolutely. And there are a wide variety of forecasting metrics that are known, I would say, in the literature. The most widespread ones are probably mean square error, you have the absolute arrow, you have the mappy mean absolute percentage general, you have the weighted mapping, you have a world bestiary of functions that let you measure the forecasting error. And all of those forecasting metrics have in common that if you have like perfect results, they just tell you that your error is zero. So we have a vast array of different metrics to use. Which one is the best one? How do you know that?
Nicole Zint: As far as supply chain is concerned, this is a very tricky question because the reality is that all the metrics that I’ve listed are actually mathematical objects. You will find them in the textbooks because it gives you tons of, I would say, interesting mathematical properties. But it’s not because something is mathematically interesting that it is probably relevant to a domain of interest. You know, you have tons of things that can be very interesting from a mathematical point of view, and that doesn’t mean that it’s going to be of any relevance from a supply chain perspective. And I believe that this here lies the crux of the problem. People expect, and that’s I believe the wrong sort of expectation, that they can just go to a mathematical textbook, review the dozens of metrics and just pick one and say, this one is the one that fits for a prime. It just does not work like that. So, in fact, if we look at sort of the problem of maximizing our accuracy in percentages, that can be quite misleading for supply chain performance.
Joannes Vermorel: Yes, I mean again, our conclusion is not that we should maximize the forecasting error. This is not what I’m discussing. You see, the finding was more puzzling than that. The finding was that if you just shrink your forecast error through the optimization of mathematical metrics, it doesn’t generate extra supply chain performance. But the converse is not simple. It’s not just because you increase your forecasting error that you actually improve your supply chain performance either. You see, this is where it gets very puzzling because there is this kind of false duality where you say, well, it’s one thing or the other. Yes, it would be this if supply chain was a one-dimensional problem where if you, you know…
Nicole Zint: So, reduce the forecasting error, you improve supply chain performance, and if you just go the other way around, you will degrade supply chain performance. If we were living in a one-dimensional world, yes, that would be that, but supply chains are many, many dimensional, so it doesn’t work like that at all. And that’s where I would say the basic intuition can get those things very, very wrong. So, let me ask you this. You mentioned supply chain performance. How do you measure supply chain performance?
Joannes Vermorel: That’s the crux of the problem actually, is that you see, those supply chain metrics, is that people have a really hard time to wrap their head around what should be a good forecast. Thus, you come up with a metric that you pick from a textbook and you say, “This is it,” but you very quickly realize that it’s not because it was found in a textbook, that because it was in a mathematical textbook that it has any relevance to your supply chain problem. And thus, if you want to have something out, the question is what else. You know this feels like a very open question, and actually, it is a very, very rapid question, and the sort of practice that we have pioneered that looks at was to think in terms of essentially dollars of error. What is it that you’re trying to optimize? And well, in order to connect the quality and the performance of your forecast with the performance of your supply chain, you need to inject a dose of economic driver, a fairly large dose, and that’s when you start measuring things in dollars of error. Then, you can start assessing whether you’re actually moving the needle in the sort of direction that makes sense for your supply chain. So, in fact, the supply chain performance, the better the supply chain performance, surely the more money the company generates, the more we’re cutting costs in our supply chain, and hence our revenue increases. So, that’s quite interesting because that means that if we look at forecasting accuracy in percentages, that does not necessarily coincide with forecasting accuracy in dollars, which is sort of the core of this problem that we’re discussing right now.
Nicole Zint: Yes, and that is very concentrated. Again, I think that’s going to be a controversial statement, but reducing the forecasting error as expressed in percentages of error do not improve supply chain performance. Sometimes it can even do the exact opposite actually. So, if we look at an example, say a supermarket. A supermarket is quite an interesting problem because human behavior can be quite unpredictable. So, if I’m managing a supermarket and I want to know, am I going to sell zero shampoo bottles today or five, and I have a forecast, and Joannes, what’s the sort of difference of the accuracy of this forecast in my scenario right now if I look at it in terms of percentages versus dollars?
Joannes Vermorel: So, first let’s start by clarifying one thing. The fact that there is a thought of uncertainty or very little of uncertainty is just going to define the sort of magnitude of forecasting error that you’re going to observe. So, that’s fine. You see, this is just completely situation dependent. If you’re looking at, let’s say, the national electricity consumption, the variation from one day to the next is very, very small. There is like a daily pattern, but otherwise, the consumption is very, very stable, so you will observe very, very small variations. And if you’re looking at something that is extremely disaggregated, like the bottle of shampoos in one supermarket, you will observe in percentage much higher variation. This
Nicole Zint: Let’s have a look at this specific supermarket example. This is an anecdote that I already gave in another episode. Years ago, we did a forecasting benchmark in this setup, and what we realized is that the vast majority of products, when you’re operating at the store level, have an average demand much lower than one. You know, you’re selling one, you need per week on average per product, or something sometimes even lower. It’s more likely to sell zero than one unit of this. Absolutely, so the most likely outcome for the vast majority of products on any given day is selling zero. How does the forecast that is looking to maximize the accuracy in percentages?
Joannes Vermorel: If you take a metric, let’s say the absolute value of your forecast minus the reality, and then you can divide by yearly sales or whatever, and normalize it. What you will get is a metric that, if you try to optimize against this metric, which means if you try to find what is the forecasting model that will give you the best results according to this metric, you will end up with a model that just forecasts zero every single day. And why? Because zero sales are by far the most likely outcome on any given day. The model that is going to be the most accurate, according to this very, very simple and straightforward metric that I’ve given you, though it’s absolute value of reality minus forecast, if you just optimize that, you will have a forecast that is going to produce zero. And even weirder and even more harmful for supply chain, is that if you have a model that forecasts zero, then you will resplend zero, and that’s very quickly your store will have nothing on the shelf, and thus very quickly your forecast will be 100% accurate because you forecast zero, you sell zero. Everything is well, except it isn’t. It’s a catastrophe for the company.
Nicole Zint: Yes, that’s quite interesting. Even when we get 100% accuracy on our forecasts, we get zero revenue. And even worse than that, we still have zero revenue, but we have all the costs. We are still operating a store. We have to pay for the people, pay for the building, everything. So, you see, it’s even worse than that. And then we are losing clients because they don’t find what they are looking for, and they don’t come back.
Joannes Vermorel: Exactly. And here we see that it’s a sort of absurdity. It’s not as obvious when we look at more aggregated time series, but the problem is exactly the same. Fundamentally, the problem here in this hypermarket example is that we have massive asymmetries. The cost of having or being short of one unit is absolutely not the same compared to the cost of having one unit sitting unsold for one extra day. This is very, very asymmetric. And thus, you see, the problem with the forecasting accuracy metric that I’ve just outlined initially, absolute value of forecast minus reality, is that it’s completely symmetric. So, it puts essentially the same weight on the over-forecast and being overstocked. And here we see that as it’s a very, very simple problem where we have a massive asymmetry, and the forecasting metric doesn’t even capture that. And why would it be from the mathematical perspective or the metrics that you’re looking at, are typically symmetrical? This is from a mathematical perspective. Why would you want to have a highly asymmetrical metric? It’s usually of no…
Nicole Zint: So, I want to talk a little bit about forecasting accuracy and its role in supply chain optimization. From a mathematical interest, it is very much of interest and this is only scratching the surface. We’re just looking at one tiny problem, but this tiny problem is already sufficiently big to completely undo all the intended benefit that would come from a process that would optimize forecasts according to a symmetric metric. So, it sounds to me like not only are we maybe looking at this problem in the wrong way, but we might even be looking at the wrong problem to begin with. We’re so focused on guessing what demand is going to be that we’re not thinking about the cost of being overstocked or understocked. And we take it away from what the actual profits that we can gain, and only look at guessing the exact demand.
Joannes Vermorel: Yes, I mean more fundamentally, a forecast is just an opinion. Ideally, it’s an educated opinion about the future, which is kind of correct. However, in the end, a forecast is just that, an educated opinion about the future. It doesn’t do anything about your supply chain. The only thing that is doing anything for your supply chain is what you actually do. The decisions that you take, do I put one extra unit in this hypermarket or not, for any given product on any given day, those are the decisions. Thus, the question becomes, when you want to think in terms of forecasting accuracy, is how does improving your forecasting model contribute to your endgame, which is to take the right decision at the right time for every single product every single day. That’s a missing link, and that’s typically what is completely absent from those accuracy metrics. And that’s why when I see discussions in supply chain communities where people say, “You know what? There are 20 different metrics that we can use for supply chain. In this situation, you can use this. In this situation, you can use that,” etcetera, etcetera, usually all those discussions are entirely missing the point. They are not even starting to connect those forecasts, which are again just an opinion, with the endgame results, which are the decisions that are being made on top of those forecasts. So, we’re kind of taking our focus away from the consequence of each of those decisions.
Nicole Zint: Absolutely. But these big companies, we still have divisions dedicated specifically to improving the accuracy of these forecasts. Should there be a division like that in the first place?
Joannes Vermorel: That’s a topic that we briefly addressed in one of the earlier episodes of the chain, which was silos and divisions within large companies. You see, the problem is that when you start introducing numerical artifacts, and numerical artifacts can be of any kind, it can be ABC classes, it can be safety stocks, it can be forecast, again, I’m saying all of that are numerical artifacts. There is no such thing as a safety stock in your warehouse. You don’t have two stocks, the working stock, and the safety stock. Now there is just one stock. What you have, and when you introduce those numerical artifacts, there is a temptation to create a subgroup of specialists within the company who are going to be experts in dealing with this numerical artifact. The problem is that it’s not real. Just because you do it, or half of the industry does it, you can be misled into thinking that it’s any kind of real, but it’s not, literally. And there are tons of things like that that are just not real. Safety stocks are not real, service levels are not real, forecasts no matter how you do
Nicole Zint: So, Joannes, when we talk about numerical artifacts, what exactly do we mean?
Joannes Vermorel: Well, they are not real; they are numerical artifacts that you produce to achieve a certain type of operation and to take certain types of decisions. And so, if we go back to those large companies, they always struggle with how to spread out the workload. Just because you’ve introduced a numerical artifact doesn’t mean that you should introduce a team. This is, on the contrary, one of the root causes of those, I would say, evils that really undermine the supply chain performance in the first place.
Nicole Zint: And why is that?
Joannes Vermorel: Well, if you start to create a team of specialists who are only going to work on improving the quality of the forecast, what is going to happen? The reality is that they are going to pick a metric. Why is that? Well, because if they don’t have a metric, they can’t operate. So, they are going to pick a metric. We need something to measure, yes. And because they have a metric, it looks very rational, you know. Yes, we are optimizing the forecast, absolute value of forecast minus reality. Obviously, if we were to produce perfectly accurate forecast, our forecast error would be zero. And so, everybody agrees. Yeah, sounds reasonable, sounds rational. Except, except that we have seen in the hypermarket example that it’s absolutely not real and not rational. You can do things that are completely insane with that. Nonetheless, if you’re a large company, you’re maybe not going to realize that. The devil is in the details, and you’re not probably people are not even going to realize that it’s absolutely completely bogus and nonsensical in the first place. Nonetheless, you have a team, and then the team that is in charge of forecasting where they operate in their own bubble, you know. So, they’re not the ones taking the actual decisions that bring them in; they’re the one producing the forecast.
Nicole Zint: And why is that a problem?
Joannes Vermorel: According to their goal and their metric, they’re improving. They will be producing a series of models, and over time, they will get better at it. They will factor in seasonality; they will factor in the religious holidays. They will factor in tons of factors, and they will get better. And thus, according to the metric, the forecast will get better. And potentially, they will bring in better software, all sorts of things over time. It’ll get better according to the metric, which does not coincide with the interest of the company expressed in dollars. So, the supply chain performance does not improve with the improved forecast. Yes, and again, people would say, “But why? We have better forecasts, and so why should it improve?” Those people are not doing anything to actually improve according to the dollars of error they are shooting for accuracy metrics. You see, that’s the trick. It’s not because you do something that is kind of like something else that you will get the result that you would be obtaining if you were doing something else. You know it’s literally it’s there was a very, I’m digressing a little bit, but there is an anecdote for foreign, and they could look it up on the Wikipedia about cargo cults, you know, that was during World War II where there were US aircraft that were flying over islands in the Pacific and they were dropping cargo – food, ammos, various goods – so that the soldiers who would just arrive on the islands they would already have
Nicole Zint: When they were actually people realized that they were even witnesses the birth of new religions where people were kind of trying to trigger the apparition of an aircraft that would deliver more cargo. As you see, this is what happens when you just try to imitate on the surface something that was beneficial to you because it did happen in the past, but there is now the core substance, and this is those cargo curls that emerged by trying literally to reinstantiate the delivery of a cargo on the island by just reproducing some sort of stuff that did happen. I believe this is what is happening with, I would say, most forecasting teams who are trying to generate better supply chain performance by just optimizing those metrics. You know, it has a rationality of its own, but if you are looking at the world picture, it is absolutely not rational. This is just a veneer of rationality, and you just take the sort of forms, so you have numbers, you have smart people, you have processes, but it’s not because you have ticking all the boxes that the whole thing makes actually any sense. So, I have to ask, Joannes, initially, we started with doing forecasts and focusing on making a forecast more and more accurate, and Joannes throughout this journey of Lokad, how did you come to realize that that, in fact, did not generate this better or does not result in better performing supply chain for our clients?
Joannes Vermorel: Because it wasn’t working, plain and simple. So, how did you realize that we were looking at the wrong problem? I mean, I realized, first, how did I realize that it wasn’t working? Because that’s a tricky question because actually, even when it wasn’t working, Lokad was acquiring clients just fine and growing just fine and profitable, kind of, you know, just fine. So, if you’re growing, if you’re profitable, and you’re a software company, sounds good, sounds good, yeah, yeah. But, it wasn’t working for the clients, you know? And when I was stepping back, and I was thinking, I was asking myself, you know, the brutal honest question, “Did I really make the situation really better for the clients?” You know, if I step back, if I forget the metrics, you know, forget the metrics, just try to have, like, a gut-feeling instinctive perception of the situation, is it really getting better? And I was starting to realize it wasn’t, you know, it wasn’t. And that was, but according to all the matrix, it was. But, if I was stepping out of the matrix for a second, though, if I was trying to coldly assess whether what we’re doing was truly making a positive change, you know, in a deeper meaning, something that would, and it wasn’t. But people would say, “Oh, but according to all the metrics, we’re good, so we’re hitting the metrics.” But we’re not bringing, but that’s the problem is that if you pick a metric and you optimize against this metric, then, yes, you’re going to be better according to this metric. This is really literally what mathematical optimization and machine learning will do for you. You pick a metric, and you run the sort of numerical optimization, and you will get something better according to this metric. So, you see, this was a bit of a tautological nature. We pick a metric, we do better by this metric, what did you expect, you know? Unless the algorithms are done right buggy, we should actually do just that. But, it doesn’t mean that on the deeper level that we’re
Nicole Zint: Can you explain why you believe in early cancellation of plans?
Joannes Vermorel: Early cancellation because you see that most of our competitors go for multi-year plans. They never realize that something is wrong because fundamentally they go through an ERP, they sell their stuff, then they embark on a five-year journey. Whether it works or not, the client has pushed so much effort on this that they can’t change, so they are stuck. You know they are sunken cars that play with. There is such a psychological trap of just sticking with that since you already invested so much. You enjoy you missed it so much and thus after five years, you’re exhausted in rolling out the super complex solution, so you don’t change, you don’t want to change right away. And then when you finally decide that you want to change, you know your 8 or 10, then you go through another RFP. So, if you lose the client at this point you just say, “Well, it’s not that we were having doing anything wrong with the forecast, it’s just that the technology, you know, evolved, some of our competition got ahead on this specific client, and so we didn’t win the client again on the second RFP.” But you don’t naturally make the connection that the connection is super loose on whether you’re doing any good and whether your forecasts are actually doing something that is of value for the client. If you buy the month, suddenly you know when the supply chain director realizes that he or she gets the same sort of gut feeling that it’s just not bringing you know value on the table no matter what the KPIs are saying, then you’re out. And so, the feedback loop is much, I would say, it’s much tighter.
Nicole Zint: Can you tell me how a product has evolved as it is right now compared to just a forecast on its own?
Joannes Vermorel: We realized that in terms of forecasting accuracy, the problem is not to optimize according to a given forecasting accuracy, to a given metric. You will pick one, it’s going to have tons of problems. This is not the problem. And if your tooling is right, you pick one metric and you will optimize against this metric, and this is it. This is very straightforward. I mean it could be made even more straightforward, but this is very straightforward. Are we optimizing right now? The tune? Now that’s the thing. The tool is optimized in the metric, but the thing is that the tool that we have developed is what it takes to discover the metrics, the accuracy metrics that you need for your company. You see that’s so that the journey was we started with the idea that we can just have a preconceived set of metrics and just optimize against those and it will be good. This is not the case, and this is much worse than I initially thought. It’s not about identifying better metrics. Yes, there are some metrics that are slightly better. For example, if we go back to this hypermarket situation, if you take let’s say the pinball loss function, that’s a highly asymmetric that can be that’s a loss function that that can be made arbitrarily asymmetric. You can get marginally better results if you go for probabilistic forecasting. You can even go for your specific metrics for probabilistic forecasts, cross-entropy, continuous rank probability score, there are others. So, there are metrics that are marginally better, but this is it. They are just marginally better. The problem is that when you face a real-
Nicole Zint: Joannes, can you talk about the paradigm shift in supply chain optimization that has happened in the last decade?
Joannes Vermorel: Yes, of course. You see, this is the sort of paradigm shift that we had to undergo during the last decade. The tooling that we have nowadays is literally answering the question of what it takes for supply chain scientists to discover. We crunch the data and discuss with the supply chain expert in the company what the accuracy metric should look like. And you will realize that it’s something that does not have the sort of numerical elegance that those mathematical metrics have because there are tons of factors, edge cases, and things that are very specific to the sort of business that you operate. If you’re doing hard luxury, it’s completely different from let’s say fresh food or aerospace. So there are tons of edge cases and edge situations that only make sense because you’re looking at a very specific company. But nonetheless, those edge cases are completely critical if you want to produce results and decisions in the end that are not completely insane. So what we’re really after is those exact supply chain decisions. Yes, that would be the end game. That would be the sort of way to measure that you’re doing any good, and that is true for all the intermediate numerical artifacts that you produce.
Nicole Zint: Can you explain what you mean by numerical artifacts?
Joannes Vermorel: Yes, of course. Forecasts that are measured with their own forecasting accuracy matrix are just one type of numerical artifact. There are typically dozens more of numerical artifacts that are in the way in the middle of the process. So we went from focusing on improving forecasting accuracy to finding those key supply chain decisions that maximize the profit or maximum supply chain performance, which is quite an interesting change of perspective.
Nicole Zint: So, what is your advice to companies who want to optimize their supply chain?
Joannes Vermorel: My advice is that if it’s not one person who is responsible for everything from making sense of the data that lies in the ERP to the final generation of the production orders, replenishment order, purchase order, stock movements, price moves, then you haven’t even started to optimize your supply chain. If you don’t have one percent that is responsible end-to-end for this entire chain, then all the effort that you’ve put into better forecast or whatever is just an illusion. If the incentives aren’t aligned, those people will be doing things that are nonsensical for the company. Just think of it as imagine you’re playing chess, and I say you’re the knight and your goal is just to make sure that the knight survived till the end of the game. The question is, do you think that if you say you are playing the knight, you are playing the tower, and I’m playing the queen?
Nicole Zint: And your goal is to survive. Your goal is to survive. Do you think that by doing that we will play overall a game that has any chance of winning against the opponent?
Joannes Vermorel: No, it doesn’t. You know, that’s misleading. The goal that we’re trying to say, yes, and people say, “Oh, in 99% of the games we have played, I was playing the knight, and the knight was still on the board by the end of the game.” Yeah, but we lost every single game.
Nicole Zint: This is good, but this game…
Joannes Vermorel: That’s the problem with misleading metrics, is that it may sound and it may look super rational, but actually, at the core, it is deeply irrational. And I believe this sort of forecasting practices and forecasting divisions that many larger companies have are completely irrational. And I know it’s very tough because those divisions are full of engineers who want to do good. They are not idiots, and their bosses and the people above them, they are not idiots either, and they want to do good. So, you see, this is not a problem of having people that should be fired or whatever, no, no, it’s the worst setup. It’s just productive counter-product there.
Nicole Zint: Exactly, it cannot. It’s broken by design. It will not achieve.
Joannes Vermorel: So my advice would be, make sure that you have this one person. This person can have as many peers as you want, you know, to have packages one end of the that connects the row they are to the final decision, and that should be, you know, a monolithic optimization. You should not slice and dice this process.
Nicole Zint: Joannes, thank you very much for this topic today. Indeed, very thought-provoking. Thank you for tuning in, and see you next time.
Joannes Vermorel: Thank you.
