00:00:07 Naked forecasts and Lokad’s background.
00:01:42 High demand for time series forecasting.
00:03:14 Lack of success in naked forecasts despite statistical accuracy.
00:05:03 Lokad’s experience with a large European retailer’s benchmark.
00:07:19 Issues with naked forecasts and their impact on businesses.
00:09:25 The problems with naked forecasts in supply chain execution.
00:12:38 The importance of extreme scenarios and the role of quantile forecasting.
00:14:47 Challenges of using good forecasts in large corporations’ S&OP processes.
00:15:37 Divergence from forecast and the need for alternative future considerations.
00:17:12 Challenges of representing probabilistic data in large quantities.
00:18:57 The limitations of Excel for handling probability distributions.
00:20:25 Importance of optimizing decisions based on forecasts.
00:21:48 The need for predictive optimization and its close relationship with forecasting.
Summary
In the interview, Kieran Chandler and Joannes Vermorel discuss the limitations of traditional forecasting methods in supply chain optimization. Vermorel emphasizes the need for quantile forecasts that consider extreme scenarios, as these have the most significant impact on supply chain management. He explains that probabilistic forecasts can offer a range of possible outcomes, but managing the vast amounts of data required for this approach creates a “big data” challenge. Traditional tools like Excel are not designed to work with probabilistic data, so specialized tools are necessary. Vermorel concludes that predictive optimization, combining prediction and optimization, is a more effective approach for managing supply chain uncertainties.
Extended Summary
In this interview, Kieran Chandler, the host, discusses with Joannes Vermorel, the founder of Lokad, a software company specializing in supply chain optimization. They delve into the concept of naked forecasts and their effectiveness in improving supply chain decision-making.
When Lokad was founded in 2008, the company focused on providing “forecasting as a service,” leveraging statistical methods for time series forecasting. The idea was to use historical data, such as past sales, to predict future demand, which also takes the form of a time series. This approach generated interest among many businesses, seeking accurate time series forecasts to improve their supply chain decisions.
There was a significant appetite for time series forecasting, with many companies asking Lokad for more accurate forecasts based on their historical data. Interestingly, despite providing highly accurate forecasts with low error rates, these improved forecasts did not seem to lead to better supply chain decisions or outcomes.
Vermorel found this outcome perplexing, as one would expect better forecasts with lower error rates to result in improved decision-making and ultimately better supply chain performance. It took him a few years to understand the underlying issue behind the counterintuitive results.
The problem was not statistical in nature; the forecasts provided by Lokad were highly accurate, with minimal issues such as overfitting. Vermorel was confident that the forecasts were statistically sound, yet they seemed to cause havoc on the client-side.
Vermorel shares a story from 2011 when Lokad participated in a benchmark for forecasting the demand of 10 mini markets, with 5,000 products per mini market. Lokad won the benchmark by achieving 20% more accuracy than the second best competitor. However, they did this using a “zero forecaster” which only predicted zero demand for all products. This method highlighted the issues with traditional forecasting and accuracy percentages. Vermorel argues that there is little correlation between reduced error percentages and actual business benefits, and that focusing on accuracy percentages can be misleading.
The host questions why companies still demand traditional forecasts despite these issues. Vermorel suggests that wishful thinking plays a significant role. People believe that if they had perfect forecasts, supply chain problems would be solved, turning the process into a simple scheduling and optimization problem. However, Vermorel emphasizes that no forecast can survive contact with the market, as the reality is far more complex.
Traditional forecasts can lead to fragile supply chain execution, as the accuracy of a forecast often depends on how it is used within a company. This can lead to unintended consequences and problems. Vermorel considers this an “anti-pattern,” meaning an intended solution that consistently fails in a predictable way.
Vermorel then discusses how Lokad shifted its approach to focus on strengthening the supply chain plan based on forecasts. He uses the example of mini markets selling fresh food, where high margins justify having a lot of stock, even if it rotates slowly. In these cases, it is more important for customers to find what they are looking for than for the store to optimize inventory. Traditional forecasts focus on average demand, whereas the costs and benefits are actually found in the extremes.
The conversation then turns to the idea of producing forecasts that consider extreme scenarios, which is what Lokad did by moving from classic forecasts to quantile forecasts. Quantile forecasting adds a bias to the forecast, focusing on the extremes where the actual costs and benefits lie. This approach, Vermorel suggests, is more effective than traditional methods for optimizing supply chain management.
They discuss the challenges of forecasting and the importance of considering various future scenarios in supply chain management.
Vermorel begins by explaining that traditional forecasts, which focus on average demand, are not sufficient for effective supply chain management. Instead, he suggests using quantile forecasts, which purposely have a bias to account for extreme scenarios, such as high or low demand. He emphasizes the importance of understanding these extreme situations since they are the ones that typically have the most significant impact on supply chain management.
Chandler then asks about the role of large corporations’ internal sales and operations planning (S&OP) teams in working with forecasts. Vermorel responds that even with good forecasts, S&OP teams cannot achieve the right result because the necessary information about alternative futures is not available. He argues that forecasts can only provide one possible future, while actual outcomes will always differ from the forecasted values.
Vermorel suggests that providing probabilistic forecasts, which offer a range of possible outcomes, could be a potential solution. However, this approach presents a new set of challenges. For one, the amount of data required to represent these probabilities is enormous, especially when considering thousands of products. This creates a “big data” problem, which requires tools capable of handling large volumes of data.
Moreover, traditional tools like Excel are not designed to work with probabilistic data. Vermorel points out that there is no way to represent probability distributions within an Excel cell, making it difficult to manipulate and analyze such data. As a result, specialized tools that can perform basic operations on probability variables are necessary to fully leverage probabilistic forecasts.
Vermorel concludes that having a good forecast that considers various scenarios is only half the picture. The other half involves using the forecasts effectively to make informed decisions. He emphasizes the importance of keeping the processes that generate forecasts and optimize decisions closely connected to avoid issues related to scalability and data processing.
The discussion highlights the need for rethinking traditional forecasting methods in supply chain management. Vermorel advocates for predictive optimization, which combines prediction and optimization, as a more effective approach for managing the uncertainties and complexities inherent in supply chains.
Full Transcript
Kieran Chandler: Today on Lokad TV, we’re going to discuss why these naked forecasts don’t actually improve resource and actually can introduce a whole range of different problems. So Joannes, with a topic like naked forecasts, it seems like the sort of thing you’d find on maybe the deep dark web. So before we get taken off YouTube, perhaps you could explain to us a bit about what you mean here.
Joannes Vermorel: When I created Lokad back in 2008, the tagline of the company was forecasting as a service. I was fresh out of the university and looking at areas where statistics could apply to businesses. There was this idea of just having time series forecasting. Conceptually, it’s something very simple: you have input time series that represent your past, typically your historical sales, and then you are just going to forecast the future, which also takes the form of a time series. For a piece of software, it’s a very well-defined problem and interesting enough, that’s why Lokad got traction. There are a lot of people that were, and still are, interested in solving their problems by just having those time series forecasts.
Kieran Chandler: Is there actually an appetite for a solution of this nature? I mean, does it actually work?
Joannes Vermorel: There is an enormous appetite for time series forecasting. We get asked about it over and over again. When I started Lokad, one of the key elements of a successful start-up is doing something people want. From this perspective, naked time series forecasting had a very significant appetite. Companies were asking, “Here is our historical data represented as time series, please give us better forecasts.” But the problem was, it didn’t work. It wasn’t a statistical problem; we were already very good in terms of forecasting accuracy a decade ago. The issue was not that the metrics were wrong.
Kieran Chandler: That seems very surprising because you’d always think that if you’re getting better results from a forecast with lower error, you’re going to end up with better supply chain decisions and ultimately act in a better way. So why didn’t it work?
Joannes Vermorel: That was my initial thought: how can I possibly be wrong? All the metrics tell me I have a better forecast. I deliver to my clients a better forecast, what can possibly go wrong? The forecast was very good, and I’m not talking about problems like overfitting. It was well under control. The issue was that a forecast that is statistically more precise could still wreak havoc on the client-side. It took me a few years to understand this. At some point, we had a large European retailer who organized a benchmark across half a dozen software vendors for forecasting.
Kieran Chandler: So, we were discussing the solutions and the problem of forecasting demand for 10 mini markets, each with 5,000 products. This was back in 2011, and the objective was to predict demand three to four days ahead, as each mini market is replenished twice a week. How did Lokad perform in this benchmark?
Joannes Vermorel: Lokad proudly won the benchmark, outperforming the second best by 20% in accuracy. The metric of quality for the forecast was the absolute difference between the forecast and reality. However, we achieved this with the zero forecaster, which only returned zeros for all the demand and sales. Interestingly, forecasting zero demand would result in zero stock, and thus, sales would rapidly converge to zero. This would make the forecast not only more accurate but 100% accurate. But, of course, this is complete nonsense and doesn’t make any sense.
Kieran Chandler: So, you’re saying that there is a disconnect between having a more accurate forecast expressed in percentages and achieving actual business benefits. Why is it that companies still demand such forecasts if they can be so misleading?
Joannes Vermorel: My basic explanation is that wishful thinking is very powerful. If the forecasts were perfect, they would have zero percent of error, zero dollars of error, and zero euros of error. A perfect forecast would solve all problems, and supply chain management would become a pure problem of optimization and scheduling. But that’s not the case, and what people don’t realize is that a naked forecast, where you predict only one future, ends up being a battle plan that doesn’t survive the first day of contact with the market. There’s a military saying that no battle plan survives the first contact with the enemy, and the supply chain equivalent is that there is literally no forecast that survives its encounter with the market.
Kieran Chandler: Know, first contact with the market, and thus, what happens when you have this more accurate forecast?
Joannes Vermorel: More generally, what happens is that because your forecast is more accurate, you create a plan that is actually more fragile against divergence compared to the forecast, and thus you make your supply chain execution more vulnerable. That’s a very abstract way to look at it. The bottom line is basically, you have the forecast, but you don’t know how this forecast is going to be used, and other people in your company are just going to use this forecast in ways that you do not expect, and that will blow up. That’s why those naked forecasts are basically a bad thing. They are used in ways they shouldn’t be used, and because it happens all the time, it’s an anti-pattern nowadays. It’s something that is like an intended solution that always blows up in a completely predictable way.
Kieran Chandler: So, you moved towards strengthening the plan that you were building on a forecast once you made this realization?
Joannes Vermorel: Exactly, and then you realize that forecasting isn’t even the right thing. If I go back to the story about those mini markets, you realize, if you’re selling fresh food in a mini market, you have very large margins. You can afford to have a lot of stock because what you want is that when a client walks in, they find what they’re looking for. You have so much margin that it’s worth your investment to have a lot of stock, even stock that is going to rotate slowly. You do not care about the average demand. If there is only one client that shows up every ten days and you are selling yogurts, you can still make a very healthy profit if you’re actually selling your products with a 70% gross margin, and your yogurts have a shelf life of one month. So, bottom line, it’s not the average that’s of interest; it’s the extremes. The costs are at the extremes.
Kieran Chandler: Okay, so then why can you not just produce a forecast that looks at those extreme scenarios?
Joannes Vermorel: That’s the thing of interest, right? And that’s what we did. In the history of Lokad, we went from classic forecast in 2008 with Lokad Forecasting as a Service, and we moved towards quantile forecasts. So, quantile forecasting was an idea that, at the time in 2012, sounded very bizarre. It was forecasting with a bias. Most of my clients were saying that a good forecast is a forecast that does not have a bias. That was the opposite of the common sense understanding in supply chain management.
If you go back to the mini market case, you do not care about the average demand. It’s the extremely high demand, which is never ever high, but that’s the extremely high situation that is of interest. The question is, what is an extremely high situation? Is it one in 30? Maybe sometimes four. That’s your extremely high. By the way, it’s a statistic. Those forecasts with a bias on purpose, they’re called quantile forecasts. You can have a forecast that has, like, a 99% quantile forecast, which says, “I give you a number, and the demand has a 99% chance to be just below this number and 1% chance to be above.” So, you control the bias, and that was the start of having more diverse forecasts.
Kieran Chandler: Actually, let’s discuss situations with risk at the tails, like extreme scenarios, where you’re sort of dealing with a stock out or an overstock scenario. I don’t understand why large corporations with their own internal S&OP processes can’t work with a good forecast to get the right result at the end of the day. What is the real challenge there?
Joannes Vermorel: That’s wishful thinking. You cannot get the right result out of a forecast, even if it’s good, because the information you need is not even there. When you say, “Here is the future,” you’re only showing one possibility, but you don’t tell anything about the alternatives. The reality is that the future will be an alternative. There will always be a divergence from your forecast. The problem is that you think you can transform your limited knowledge of alternative futures into decisions like how much to buy, produce, or move stock from one location to another, without it negatively impacting the quality of your decisions. It’s like magic.
Kieran Chandler: But what if we provided them with a probabilistic forecast, giving them a range of possible factors to work with?
Joannes Vermorel: That’s an interesting idea. Conceptually, it could work. However, you face another issue, which is very mundane. A deterministic forecast is concise: for one product, one year ahead, and forecasting at a weekly level, you have 52 numbers. It’s a small dataset that can fit nicely in an Excel sheet. But with a probabilistic approach, you have a massive histogram of probabilities for every single week. These probabilities are not additive, so if you want to know the demand from week 5 to week 10, that’s going to be another histogram of probabilities.
We can provide you with this data, but it suddenly becomes a big data problem because you have thousands of products and tens of gigabytes of probabilities. You need tools that are able to crunch that much data.
Kieran Chandler: From a technical standpoint, how easy would it be to manipulate these tools and pieces of data? So, one of the problems with Excel is that it’s not designed to handle probabilistic calculations. It’s great for organizing tabular data, but it doesn’t have a way to represent probability distributions.
Joannes Vermorel: Yes, exactly. If you want to manipulate data that comes as a probability distribution, you don’t have an entry in a cell in Excel that represents a distribution of probabilities. Excel is not designed to deal with those kinds of things, and you end up with a lot of problems when you want to exploit and leverage a probabilistic forecast about your future.
Kieran Chandler: Right, so you need tools that give you all sorts of operations for probability variables. Basic operations like adding, multiplying, or dividing random variables are essential. If you don’t have these basic tools, you can’t properly work with probabilistic forecasts.
Joannes Vermorel: Yes, and having a good forecast that takes into account all possible scenarios is only half the picture. What you do with those forecasts is much more important. When you want to optimize those decisions, the process that generates the forecasts and the process that optimizes the decision need to be completely entangled.
Kieran Chandler: I see. So, the data processing for these kinds of large matrices of probabilities can be a scalability issue. It sounds like you need to keep everything close together to make it work practically.
Joannes Vermorel: Exactly. To have a practical solution, you need to keep these things very close. You should start thinking about predictive optimization. The two things, prediction and optimization, go together and cannot be taken apart.
Kieran Chandler: Okay, that makes sense. Well, we’ll have to wrap it up there. Thanks for joining us today, Joannes. It’s been really great to talk to you about supply chain optimization.
Joannes Vermorel: Thanks for having me, Kieran. It’s been a pleasure.
